I can apply the rule that p below 0.05 means significant, but I cannot say what has been established when a paper reports it. Some of my classmates read it as a 95 percent chance the hypothesis is true, and I am fairly sure that is wrong.
What does the number actually claim?
Accepted Answer
p is the probability of getting data at least this extreme if the null hypothesis were true. Nothing more.
Read the order of that carefully. It is the probability of the data given the null, not the probability of the null given the data. Those are different quantities and swapping them is the most common error in the field.
A p of 0.03 does not mean a 97 percent chance the effect is real. It means that if there were no effect at all, data this extreme would show up 3 percent of the time.
That is also why p says nothing about how big the effect is. With a large enough sample, a trivial difference becomes significant.
And the 0.05 threshold is worth knowing the history of, because people treat it as a law of nature.
It was a convention Fisher suggested as a rough guide, not a discovery. He explicitly expected researchers to use judgement rather than a fixed cutoff.
Treating it as a pass mark is a large part of what produced the replication crisis, because a study at p = 0.049 gets published and one at p = 0.051 goes in a drawer, while the two are almost identical evidence.
Which is why journals increasingly ask for effect sizes and confidence intervals alongside p. Those answer "how big, and how sure" rather than "did it clear the bar".
1 more answer and the discussion on each one are open to members.
Join free to read the rest