Deciding When a Result Isn't Just Chance
Hypothesis testing begins by stating a null hypothesis (H0) — that there is NO real effect or relationship, and any observed difference is due to chance — and an alternative hypothesis (H1) — that a real effect does exist. Researchers then calculate a p-value: the probability of observing data at least as extreme as what was found, IF the null hypothesis were actually true. A small p-value suggests the observed result would be unlikely under pure chance, casting doubt on the null hypothesis.
By convention, psychology (like most sciences) uses a threshold of alpha = 0.05: if , the result is called statistically significant, and researchers reject the null hypothesis in favor of the alternative. This threshold is a convention, not a magic guarantee of truth — a p-value just below 0.05 and one just above it are not meaningfully different in what they tell you about the world, even though the significance label flips.
Because this is a decision made under uncertainty, two kinds of errors are possible. A Type I error (a false positive) means rejecting a TRUE null hypothesis — concluding there is an effect when there really isn't one; the alpha level directly sets the Type I error rate. A Type II error (a false negative) means failing to reject a FALSE null hypothesis — missing a real effect that is actually there. A study's statistical power is its ability to correctly detect a true effect when one exists; power typically rises with larger sample sizes.
| Reality | Test says "significant" | Test says "not significant" |
|---|---|---|
| No real effect (H0 true) | Type I error (false positive) | Correct |
| Real effect exists (H0 false) | Correct | Type II error (false negative) |
Common pitfall: interpreting a p-value as "the probability the null hypothesis is true." It is NOT that — a p-value is the probability of the observed data (or more extreme), ASSUMING the null hypothesis is true; it says nothing directly about the probability that the null hypothesis itself is true or false.
A bell-curve distribution: a symmetric null-hypothesis curve centered at zero; a shaded rejection region appears in the tail beyond the alpha = 0.05 cutoff line.