The Mechanics Behind the P-Value
Building on the logic of null-hypothesis testing introduced earlier in this course, this lesson looks at the statistical MACHINERY behind a p-value. A test statistic (such as a t-statistic or z-statistic) converts the observed data into a single standardized number measuring how far the result sits from what the null hypothesis predicts. That test statistic is then compared to a critical value: the threshold the test statistic must exceed for the result to be declared statistically significant at the chosen alpha level.
Tests can be one-tailed or two-tailed, depending on the hypothesis being tested. A one-tailed test predicts an effect in ONE specific direction only (e.g. "the new therapy will reduce anxiety," not simply "change" it) and places the entire rejection region in one tail of the distribution. A two-tailed test allows for an effect in EITHER direction (e.g. "the new therapy will change anxiety, somehow") and splits the rejection region between both tails — making a two-tailed test more conservative, since it requires a more extreme result in the ONE relevant direction to reach the same overall significance level.
| Test type | Predicts | Rejection region |
|---|---|---|
| One-tailed | An effect in one specific direction | Entirely in one tail |
| Two-tailed | An effect in either direction | Split between both tails |
An increasingly favored companion (and, some argue, alternative) to the p-value is the confidence interval (CI): a range of plausible values for the true population effect, computed from the sample data. A 95% confidence interval means that if the same study were repeated many times, about 95% of the resulting intervals would contain the true population value. Confidence intervals convey not just WHETHER an effect is statistically significant, but a sense of its plausible SIZE and precision — information a bare p-value discards entirely.
Common pitfall: interpreting a 95% confidence interval as meaning "there is a 95% probability the true value falls within THIS SPECIFIC interval." The more precise interpretation is about the LONG-RUN PROCESS: if the study were repeated many times, about 95% of the resulting intervals (which would differ from sample to sample) would capture the true value — a subtle but real distinction from a probability statement about this one particular interval.