Statistical Significance

Deciding When a Result Isn't Just Chance

Hypothesis testing starts with two opposing claims. The null hypothesis (H0) states there is NO real effect, while the alternative hypothesis (H1) states a real effect exists.

A p-value measures the probability of observing data this extreme IF the null hypothesis were true. A small p-value casts doubt on pure chance.

By convention, if $p < 0.05$, the result is statistically significant, and we reject $H_0$.

Pitfall: A p-value is NOT the probability that the null hypothesis is true. It assumes $H_0$ is true and evaluates the data.

Errors and Power in Testing

Because decisions happen under uncertainty, tests can fail in two ways. Type I error is a false positive (rejecting a true $H_0$). Type II error is a false negative (failing to reject a false $H_0$).

Statistical power is the ability to correctly detect a true effect, which rises with larger sample sizes.

$$\text{Reject } H_0 \text{ when } p < 0.05$$