Hypothesis Testing

Testing Claims Against Data

Hypothesis testing is a formal procedure for deciding whether data provide enough evidence to support a claim. It frames the question as a contest between two hypotheses:

• The null hypothesis $H_0$ — the default, "no effect / no difference" claim (e.g. "the coin is fair").
• The alternative hypothesis $H_1$ — the claim you are gathering evidence for (e.g. "the coin is biased").

The logic is like a courtroom: $H_0$ is presumed true until the evidence is strong enough to reject it. You compute a test statistic from the data, then the p-value — the probability of observing data at least as extreme as yours if $H_0$ were true. A small p-value means the data would be very surprising under $H_0$, so you reject $H_0$ in favour of $H_1$; a large p-value means the data are consistent with $H_0$, so you fail to reject it. The threshold is the significance level $\alpha$ (commonly 0.05): reject when $p < \alpha$.

Because the decision is based on chance data, two errors are possible:

• Type I error — rejecting a true $H_0$ (a false positive). Its probability is $\alpha$.
• Type II error — failing to reject a false $H_0$ (a false negative).

There is a trade-off: lowering $\alpha$ reduces false positives but raises false negatives.

A critical interpretive point: "fail to reject $H_0$" is not "prove $H_0$ true." Absence of strong evidence against the null is not evidence for it — the test may simply lack power (too little data). And the p-value is not the probability that $H_0$ is true; it is the probability of the data given $H_0$.

Common pitfall: reading a p-value as "the probability that $H_0$ is true," or treating "fail to reject $H_0$" as "$H_0$ is proven true." The p-value is the probability of data at least as extreme as observed assuming $H_0$ holds — a statement about the data, not about the hypothesis. And failing to reject means only that the evidence was insufficient, not that the null is established; you never prove the null.