How Big, and How Likely to Detect It
Statistical power, introduced earlier in this course, is a study's probability of correctly detecting a true effect when one genuinely exists (formally, , where beta is the Type II error rate). Power depends on four main factors: a larger sample size raises power; a larger effect size (a bigger true difference) raises power, since bigger effects are easier to detect; a larger alpha level (a more lenient significance threshold) raises power, at the cost of a higher Type I error rate; and lower variability in the data raises power, since less "noise" makes a true signal easier to detect.
| Factor | Effect on power |
|---|---|
| Larger sample size | Increases power |
| Larger effect size | Increases power |
| More lenient alpha | Increases power (but raises Type I error risk) |
| Lower data variability | Increases power |
Effect size measures the MAGNITUDE of a difference or relationship, independent of sample size — a crucial complement to the p-value, since a p-value alone conflates "how big" with "how much data." Cohen's d is a widely used standardized effect size for comparing two group means, expressing the difference in standard deviation units:
Jacob Cohen proposed rough conventions for interpreting d: small (around 0.2), medium (around 0.5), and large (around 0.8) — though these are general guidelines, not rigid cutoffs, and what counts as a practically meaningful effect size can vary considerably by research area.
Common pitfall: treating statistical significance (a small p-value) as automatically meaning a LARGE, practically important effect. With a big enough sample size, even a tiny, practically trivial effect size can reach statistical significance — which is exactly why researchers report effect size ALONGSIDE significance, not instead of it.
Two overlapping bell curves: two curves heavily overlapping (small d); they separate further apart (medium d), overlap visibly shrinking; they separate to barely touch (large d).