# Statistical Significance

Psychology I · Research Techniques · https://tryals.app/learn/psychology-i/statistical-significance

## Deciding When a Result Isn't Just Chance

**Hypothesis testing** starts with two opposing claims. The **null hypothesis (H0)** states there is NO real effect, while the **alternative hypothesis (H1)** states a real effect exists.

A **p-value** measures the probability of observing data this extreme IF the null hypothesis were true. A small p-value casts doubt on pure chance.

By convention, if **$p < 0.05$**, the result is **statistically significant**, and we reject $H_0$. 

> **Pitfall:** A p-value is NOT the probability that the null hypothesis is true. It assumes $H_0$ is true and evaluates the data.

## Errors and Power in Testing

Because decisions happen under uncertainty, tests can fail in two ways. **Type I error** is a false positive (rejecting a true $H_0$). **Type II error** is a false negative (failing to reject a false $H_0$).

| Reality | Test says "significant" | Test says "not significant" |
|---|---|---|
| No real effect ($H_0$ true) | Type I error | Correct |
| Real effect ($H_0$ false) | Correct | Type II error |

**Statistical power** is the ability to correctly detect a true effect, which rises with larger sample sizes.

$$\text{Reject } H_0 \text{ when } p < 0.05$$

## Practice questions

5 of this lesson's 11 practice questions, with answers. The full set is in the app.

### 1. Sort each statement about p-values as a correct interpretation or a misinterpretation.

**Answer:**

- Correct interpretation: It is the probability of the data (or more extreme) if H0 is true, A smaller p-value means the observed result is more surprising under H0
- Misinterpretation: It is the probability that H0 is true, p = 0.04 proves the effect is large and important, p just below and just above 0.05 differ dramatically in what they tell you

**Why:** Correct: the p-value is computed under H0 and smaller means more surprising under H0. Misinterpretations: that it is the probability H0 is true, that significance proves a large effect, or that 0.05 is a sharp cliff. Each misreading is a specific, named trap.

Page: https://tryals.app/practice/psychology-i/statistical-significance/sort-each-statement-about-p-values-as-a-correct-interpretation-or-a

### 2. A researcher analysing one dataset can choose which outliers to drop, which covariates to include and when to stop collecting. Each choice is defensible. Why is the combination a problem?

A. Because the choices introduce random error that compounds across steps
B. Because dropping outliers is never defensible, whatever the justification
C. Because the defensible choices form a garden of forking paths
D. Because covariates always inflate effect sizes, which biases the result upward

**Answer:** C. Because the defensible choices form a garden of forking paths

**Why:** No single step is wrong and the ensemble is: many tests were effectively available, one was reported, and the nominal 5% describes a procedure nobody followed. It needs no dishonesty, which is exactly why preregistration exists, the fix is making the path visible, not trusting intent.

Page: https://tryals.app/practice/psychology-i/statistical-significance/a-researcher-analysing-one-dataset-can-choose-which-outliers-to-drop

### 3. A study reports p = 0.03. Why is it wrong to conclude there is a 3% chance the null hypothesis is true, and what does the 3% actually refer to?

A. Because the correct figure is 6%, since a two-tailed test doubles the probability that the null is true.
B. Because the p-value is computed by assuming the null is true.
C. Because p-values say nothing about the null hypothesis at all, only about the size of the effect observed.
D. Because the null hypothesis is either true or false, so no probability of any kind can be attached to any hypothesis.

**Answer:** B. Because the p-value is computed by assuming the null is true.

**Why:** The calculation begins by assuming the null is true, so it cannot deliver a probability that it is false, the output would contradict its own input. And the two quantities can differ wildly depending on prior plausibility: a low p-value on a wild claim is far likelier to be one of the 3% than a discovery.

Page: https://tryals.app/practice/psychology-i/statistical-significance/a-study-reports-p-0-03-why-is-it-wrong-to-conclude-there-is-a-3

### 4. Select every statement that is TRUE about hypothesis testing.

A. A significant result at p = 0.049 is worlds apart from p = 0.051.
B. A Type II error is failing to detect a real effect.
C. The alpha level directly sets the Type I error rate.
D. A significant result tells you how surprising the data would be if nothing were there.
E. Statistical power typically rises with larger sample sizes.

**Answer:** B. A Type II error is failing to detect a real effect.; C. The alpha level directly sets the Type I error rate.; E. Statistical power typically rises with larger sample sizes.

**Why:** Alpha setting the Type I rate, Type II as a miss, and power rising with sample size are all true. A p-value is not the probability the null is true, and results just either side of 0.05 are nearly identical - the two false options are classic misreadings.

Page: https://tryals.app/practice/psychology-i/statistical-significance/select-every-statement-that-is-true-about-hypothesis-testing

### 5. A result with p = 0.049 and a result with p = 0.051 tell you almost the same thing about the world, even though only the first is labeled 'statistically significant'.

**Answer:** True

**Why:** True. The 0.05 cutoff is a convention, so p = 0.049 and p = 0.051 reflect nearly identical evidence - only the significance label changes at the arbitrary line. Treating that line as a sharp truth boundary is a well-known pitfall.

Page: https://tryals.app/practice/psychology-i/statistical-significance/a-result-with-p-0-049-and-a-result-with-p-0-051-tell-you-almost