Practice question · Multiple choice
A researcher analysing one dataset can choose which outliers to drop, which covariates to include and when to stop collecting. Each choice is defensible. Why is the combination a problem?
Hints
- Count the analyses the researcher could have run. How many were reported?
- Nobody cheated. Ask what the reported error rate now describes.
Show the answer
C. Because the defensible choices form a garden of forking paths
Why
No single step is wrong and the ensemble is: many tests were effectively available, one was reported, and the nominal 5% describes a procedure nobody followed. It needs no dishonesty, which is exactly why preregistration exists, the fix is making the path visible, not trusting intent.
Practise Statistical Significance
The app has 6 more questions on this lesson, and keeps your place in the course. Psychology I is free to start.
More questions on Statistical Significance
- Sort each statement about p-values as a correct interpretation or a misinterpretation.
- A study reports p = 0.03. Why is it wrong to conclude there is a 3% chance the null hypothesis is true, and…
- Select every statement that is TRUE about hypothesis testing.
- A result with p = 0.049 and a result with p = 0.051 tell you almost the same thing about the world, even…