A new questionnaire has just been written. Order the checks a psychometrician runs on it, from the one that must pass first to the one that can only be answered last.

1. Do the items agree with each other, or is the scale measuring several unrelated things?
2. Do the same people score similarly when retested a fortnight later?
3. Does the scale correlate with established measures of the same construct?
4. Does it distinguish people already diagnosed from people who are not?
5. Do the findings hold in a different population and setting from the one it was built on?

The order is not arbitrary - it follows the dependency between the concepts. Internal consistency and test-retest stability come first because they are prerequisites: a scale whose items disagree with each other, or whose scores drift between fortnights, cannot be tracking anything stable, and there is no point asking what it measures until it measures something.

Once it is reliable, the validity questions begin, and they compare the scale to something outside itself. Agreement with established measures and the ability to separate diagnosed from undiagnosed groups are both evidence about the construct - the questions the reliability statistics could not answer.

External validity comes last because it needs everything before it plus a second population. A scale validated entirely on undergraduates may behave quite differently in a clinical sample or another culture, and that is a question no amount of work on the original sample can settle.