# Reliability and Validity

Psychology I · Research Techniques · https://tryals.app/learn/psychology-i/reliability-and-validity

## Reliability vs. Validity

Two distinct standards determine whether a measure is trustworthy. **Reliability** is CONSISTENCY: does the measure give the same result under the same conditions?

**Validity** is ACCURACY: does the measure actually capture what it claims to capture? A bathroom scale reading 5 lbs too heavy is reliable (consistent) but not valid (accurate).

| Concept | Core Question | Example Check |
| --- | --- | --- |
| **Reliability** | Is it consistent? | Test-retest, inter-rater |
| **Construct** | Measures the right concept? | Does an anxiety scale tap anxiety? |
| **Internal** | Study justifies causal claims? | Are confounds ruled out? |
| **External** | Findings generalize? | Beyond this sample and setting? |

## Types and The Golden Rule

**Test-retest reliability** checks if people score similarly later. **Inter-rater reliability** checks if independent raters agree. **Internal consistency** (via Cronbach's alpha) checks if scale items correlate.

$$\text{Validity requires reliability, not the reverse}$$

> **Common pitfall:** Assuming a reliable measure must be valid. Consistency is necessary but NOT sufficient for accuracy; a measure can be unfailingly consistent while measuring the wrong thing.

Remember the dartboard: a tight cluster far from the bullseye is reliable, not valid.

## Practice questions

6 of this lesson's 12 practice questions, with answers. The full set is in the app.

### 1. A scale reading 73.0, 73.1, 72.9 and 73.0 for a certified 70 kg weight is reliable but not valid.

**Answer:** True

**Why:** True. Reliability is agreement with itself and validity is agreement with the truth; this scale is highly consistent and consistently 3 kg wrong. Reliability is necessary for validity and nowhere near sufficient, and a systematic bias is invisible to any repeatability check.

Page: https://tryals.app/practice/psychology-i/reliability-and-validity/a-scale-reading-73-0-73-1-72-9-and-73-0-for-a-certified-70-kg

### 2. A bathroom scale reading 3 kg heavy every time is perfectly reliable and not valid at all. Why can reliability not substitute for validity?

A. Because a reliable measure is valid provided it is calibrated
B. Because validity is a stronger form of reliability
C. Because reliability is only consistency, not correctness
D. Because reliability applies to instruments and validity to people

**Answer:** C. Because reliability is only consistency, not correctness

**Why:** Reliability is necessary and not sufficient. An unreliable measure cannot be valid, but a perfectly reliable one can be measuring the wrong thing forever.

Page: https://tryals.app/practice/psychology-i/reliability-and-validity/a-bathroom-scale-reading-3-kg-heavy-every-time-is-perfectly-reliable

### 3. A depression questionnaire has excellent internal consistency and excellent test-retest reliability. A critic argues it may still be measuring the wrong thing. Why do the reliability statistics fail to answer that objection?

A. Because the reliability statistics were probably computed on too small a sample to be trusted.
B. Because internal consistency and test-retest reliability measure the same thing, so reporting both adds nothing.
C. Because both compare the measure to itself, not to the construct.
D. Because a measure with high reliability is necessarily invalid, the two being inversely related.

**Answer:** C. Because both compare the measure to itself, not to the construct.

**Why:** Both statistics are internal to the instrument, the items agree with each other, and the scale agrees with itself a fortnight later. Twenty items about sleep would be beautifully consistent, stable, and correlated with depression, without measuring it. Construct validity needs outside evidence: convergence, discrimination, and movement when treatment works.

Page: https://tryals.app/practice/psychology-i/reliability-and-validity/a-depression-questionnaire-has-excellent-internal-consistency-and

### 4. Select every statement that is TRUE.

A. External validity concerns whether findings generalize beyond the sample.
B. Validity requires reliability, but not the reverse.
C. A scale whose items pull in different directions is measuring more than one thing.
D. A measure can be highly reliable without being valid.
E. A perfectly consistent measure must be accurate.

**Answer:** A. External validity concerns whether findings generalize beyond the sample.; B. Validity requires reliability, but not the reverse.; C. A scale whose items pull in different directions is measuring more than one thing.; D. A measure can be highly reliable without being valid.

**Why:** Reliable-without-valid, validity-requires-reliability, internal consistency, and external validity are all true. A perfectly consistent measure need NOT be accurate - the one false option is the reliability-implies-validity error.

Page: https://tryals.app/practice/psychology-i/reliability-and-validity/select-every-statement-that-is-true

### 5. Before you can study happiness you must decide what counts as happiness, a rating scale, hours smiling, a hormone. Why is that **operationalisation** step the one most likely to sink the study?

A. Because the choice affects statistical power more than any other decision
B. Because concepts like happiness cannot be measured and should not be studied
C. Because operational definitions are arbitrary, so any choice is as good as any other
D. Because every conclusion afterwards is about the measure, not the concept

**Answer:** D. Because every conclusion afterwards is about the measure, not the concept

**Why:** Everything downstream inherits the definition, and no analysis can widen it afterwards. This is construct validity, and it is why two studies of 'aggression' can disagree without either being wrong, they measured different things and used the same word.

Page: https://tryals.app/practice/psychology-i/reliability-and-validity/before-you-can-study-happiness-you-must-decide-what-counts-as

### 6. A new questionnaire has just been written. Order the checks a psychometrician runs on it, from the one that must pass first to the one that can only be answered last.

**Answer:**

1. Do the items agree with each other, or is the scale measuring several unrelated things?
2. Do the same people score similarly when retested a fortnight later?
3. Does the scale correlate with established measures of the same construct?
4. Does it distinguish people already diagnosed from people who are not?
5. Do the findings hold in a different population and setting from the one it was built on?

**Why:** The order is not arbitrary - it follows the dependency between the concepts. Internal consistency and test-retest stability come first because they are prerequisites: a scale whose items disagree with each other, or whose scores drift between fortnights, cannot be tracking anything stable, and there is no point asking what it measures until it measures something.

Once it is reliable, the validity questions begin, and they compare the scale to something outside itself. Agreement with established measures and the ability to separate diagnosed from undiagnosed groups are both evidence about the construct - the questions the reliability statistics could not answer.

External validity comes last because it needs everything before it plus a second population. A scale validated entirely on undergraduates may behave quite differently in a clinical sample or another culture, and that is a question no amount of work on the original sample can settle.

Page: https://tryals.app/practice/psychology-i/reliability-and-validity/a-new-questionnaire-has-just-been-written-order-the-checks-a