Psychology I / Qualitative Data Analysis
Practice question · Multiple choice

Two researchers independently code 40 transcript segments and agree on 34. Why is the resulting 85% agreement figure a weaker guarantee than it appears?

Hints
  1. Suppose there were only two codes and both researchers assigned them by coin flip. What agreement would you expect?
  2. The figure has a floor built into it. What is it?
Show the answer

D. Because some agreement would occur by chance alone.

Why

Raw agreement has a floor that has nothing to do with skill: two coders assigning at random over two categories agree about half the time, so 85% is 85 against a baseline of 50. Kappa asks how much of the room above chance was actually captured, which is why the same 85% can be excellent or mediocre depending on the coding scheme's structure.

Read the lesson: Qualitative Data Analysis →

Practise Qualitative Data Analysis

The app has 4 more questions on this lesson, and keeps your place in the course. Psychology I is free to start.

More questions on Qualitative Data Analysis