Verbatim · Inter-coder reliability
Cohen's kappa
Chance-corrected agreement; Pe from the PRODUCT of each coder's own marginal proportions.
kappa = (Po - Pe) / (1 - Pe); Pe = sum_c (p_a(c) * p_b(c))
- Measurement level
- nominal
- Coders
- Exactly two coders; for >2 use Fleiss' kappa or Krippendorff's alpha.
- Seminal reference
- Cohen (1960), Educational and Psychological Measurement 20(1):37-46, Eqs. 1-3
Benchmark bands
Benchmarks are reporting guidance, not strict cut-offs — the engine presents them with their source.
| Agreement | Reading | Source |
|---|---|---|
| 0.80 – 1.0 | almost perfect | Landis & Koch (1977), Biometrics 33(1):159-174, Table benchmark |
| 0.60 – 0.80 | substantial | Landis & Koch (1977) |
| 0.40 – 0.60 | moderate | Landis & Koch (1977) |
| 0.20 – 0.40 | fair | Landis & Koch (1977) |
| 0.0 – 0.20 | slight | Landis & Koch (1977) |
| -1.0 – 0.0 | poor or below chance | Landis & Koch (1977) |
Verbatim computes Cohen's kappa with a deterministic, golden-verified math kernel — never a language model — with a bootstrap confidence interval and full provenance (coder + unit + kernel + fixture + run-id). The number is measured; the interpretation stays yours.
Measure this in the workshop →