Verbatim · Inter-coder reliability
Fleiss' kappa
Generalisation to a FIXED number n of raters per unit (raters may differ across units); nominal.
kappa = (Pbar - Pe) / (1 - Pe); P_i = (sum_j n_ij^2 - n)/(n(n-1)); Pe = sum_j p_j^2
- Measurement level
- nominal
- Coders
- n raters per unit, fixed; n>=2 required (n-1 in P_i denominator).
- Seminal reference
- Fleiss (1971), Psychological Bulletin 76(5):378-382
Benchmark bands
Benchmarks are reporting guidance, not strict cut-offs — the engine presents them with their source.
| Agreement | Reading | Source |
|---|---|---|
| 0.80 – 1.0 | almost perfect | Landis & Koch (1977) |
| 0.60 – 0.80 | substantial | Landis & Koch (1977) |
| 0.40 – 0.60 | moderate | Landis & Koch (1977) |
| 0.20 – 0.40 | fair | Landis & Koch (1977) |
| 0.0 – 0.20 | slight | Landis & Koch (1977) |
| -1.0 – 0.0 | poor or below chance | Landis & Koch (1977) |
Verbatim computes Fleiss' kappa with a deterministic, golden-verified math kernel — never a language model — with a bootstrap confidence interval and full provenance (coder + unit + kernel + fixture + run-id). The number is measured; the interpretation stays yours.
Measure this in the workshop →