Verbatim · Inter-coder reliability
Per-category reliability (one-vs-rest Cohen kappa)
Diagnostic decomposition to localise WHICH code drives disagreement; not a standalone reported coefficient.
For each category c: collapse to 2x2 (c vs not-c) then Cohen's kappa
- Measurement level
- nominal
- Coders
- Exactly 2 coders
- Seminal reference
- Cohen (1960) applied one-vs-rest; standard reliability-diagnostics practice (Neuendorf 2017)
Benchmark bands
Benchmarks are reporting guidance, not strict cut-offs — the engine presents them with their source.
| Agreement | Reading | Source |
|---|---|---|
| 0.80 – 1.0 | almost perfect | Landis & Koch (1977) |
| 0.60 – 0.80 | substantial | Landis & Koch (1977) |
| 0.40 – 0.60 | moderate | Landis & Koch (1977) |
| -1.0 – 0.40 | weak | Landis & Koch (1977) |
Verbatim computes Per-category reliability (one-vs-rest Cohen kappa) with a deterministic, golden-verified math kernel — never a language model — with a bootstrap confidence interval and full provenance (coder + unit + kernel + fixture + run-id). The number is measured; the interpretation stays yours.
Measure this in the workshop →