Verbatim · Inter-coder reliability
Scott's pi
Like Cohen's kappa but Pe uses POOLED (averaged-across-coders) marginal proportions; equals Krippendorff's alpha (2 coders, nominal) up to the (n-1) finite-sample correction.
pi = (Po - Pe) / (1 - Pe); Pe = sum_c p_bar(c)^2, p_bar = pooled marginal proportion
- Measurement level
- nominal
- Coders
- Exactly 2 coders
- Seminal reference
- Scott (1955), Public Opinion Quarterly 19(3):321-325
Benchmark bands
Benchmarks are reporting guidance, not strict cut-offs — the engine presents them with their source.
| Agreement | Reading | Source |
|---|---|---|
| 0.80 – 1.0 | acceptable | Krippendorff (2004), Content Analysis 2nd ed., p.241 (alpha-family >=.80 reporting convention; applied to pi as same chance-corrected family) |
| 0.67 – 0.80 | tentative | Krippendorff (2004), p.241 (.667-.80 tentative band) |
| -1.0 – 0.67 | insufficient | Krippendorff (2004), p.241 |
Verbatim computes Scott's pi with a deterministic, golden-verified math kernel — never a language model — with a bootstrap confidence interval and full provenance (coder + unit + kernel + fixture + run-id). The number is measured; the interpretation stays yours.
Measure this in the workshop →