Verbatim
Inter-coder reliability coefficients
Verbatim measures how far coders agree with these chance-corrected coefficients. Each number is computed by a golden-verified math kernel with a bootstrap confidence interval — never by a language model.
- Cohen's kappa — Chance-corrected agreement; Pe from the PRODUCT of each coder's own marginal proportions. (Cohen (1960), Educational and Psychological Measurement 20(1):37-46, Eqs. 1-3)
- Scott's pi — Like Cohen's kappa but Pe uses POOLED (averaged-across-coders) marginal proportions; equals Krippendorff's alpha (2 coders, nominal) up to the (n-1) finite-sample correction. (Scott (1955), Public Opinion Quarterly 19(3):321-325)
- Fleiss' kappa — Generalisation to a FIXED number n of raters per unit (raters may differ across units); nominal. (Fleiss (1971), Psychological Bulletin 76(5):378-382)
- Krippendorff's alpha (nominal) — Any number of coders, missing data permitted; the recommended general-purpose reliability index for content analysis. Bootstrap CI per Hayes & Krippendorff (2007). (Krippendorff (2004), Content Analysis 2nd ed., Ch.11; Krippendorff (2011/2007) Computing Krippendorff's Alpha-Reliability)
- Percent (observed) agreement — Raw observed agreement; NOT chance-corrected. Report alongside a chance-corrected coefficient, never instead of one — it overstates reliability when categories are skewed. (Scott (1955) (critique of raw agreement); Neuendorf (2017), The Content Analysis Guidebook 2nd ed.)
- Per-category reliability (one-vs-rest Cohen kappa) — Diagnostic decomposition to localise WHICH code drives disagreement; not a standalone reported coefficient. (Cohen (1960) applied one-vs-rest; standard reliability-diagnostics practice (Neuendorf 2017))
- Krippendorff's unitizing alpha (uα) — DIFFERENT problem from fixed-unit alpha: here the coders independently SEGMENT a continuum, so the unit BOUNDARIES vary, not just the codes. Computed from the existing anchored coding spans (char_start/char_end) — each coder's coded spans are their units, the uncoded matter is gaps. The headline is the no-metric coefficient (unit IDENTIFICATION agreement); it answers 'did the coders mark the SAME stretches of text as relevant', not 'did they give them the same label'. (Krippendorff (1995a), Sociological Methodology 25:47-76; corrected in Krippendorff, Mathet, Bouvry & Widlocher (2016), Quality & Quantity 50:2347-2364; canonical text: Krippendorff (2013/2015), 'Agreement when Unitizing and Coding Finite Continua', replacement of Section 12.4 in Content Analysis 3rd ed., Sage)