AI Glossary / Causal inference and experimentation / Intermediate

Cohen's kappa

Definition

Cohen's kappa is a statistic measuring agreement between two raters who assign categorical labels to the same items, adjusted for agreement expected under a model based on their label frequencies. It is calculated as observed agreement minus expected agreement, divided by one minus expected agreement. A value of 1 indicates perfect agreement when the denominator is defined; zero matches the chosen chance baseline. Kappa measures agreement, not whether either rater is correct.

What it means

Two reviewers may agree often simply because both usually choose the same common category. Kappa compares their observed agreement with what their separate category frequencies would predict if their label assignments were independent. This adjustment can help interpret annotation consistency, but the chance model is a statistical reference, not a claim that reviewers literally guessed. The label distribution affects the result, so raw agreement, category counts, and the disagreement table remain important companions.

How it works

Create a contingency table whose rows are one rater's labels and columns are the other's. Observed agreement p_o is the diagonal total divided by the number of jointly rated items. For each category, multiply the first rater's proportion using that category by the second rater's proportion. Sum those products to obtain p_e. Calculate kappa = (p_o - p_e)/(1 - p_e). For ordered categories, weighted kappa can penalize distant disagreements more than neighboring ones, but the weight scheme must be specified.

Three practical examples

• Training-data review: assess consistency between two annotators labeling messages as relevant or irrelevant.
• Rubric validation: compare two reviewers' categorical judgments of generated answers before using the rubric more widely.
• Evaluator comparison: measure agreement between an automatic judge and a reviewed label set while still inspecting which kinds of cases disagree.

Think of it this way

Think of two inspectors sorting the same cards into labeled boxes. Their matching choices matter, but if nearly every card goes into one box, a large match count is less informative than it first appears. The analogy captures frequency adjustment. It does not establish that the boxes are well designed or that both inspectors have avoided the same systematic mistake.

Worked example

Suppose two reviewers label 100 items as acceptable or unacceptable. They both mark 40 acceptable and both mark 40 unacceptable. On 10 items, reviewer A says acceptable and reviewer B says unacceptable; on another 10, their choices reverse. Observed agreement is (40 + 40)/100 = 0.8. Each reviewer uses each category 50% of the time, so expected agreement is 0.5 times 0.5 + 0.5 times 0.5 = 0.5. Kappa is (0.8 - 0.5)/(1 - 0.5) = 0.6.
Now consider a different table: both mark 80 acceptable, they disagree in each direction on 10, and neither jointly marks an item unacceptable. Raw agreement is still 80%, but both reviewers now label 90% acceptable. Expected agreement is 0.9 squared + 0.1 squared = 0.82. Kappa is (0.8 - 0.82)/(1 - 0.82), approximately -0.111. The same raw agreement can therefore produce very different kappa values. The second result calls for examining prevalence and disagreement patterns, not automatically concluding that the reviewers are broadly incompetent.

Illustrative case study

A fictional content-labeling team reports strong overall agreement but finds that one rare category causes most disputes. Rather than relying on a single kappa threshold, it reviews the disputed examples and clarifies the rubric's boundary cases. The reviewers then label a fresh sample independently. The team reports the new contingency table, raw agreement, and kappa together. This makes the effect of the revised guidance visible without treating a numerical label such as “substantial” as a universal certification of annotation quality.

Common misconceptions and mistakes

Kappa is not classification accuracy and does not require one rater to be ground truth. High agreement can result from shared errors. A value of zero does not mean the raters never agree; it means observed agreement equals the specified baseline. Negative values indicate less agreement than that baseline. Fixed verbal bands are context-dependent, and prevalence can strongly influence interpretation. If both raters assign every item to the same single category, expected agreement is 1 and the ordinary formula is undefined despite perfect raw agreement.

Business use

Use kappa to complement a transparent review of labeling consistency. Record the categories, sampling process, number of items, and whether judgments were independently made. Report uncertainty when appropriate and avoid comparing values from very different label distributions as if only rater quality changed. For more than two raters, missing judgments, or complex rating designs, choose a suitable agreement method rather than applying two-rater kappa indiscriminately. When one set of labels is a trusted reference, also inspect task-specific error measures.

People also ask

Can kappa be negative?

Yes. Observed agreement can fall below the independent-marginal baseline.

Is weighted kappa appropriate for unordered labels?

Ordinarily no: weighting requires a defensible meaning for the distance or seriousness of different disagreements.

Does a high kappa prove the rubric measures the right concept?

No. It indicates consistency under that rubric, while validity requires separate evidence about what the categories represent.

Check your understanding

Why does the second worked table have a lower kappa despite identical raw agreement?

Answer: Its category frequencies imply much higher expected agreement. Kappa subtracts and rescales relative to that different baseline, so it responds to the marginal label distribution as well as the diagonal matches.

Related terms

• Inter-rater agreement
• Contingency table
• annotation reliability
• raw agreement
• weighted kappa
• category prevalence
• rubric validity
• Cohen's kappa coefficient
• adjudication

Learn next

Construct a two-by-two rater table and calculate the diagonal agreement and marginal proportions manually. Change the margins while holding raw agreement fixed to understand the statistic's sensitivity. Next study confidence intervals and alternative agreement measures, then connect numerical results to a review of actual labeling disagreements.

More in Causal inference and experimentation

• Character N-gram F-score - Character N-gram F-score evaluates generated text by combining precision and recall of matching character sequences against reference text.
• Citation precision - Citation precision measures how well the citations an AI system supplies support the statements to which they are attached, under a specified evaluation rule.
• Citation recall - Citation recall measures how much of an answer's evaluated content is supported by the citations provided, under a defined claim or statement-level protocol.
• Collider - A collider is a variable on a causal path where two arrowheads meet, as in X→Z←Y.
• Computer experiment - A computer experiment is a planned study that runs a computational model at selected input settings and analyses its outputs.
• Computerized adaptive testing - Computerized adaptive testing (CAT) is assessment in which a computer selects later questions using information from a person's earlier responses.

Sources

Scikit-learn, cohen_kappa_score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html Scikit-learn, model evaluation guide: https://scikit-learn.org/stable/modules/model_evaluation.html Scikit-learn, original classification metric implementation: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/_classification.py

Previous: Cohen's h Next: Coherent extrapolated volition