Are inter-rater and inter-annotator agreement the same idea?

Yes, in annotation settings the terms commonly describe consistency among people judging the same items.

Does low agreement mean the raters are careless?

Not necessarily. The task, instructions, available context, or category definitions may create legitimate uncertainty.

Is raw percentage agreement sufficient?

It is useful but may be incomplete. Category frequencies and an appropriate additional measure can change how the result should be interpreted.

AI Glossary / Causal inference and experimentation / Intermediate

Inter-rater agreement

Definition

Inter-rater agreement measures how consistently different people assign judgments to the same items under a specified procedure. In AI work, it is used to examine annotations, evaluation scores, relevance judgments, and other human assessments. Agreement can be summarized in several ways, depending on the rating scale and study design. It indicates consistency between raters; it does not by itself establish that their shared judgments are correct.

What it means

If two reviewers label the same customer message differently, the disagreement might arise from unclear instructions, overlapping categories, missing context, or a genuinely ambiguous message. Measuring agreement helps make these differences visible before the labels are treated as dependable evidence.

The result belongs to a particular task, sample, group of raters, and set of instructions. Agreement on easy examples need not describe performance on difficult ones. Likewise, agreement after reviewers discuss their answers is different from independent agreement. A useful report explains how judgments were obtained rather than presenting a coefficient without the setting that produced it.

How it works

Ask raters to judge overlapping items independently using the same instructions. Retain their individual answers before any adjudication or discussion. For two raters assigning categories, raw agreement is the number of matching judgments divided by the number of jointly rated items.

Cohen's kappa adjusts observed agreement using an expected-agreement calculation derived from each rater's category frequencies. Other measures suit different arrangements, such as more raters, ordered scores, or numerical measurements. The choice should follow the data and design rather than whichever statistic produces the highest number.

Inspect the disagreement table and category frequencies alongside the summary. Then investigate whether disagreements reflect instruction problems, rater differences, or meaningful ambiguity that the task should preserve.

Three practical examples

1. Support classification: two annotators independently classify overlapping tickets to discover whether billing and account-access categories are being applied consistently.

2. Generated-answer review: reviewers score factual support using a shared rubric, and the team checks whether apparently small model differences are obscured by inconsistent judgments.

3. Search assessment: assessors judge whether the same documents satisfy an information need, exposing differences in how they interpret relevance or the user's intent.

Think of it this way

Think of two inspectors using the same checklist on the same packages. Matching results suggest the checklist can be applied consistently. The analogy has limits: inspectors might both overlook the same defect, and an unclear package may reasonably fit more than one category. Agreement is evidence about the measurement process, not an automatic certificate of truth or a reason to suppress legitimate disagreement.

Worked example

Two raters classify 100 items as positive or negative. Both mark 35 positive and 45 negative. On 15 items, A says positive while B says negative; on five, A says negative while B says positive.

Observed agreement is (35 + 45)/100 = 0.80. A uses positive and negative equally often: 50 of each. B uses positive 40 times and negative 60 times. The expected agreement used by Cohen's kappa is 0.50 × 0.40 + 0.50 × 0.60 = 0.50.

Kappa is therefore (0.80 − 0.50)/(1 − 0.50) = 0.60. This is a different quantity from the 80% raw agreement. Neither number reveals which rater is correct on the 20 disagreements, and a project-specific interpretation is still needed.

Illustrative case study

Fictional case: River Desk creates labels for customer requests. Reviewers initially show strong overall agreement, but a disagreement table reveals repeated confusion between cancellation requests and complaints about a cancellation. The broad score hides a problem concentrated in a category important to the workflow.

The team clarifies examples and adds a rule for messages containing both intentions. It pilots the revised instructions on fresh overlap items rather than replacing earlier judgments and declaring the original agreement improved. Ambiguous cases retain notes for adjudication. The resulting process distinguishes consistency measurement from the separate decision about which final label should enter the training dataset.

Common misconceptions and mistakes

Do not equate agreement with accuracy. Two raters can consistently share a mistaken interpretation.

Avoid claiming that one universal coefficient threshold proves annotations are adequate. Category balance, task difficulty, consequences of disagreement, and the chosen measure affect interpretation.

Do not calculate independent agreement using labels already harmonized through discussion. That measures the outcome of reconciliation. Similarly, a high correlation between numerical ratings does not necessarily mean the raters assign similar values: one might systematically score every item higher. Choose a measure that matches what consistency means for the task.

Business use

Agreement analysis helps a business identify whether its training labels and evaluation results are stable enough to support a decision. It can guide improvements to instructions, reviewer training, category definitions, and allocation of expert review.

Budget for overlapping assessments rather than assigning every item to only one person. Review disagreements in categories where errors are costly or frequent, and retain the original ratings for audit and analysis. When comparing model versions, consider whether evaluator inconsistency could explain an apparent difference. The purpose is to improve the measurement process, not merely to obtain an impressive agreement score.

Check your understanding

After two reviewers discuss every disagreement, their final labels match perfectly. Does that demonstrate perfect independent agreement?

Answer: No. The reconciled labels show consensus after discussion. Independent agreement must be assessed from their original judgments before that reconciliation.

Related terms

Human evaluation
Cohen's kappa
Reliability
Validity
Confusion matrix
annotation
Krippendorff's alpha
adjudication
evaluation rubric

Learn next

Have two people independently label an overlapping sample. Build a table of paired judgments, compute raw agreement, and inspect which categories cause disagreements. If using kappa, calculate its expected-agreement term and explain why it differs from raw agreement. Preserve the original labels when revising the instructions.

Sources

Primary references: Google Machine Learning Glossary, inter-rater agreement, https://developers.google.com/machine-learning/glossary#inter-rater-agreement ; scikit-learn, cohen_kappa_score, https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html . The glossary definition and implementation documentation's formula were consulted. This entry introduces the measurement concept without prescribing a universal acceptance threshold. The paired-count calculation and fictional classification case are original teaching examples.

Previous: Inter-quartile range Next: Inter-record redundancy