Can two raters agree perfectly and both be wrong?

Yes; agreement does not supply a truth standard.

Does 90% agreement mean every category is handled well?

No; common categories can dominate the total.

Can I compare two agreement percentages directly?

Only with attention to their matching rules, sample composition, and uncertainty.

AI Glossary / Statistics and probability / Intermediate

Percentage of agreement

Definition

Percentage of agreement is the percentage of comparable observations for which two or more assessments match, under an explicitly stated agreement rule. For two raters assigning one category to each item, exact agreement is the number of matching category assignments divided by the number of items rated by both, multiplied by 100. It is a descriptive agreement measure without a chance correction.

What it means

The measure answers a direct operational question: how often did the assessments give the same answer? That makes it easy to communicate, but agreement is not automatically correctness. Two assessors can make the same mistake.
Its meaning depends on the items assessed, the categories, the raters, and the matching rule. Agreement on an easy, highly repetitive sample may not describe performance on difficult cases or on a different population.

How it works

Define the unit of analysis and what counts as a match. For a single nominal label, exact equality is natural. For numerical measurements, agreement within a tolerance requires a separately justified rule and is not the same as exact categorical agreement.
Have assessors evaluate the same items under the intended working conditions. If the goal is independent agreement, prevent them from seeing one another's answers before recording their own. Keep adjudicated labels separate from initial ratings.
Construct a table with one rater's categories as rows and the other's as columns. Add the diagonal counts, which represent identical labels. Divide by the total number of comparable jointly rated items.
Document missing or unclassifiable ratings. Excluding such cases changes the denominator and can make agreement appear stronger if the excluded cases are difficult. Report how many cases were excluded and why.
For more than two raters, specify whether agreement means unanimity, average pairwise agreement, or another rule. These measures answer different questions and cannot be compared solely because all are percentages.
Inspect the entire table. If almost all items receive one category, a high overall percentage may conceal substantial disagreement on the uncommon category. Category-specific summaries and an appropriately chosen chance-adjusted measure can add context.
Cohen's kappa compares observed agreement with an expected-agreement value calculated from the two raters' marginal category proportions. It is a different measure with its own limitations, not a universal replacement for reporting the actual counts.
If generalizing beyond the assessed sample, consider uncertainty and the sampling structure. Repeated items from the same source, multiple sessions, or clustered documents can invalidate a confidence interval that treats all observations as independent.

Three practical examples

AI annotation: two labelers classify the same messages before a training dataset is finalized.
Quality control: inspectors compare pass or fail decisions on a shared set of products.
Customer research: analysts assess whether a coding guide produces consistent categories for interview responses.

Think of it this way

Think of two people sorting the same cards into labeled boxes. Percentage agreement counts how often a card lands in the same box. It does not establish that the box is the correct one or that the cards represent every situation the sorters will encounter.

Worked example

Suppose two raters independently label 100 items as positive or negative. Both label 5 positive and 85 negative. Rater A labels 3 positive that B labels negative, while B labels 7 positive that A labels negative.
The matching count is 5 + 85 = 90, so exact agreement is 90%. The disagreement count is 10, or 10%.
A labels 8 items positive and 92 negative; B labels 12 positive and 88 negative. The marginal-based expected-agreement term used in Cohen's kappa is 0.08 × 0.12 + 0.92 × 0.88 = 0.8192. Kappa is (0.90 minus 0.8192)/(1 minus 0.8192), approximately 0.4469.
The 90% agreement and the kappa value describe different quantities. Most matching labels are negative, so the overall percentage alone gives limited insight into rare positive cases. Neither number establishes accuracy because the example provides no independently justified reference labels.

Illustrative case study

Fictional case: Willow Review prepares labels for a customer-message classifier. Its first report shows high overall agreement, but the table reveals uncertainty around a small escalation category. The team clarifies that category's instructions, trains reviewers with fresh examples, and evaluates a new sample while retaining the original disagreement record for analysis.

Common misconceptions and mistakes

Do not equate agreement with accuracy or correlation. A systematic numerical offset can coexist with strong correlation but poor agreement.
Do not silently count missing responses as agreements. Avoid calculating a percentage after adjudication and presenting it as independent initial-rater consistency.
A universal acceptable threshold is usually inappropriate. The required agreement depends on consequences, category prevalence, sample difficulty, and the intended use.
Chance-adjusted measures require their own interpretation; disagreement between kappa and raw agreement is not automatically a calculation error.

Business use

Agreement reporting helps organizations improve annotation guidelines, train reviewers, and identify ambiguous decisions before deploying AI. Useful reports include the sample selection, rating conditions, full category table, denominator, missing-data treatment, and consequences of the observed disagreements.

Check your understanding

How many disagreements occur in the worked example?

Answer: Ten out of 100.

Why is a category table useful beyond the headline percentage?

Answer: It shows where agreement and disagreement occur and whether a dominant category drives the summary.

Related terms

Cohen's kappa
Confusion matrix
Measurement error
Accuracy
inter-rater reliability
annotation
consensus labeling
category prevalence

Learn next

Learn next how exact agreement, category-specific agreement, weighted agreement, and chance-adjusted coefficients answer different questions about a labeling process.

Sources

Sources: Howell, Statistical Methods for Psychology glossary, https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html ; MedCalc, Inter-Rater Agreement (Kappa), https://www.medcalc.org/en/book/inter-rater-agreement.php . These primary author and software references support agreement terminology and the kappa comparison. The table and fictional case are original.

Previous: Perceiver Next: Percentile