AI Glossary / Causal inference and experimentation / Intermediate

Error rate

Definition

Error rate is the proportion of evaluated units counted as incorrect under a defined rule. In ordinary single-label classification, it is the number of misclassified examples divided by the total number evaluated, and equals one minus accuracy. The term is broader than that formula in other tasks: words, characters, labels, requests, or complete workflows may be the units. A meaningful error rate always specifies what counts as an error and what is included in the denominator.

What it means

Saying an AI system has a 2% error rate is incomplete unless the reader knows what was tested. Two errors among 100 document fields differ from two incorrect documents among 100 documents containing many fields. Some mistakes are small, while others invalidate an entire result. A simple rate treats counted errors equally unless a weighting scheme says otherwise. It is a useful summary of frequency, but it does not by itself describe severity, uncertainty, or the consequences of the mistakes.

How it works

Define the evaluation unit, reference outcome, and correctness rule before inspecting results. Compare predictions with the reference and mark each unit correct or incorrect. Divide incorrect units by all eligible evaluated units, then express the result as a fraction or percentage. Record exclusions, missing outputs, abstentions, and weights. In multilabel tasks, an error can mean any mismatch in the complete label set or a mismatch in each individual label decision; these are different metrics. Evaluate on data appropriate to the intended deployment question.

Three practical examples

• Classification: report the fraction of messages routed to the wrong category on a held-out sample.
• Extraction: measure field-level errors and whole-document failures separately because their denominators capture different concerns.
• Automated workflows: count requests that fail a clearly defined completion criterion, while separating model mistakes from infrastructure failures for diagnosis.

Think of it this way

Think of counting damaged items in a shipment. The fraction is easy to understand only after deciding whether the unit is a box, an individual product, or a component inside a product. The analogy explains denominator choice. It does not capture that different defects can have very different consequences even when each adds one to the count.

Worked example

Suppose a classifier evaluates 500 messages and assigns the wrong category to 35. Its error rate is 35/500 = 0.07, or 7%, and accuracy is 93%. Of the 500 messages, 50 belong to an uncommon category. If 20 of those 50 are wrong, that category's error rate is 40%. The remaining 450 contain 15 errors, giving 15/450, approximately 3.33%. The overall 7% rate hides this uneven performance.
Now suppose a revised classifier makes 25 errors on the same 500 examples. Its error rate is 5%, a reduction of 2 percentage points. Relative to the original 7%, the reduction is 2/7, approximately 28.57%. These are different ways of describing the same change. Neither establishes that the improvement is statistically reliable or that the revised system is better on every important category. The paired examples and error types should be reviewed before making a broader performance claim.

Illustrative case study

A fictional invoice team reports a low field-level error rate but receives complaints about many invoices needing correction. Each invoice contains several fields, so even occasional field errors can affect a substantial number of complete documents. The team begins reporting both field-level error rate and document-level all-fields-correct performance. It also records which fields cause the most rework. The new report explains the workload more clearly without changing the underlying predictions or pretending that one denominator is universally correct.

Common misconceptions and mistakes

Error rate is not always false positive rate or false negative rate; those condition on different actual classes. It is not generally a measure of error magnitude in regression. A low training error rate does not establish generalization. Zero observed errors does not prove zero population risk, especially with a small or unrepresentative sample. Word error rate is a task-specific edit measure and can exceed 100% because insertions contribute to the numerator; it should not be confused with a simple fraction of incorrect classified items.

Business use

Use error rate when a clear correct-or-incorrect outcome supports the decision being made. Report the numerator, denominator, evaluation period, and sample selection, not just a rounded percentage. Break out important error types and affected groups of cases. If the system abstains, report coverage as well so apparent improvement is not achieved merely by avoiding hard cases. For operational planning, combine frequency with severity and handling effort. Track changes using a consistent definition and investigate whether data composition shifted.

People also ask

Is error rate always one minus accuracy?

Yes for the same ordinary binary correctness rule, denominator, and weighting; not for every task-specific measure called an error rate.

Can a system lower error rate by refusing difficult cases?

It can lower error among answered cases, so coverage and the treatment of refusals must be reported.

Does 1% error mean one error in every group of 100?

No. It is an aggregate or estimated frequency, not a guarantee about every small batch.

Check your understanding

What is the difference between a drop from 7% to 5% and a 2% relative reduction?

Answer: The drop is 2 percentage points, or about 28.57% relative reduction. A 2% relative reduction from 7% would instead produce 6.86%.

Related terms

• Accuracy
• False positive rate
• False negative rate
• Word error rate
• Hamming loss
• zero-one loss
• misclassification
• coverage
• error analysis
• percentage points

Learn next

Take a small set of predictions and calculate both total and category-specific error rates. Then compare field-level and whole-record error definitions on the same data. Study uncertainty intervals and selective prediction next so observed error, expected error, and the fraction of cases actually handled remain distinct.

More in Causal inference and experimentation

• Disparate treatment - In machine-learning fairness discussions, disparate treatment refers to treating people differently through a decision process that uses sensitive group...
• Dueling bandit - A dueling bandit is a sequential learning problem in which the learner chooses pairs of options and receives relative feedback, such as which option was...
• E-optimal design - E-optimal design chooses an experimental allocation that maximizes the smallest eigenvalue of its information matrix.
• Estimand - An estimand is the precise quantity that a study aims to learn about.
• Evaluation - Evaluation is the systematic assessment of an AI model, output, or complete system against defined purposes and criteria.
• Evaluation function - An evaluation function is a rule or program that assigns a value to something being assessed.

Sources

Scikit-learn, zero_one_loss: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.zero_one_loss.html Scikit-learn, accuracy_score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html Scikit-learn, model evaluation guide: https://scikit-learn.org/stable/modules/model_evaluation.html

Previous: Error propagation Next: Error rate per comparison ( PC )