AI Glossary / Causal inference and experimentation / Intermediate

Evaluation metric

Definition

An evaluation metric is a defined measure used to summarize an aspect of a model's or system's performance. Examples include accuracy, recall, mean absolute error, latency, and task-completion rate. A metric specifies what is counted or calculated and how results are combined. It provides evidence about a particular property under particular test conditions. No single metric automatically represents overall AI quality, suitability, or the value of the system to its users.

What it means

A metric turns a question into a repeatable measurement. “How often are the predicted categories correct?” suggests accuracy. “How many relevant cases are found?” suggests recall. “How large are the numerical errors?” calls for a regression measure. These are different questions, and a model can improve on one while worsening on another. Choosing a metric therefore requires deciding which property matters, what errors deserve attention, and how the measure relates to the actual task.

How it works

Define the evaluation unit and target property, then select a formula or scoring rule that captures it. Specify inputs, such as class labels, probability estimates, ranked items, or human judgments. State averaging, weighting, thresholds, missing-case handling, and score direction. Calculate the metric on appropriate evaluation data and inspect uncertainty and important subgroups. Check whether examples that receive good scores are actually useful for the intended purpose. Preserve the implementation and configuration so comparisons remain meaningful across model versions.

Three practical examples

• Rare-event classification: pair recall with precision to understand both missed relevant cases and unnecessary alerts.
• Numerical prediction: use mean absolute error to express average prediction deviation in the target's units, while separately inspecting large errors.
• AI assistance: measure successful task completion together with correction effort and response time, because fast answers can still be unhelpful.

Think of it this way

Think of a dashboard containing a fuel gauge, speedometer, and temperature reading. Each instrument measures a specific property, and none alone tells whether the whole journey is successful. The analogy explains why metrics complement one another. It also has a limit: AI metrics can depend on contested labels or proxies, so the measurement itself may need validation.

Worked example

Suppose a fictional test contains 100 messages, of which 10 require specialist review. Model A flags 5 messages and all 5 truly require review. Its precision is 5/5 = 100%, but recall is 5/10 = 50%. Model B flags 15 messages, including 9 of the 10 specialist cases and 6 routine cases. Its precision is 9/15 = 60%, while recall is 9/10 = 90%.
If the team ranks models only by precision, A wins; if it ranks only by recall, B wins. F1 combines precision and recall through their harmonic mean. A has F1 = 2 times 1 times 0.5 divided by 1.5, approximately 0.6667. B has F1 = 2 times 0.6 times 0.9 divided by 1.5 = 0.72. That composite favors B, but it still does not decide whether the six extra routine reviews fit the team's capacity. Metric selection clarifies a tradeoff; it does not remove the need to understand operational consequences.

Illustrative case study

A fictional drafting team initially uses response length as a proxy for completeness. The model begins producing longer answers, but reviewers find repeated explanations and missing key facts. The team replaces the proxy with a requirement-based rubric and keeps length only as a separate usability measure. It checks examples that score well and poorly to verify the new rubric's meaning. The lesson is that an easy-to-measure quantity can be a weak stand-in for the quality the team actually wants.

Common misconceptions and mistakes

A metric is not the same as the complete evaluation process. “Higher is better” is not universal: losses and error measures usually favor smaller values. Identical metric names can hide different averaging or normalization conventions. A composite score can conceal a serious failure in one dimension. Optimizing a proxy too aggressively can also produce behavior that exploits the measure while missing the intended objective. A precise numerical result does not compensate for an unrepresentative test set or unreliable reference labels.

Business use

Use metrics as a small, justified set tied to requirements. Name a primary measure when a comparison needs a clear objective, and retain additional measures or limits for properties that must not deteriorate. Report sample sizes, denominators, configurations, and relevant uncertainty. Keep results by important case type, rather than only an overall average. When changing a metric or its implementation, distinguish the measurement change from a model improvement and preserve enough information to reproduce earlier comparisons.

People also ask

Can the training loss also be an evaluation metric?

Yes, when it measures a relevant property, but the evaluation should still use suitable held-out data and may need additional measures.

Is a benchmark score a metric?

It is a result produced by a metric or scoring protocol on a particular benchmark.

Should every quality dimension be combined into one number?

No. Separate measures often make tradeoffs and unacceptable failures easier to see than a weighted average.

Check your understanding

Why does perfect precision not prove Model A is best in the worked example?

Answer: It finds only half of the cases requiring specialist review. Precision describes the correctness of flagged cases, while recall reveals the missed cases. The preferred tradeoff depends on the workflow.

Related terms

• Evaluation
• Accuracy
• Precision
• Recall
• Loss function
• Benchmarks
• Aggregation
• Uncertainty
• scoring rule
• F1 score
• proxy measure
• benchmark
• acceptance criterion

Learn next

Write a plain-language question for each metric you plan to report. Check its denominator, score direction, and assumptions, then construct an example where the metric looks good but the user outcome is poor. Study multi-metric evaluation and measurement validity next so the dashboard reflects actual requirements rather than convenient numbers.

• Evaluation

More in Causal inference and experimentation

• Error rate - Error rate is the proportion of evaluated units counted as incorrect under a defined rule.
• Estimand - An estimand is the precise quantity that a study aims to learn about.
• Evaluation function - An evaluation function is a rule or program that assigns a value to something being assessed.
• Exact match - Exact match is an evaluation rule that gives credit only when a prediction equals an accepted reference answer under the specified comparison procedure.
• Experiment - An experiment is a planned investigation in which a researcher deliberately changes one or more conditions and observes the resulting outcomes.
• Extrinsic evaluation - Extrinsic evaluation assesses an AI component or output by measuring its effect on a downstream task or wider application.

Sources

Scikit-learn, metrics and scoring guide: https://scikit-learn.org/stable/modules/model_evaluation.html NIST, AI measurement and evaluation: https://www.nist.gov/ai-measurement-and-evaluation NIST, AI RMF Core and measurement practices: https://airc.nist.gov/airmf-resources/airmf/5-sec-core/

Previous: Evaluation machine learning pipeline Next: Evaluation of binary classifiers