AI Glossary / Causal inference and experimentation / Intermediate

Extrinsic evaluation

Definition

Extrinsic evaluation assesses an AI component or output by measuring its effect on a downstream task or wider application. Instead of judging the component only on its own immediate properties, it asks whether using it helps the larger activity succeed. For example, a summary can be evaluated by how well readers answer questions using it. The approach connects evaluation to use, but results depend on the surrounding system, users, task, and experimental design.

What it means

A component can look strong in isolation without improving the process that uses it. A retrieval model may rank relevant passages well, yet an answer generator may fail to use them. A concise summary may read smoothly but omit the detail needed for a particular decision. Extrinsic evaluation tests these connections. It complements intrinsic evaluation, which examines properties such as retrieval relevance, translation overlap, or summary coherence directly. The distinction depends on which component is being assessed and what counts as its downstream use.

How it works

Identify the component, the downstream task, and the comparison condition. Integrate alternative component versions into a controlled workflow while keeping other factors consistent where possible. Measure outcomes such as task accuracy, completion, time, or correction effort. For human tasks, consider random assignment, balanced task difficulty, order effects, and participant differences. For automated pipelines, document training and integration choices. Analyze whether the observed difference can reasonably be attributed to the component or whether other changes could explain it. Keep diagnostic component-level measures to understand failures.

Three practical examples

• Summarization: evaluate whether readers can answer document questions accurately and efficiently from a summary.
• Language representations: compare embeddings through the performance of a downstream classifier trained under a consistent protocol.
• Retrieval systems: compare retrievers by their effect on a complete assistant's supported-answer success, in addition to ranking quality.

Think of it this way

Think of testing an ingredient by preparing the intended dish rather than tasting the ingredient alone. Its contribution depends on the recipe and the other ingredients. The analogy captures downstream usefulness, but a poor dish does not prove the ingredient is always bad; the integration, preparation, or chosen use may be responsible.

Worked example

A fictional study compares two summaries as aids for answering questions about the same type of report. Under a balanced evaluation design, users complete 100 scored tasks with summary A and 100 with summary B. They answer 82 tasks correctly with A and 88 with B. The observed downstream accuracy is therefore 82% versus 88%, a difference of 6 percentage points. Median completion time is 70 seconds with A and 85 seconds with B.
The results reveal a quality-time tradeoff: B supports more correct answers in this sample but takes longer. They do not show that B is universally better or that the difference is statistically reliable. If B was given only easier questions, the comparison would be confounded. If the same users saw both summaries for the same report, memory could affect the second condition. A defensible design balances or randomizes relevant factors and analyzes dependence appropriately. Separate summary-quality ratings can help explain the outcome, but they cannot replace the downstream measurements.

Illustrative case study

A fictional search team improves a retriever's intrinsic ranking score, then tests it inside an assistant. Answer success barely changes because the generator reads only the first few passages and ignores information spread across several sources. The team investigates the integration rather than declaring the retrieval work useless. It tests a revised passage-selection strategy and keeps both retrieval and answer-level metrics. Extrinsic evaluation identifies where component gains fail to translate into system gains, while intrinsic measures help locate the bottleneck.

Common misconceptions and mistakes

Extrinsic does not necessarily mean a live production experiment; a controlled laboratory task can assess downstream use. It does not always require human participants, because a component can feed another automated model. A downstream improvement does not automatically prove causation unless the comparison controls relevant alternatives. Results also do not transfer automatically to different users, tasks, or interfaces. A single end-to-end score can obscure why a component succeeds or fails, so abandoning intrinsic diagnostics would make interpretation harder.

Business use

Use extrinsic evaluation when component quality is valuable mainly through what it enables elsewhere. Choose outcomes that reflect the intended task and include relevant effort or time costs. Keep integration settings comparable and document necessary adaptations. Examine which user groups or task types benefit, and whether gains persist beyond the initial sample. Combine downstream results with component diagnostics before deciding what to improve. If deployment differs substantially from the evaluation setting, treat the result as bounded evidence rather than a blanket forecast of business impact.

People also ask

Can intrinsic and extrinsic scores disagree?

Yes. A component-level improvement may not affect the downstream bottleneck, or the application may need properties the intrinsic metric misses.

Is extrinsic evaluation always more expensive?

It often requires more integration or task design, but the effort depends on the system and can be modest for an existing automated pipeline.

Should it replace intrinsic evaluation?

Usually the two are complementary: one tests usefulness in context, while the other helps explain component behavior.

Check your understanding

Why is question-answering accuracy from a summary an extrinsic measure of that summary?

Answer: It assesses how the summary helps perform another task. A direct rating of the summary's coherence would instead assess a property of the summary itself.

Related terms

• Intrinsic evaluation
• Confounding
• Human evaluation
• downstream task
• end-to-end evaluation
• task success
• controlled experiment
• component ablation
• integration
• user study

Learn next

Choose one component in an AI workflow and list its immediate quality measures and the downstream outcome it is meant to improve. Design a comparison that changes the component while controlling other important factors. Then study experimental design and error attribution so a system-level difference can be interpreted without losing sight of integration effects.

More in Causal inference and experimentation

• Evaluation metric - An evaluation metric is a defined measure used to summarize an aspect of a model's or system's performance.
• Exact match - Exact match is an evaluation rule that gives credit only when a prediction equals an accepted reference answer under the specified comparison procedure.
• Experiment - An experiment is a planned investigation in which a researcher deliberately changes one or more conditions and observes the resulting outcomes.
• F 1 - F1, sometimes written F 1 or F1 score, combines precision and recall through their harmonic mean.
• False negative rate - False negative rate is the proportion of actual positive cases that a binary classifier incorrectly labels negative.
• False positive rate - False positive rate is the proportion of actual negative cases that a binary classifier incorrectly labels positive.

Sources

Pu, Gao and Wan, Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks: https://aclanthology.org/2024.lrec-main.821/ Fares and colleagues, The 2018 Shared Task on Extrinsic Parser Evaluation: https://aclanthology.org/K18-2002/ Cao and colleagues, On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations: https://aclanthology.org/2022.acl-short.62/

Previous: Extremal optimization Next: Extrusion detection