Is a proxy metric inherently bad?

No. Many useful measurements are indirect. The issue is whether the evidence supports the inference being made from the proxy.

Can a proxy become less useful over time?

Yes. Users, products, models, and optimization strategies can change the relationship between the measure and the intended outcome.

Is a training loss a proxy metric?

It can serve as a proxy for practical task quality, but the terms describe different roles. A loss is an optimization criterion; a proxy is an indirect indicator of another goal.

AI Glossary / Causal inference and experimentation / Beginner

Proxy metric

Definition

A proxy metric is a measurable quantity used as an indirect indicator of an outcome or quality that is harder to measure directly. For example, clicks on recommended material might be used as a proxy for user interest, while a text-overlap score might stand in for some aspects of summary quality. A proxy is useful only to the extent that it reliably informs the intended judgment under the relevant conditions.

What it means

Important outcomes may arrive slowly or require expensive assessment. A service might seek lasting customer satisfaction but need feedback from a short experiment. An AI team might lack expert judgments for every generated summary.

A proxy can make evaluation faster and more scalable. The difficulty is that the measurable substitute and the desired outcome are not identical. A customer may click because an item is useful, confusing, or misleading. A summary may overlap a reference while omitting an important qualification.

Calling a measure a proxy invites evidence about that gap rather than equating a convenient number with the goal.

How it works

Begin by stating the underlying outcome clearly. Then identify candidate measures that can be collected and explain why they might track that outcome. Evaluate the relationship using direct assessments, historical evidence, or experiments appropriate to the claim.

Check whether the relationship holds across important populations and conditions. For decision-making, it matters whether changes that improve the proxy also improve the intended outcome, not merely whether high-proxy users happen to have better outcomes in existing data.

Use the proxy with complementary measures and periodically reassess it. A model or product change can exploit features of the proxy that were previously unimportant. Record the time window, measurement process, known blind spots, and conditions under which the proxy should not be relied on alone. No formula validates every substitute.

Three practical examples

1. Recommendation evaluation: clicks provide a quick signal of engagement, while longer-term usefulness or satisfaction requires additional evidence.

2. Generated-text assessment: automatic overlap or similarity scores help screen many outputs, with human review used to examine meaning, completeness, and factual consistency.

3. Learning support: exercise completion rates offer an early signal of participation, while actual learning may require later assessments of understanding and retention.

Think of it this way

Think of using a fuel gauge to estimate how far a vehicle can travel. The gauge is informative, but range also depends on consumption, route, and conditions. The analogy is limited because some AI proxies have a much weaker or less stable connection to their intended outcome than a fuel gauge has to remaining fuel. The relationship needs evidence, not just an appealing story.

Worked example

Suppose a fictional help system compares two versions using 100 sessions per version. Version A produces 60 article clicks and 50 successfully completed tasks. Version B produces 80 clicks but only 40 completed tasks.

The click count rises by 20, a relative increase of 20/60, approximately 33.3%. Task completion falls from 50% to 40%, a decrease of 10 percentage points. If clicks were the only proxy used for helpfulness, the team might declare an improvement even though the observed direct outcome moved in the opposite direction.

These counts do not explain why. More clicks could reflect exploration, confusion, or another difference in the sessions. A proper comparison also needs a study design and uncertainty assessment. The arithmetic simply shows that improvement in a proxy and improvement in the intended outcome are logically different claims.

Illustrative case study

Fictional case: Cypress Writing evaluates an AI summarizer with a reference-overlap metric because it is inexpensive to run. Reviewers notice that copying long phrases from the source can raise the score without producing a concise or complete summary.

The team keeps the automatic measure for a limited screening role and adds judgments of important-fact coverage and unsupported claims. It records cases where the measures disagree and uses them to refine evaluation. The case illustrates how a proxy can remain useful while its role becomes more carefully bounded. It is not a claim about a real product or a demonstrated performance improvement.

Common misconceptions and mistakes

A correlated measure is not automatically a valid decision proxy. Correlation across users can arise from other factors, and an intervention may change the relationship. Historical association is evidence to examine, not a guarantee about future improvements.

Do not assume that more precise measurement of a proxy fixes a mismatch with the goal. Counting clicks perfectly still does not make every click beneficial. Likewise, optimizing a proxy very aggressively can expose weaknesses that were less visible when the measure was only observed.

Combining several proxies can broaden coverage but does not automatically solve the problem. The combination has its own weights and blind spots. Validate what the combined score actually supports and retain direct outcome checks when feasible.

Business use

Proxy metrics help businesses run faster development cycles and monitor large systems. They are valuable when the direct outcome is delayed or expensive, provided their limitations are visible to decision-makers.

For each important proxy, maintain a short rationale: the intended outcome, evidence linking them, and plausible ways they could diverge. Make major decisions with stronger evidence when the stakes or uncertainty warrant it. A useful dashboard distinguishes early indicators from confirmed outcomes rather than presenting every upward movement as business success.

Check your understanding

An AI assistant's average response length increases, and a team calls that proof of greater helpfulness. What evidence is missing?

Answer: Evidence that longer responses improve the intended user outcome in this setting. Length may instead reflect repetition or unnecessary detail, so direct quality or task-success checks are needed.

Related terms

Learn next

Choose one AI feature and name its real goal. Propose a convenient proxy, then write two scenarios where the proxy improves while the goal worsens. Design a direct assessment that could detect those failures.

Sources

Tripuraneni and colleagues, Choosing a Proxy Metric from Past Experiments: https://arxiv.org/abs/2309.07893

Jurafsky and Martin, Speech and Language Processing, discussion of intrinsic language-model evaluation and downstream performance: https://web.stanford.edu/~jurafsky/slp3/3.pdf

The sources distinguish convenient indicators from intended outcomes. Examples, arithmetic, case, and exercise are original. No automatic text metric is claimed to measure every aspect of quality.

Previous: Proxy labels Next: Proxy (sensitive attributes)