Must people judge the outputs?

No. Automated judge models can perform pairwise comparisons, but their reliability needs evaluation for the task and criteria being used.

Should the evaluator always force a winner?

No. Ties, insufficient evidence and both-unacceptable judgments can preserve information that a forced choice would hide.

Does winning more comparisons mean higher accuracy?

Not necessarily. It means better performance according to the chosen judgment process. Accuracy requires criteria and evidence that actually assess correctness.

AI Glossary / Causal inference and experimentation / Beginner

Side-by-side evaluation

Definition

Side-by-side evaluation compares two systems by judging their outputs for the same input. A reviewer sees the competing responses together and applies a stated criterion to decide which is better, whether they are tied, or whether neither meets the required standard.

What it means

The method is especially useful when there is no single exact answer, such as a summary, illustration or explanation. Comparing alternatives can reveal differences that a single overall score hides.

The comparison can concern two models, two prompts, two retrieval settings or two versions of the same system. To interpret the result, the evaluator must know what was changed and what conditions were held comparable.

A preference is relative. Choosing response A over response B does not establish that A is correct, acceptable for publication or suitable for every user. Both outputs may fail an important requirement even when one is clearly less poor.

How it works

Start with a representative set of inputs and a clear evaluation rubric. For example, a summary rubric might prioritize factual support, coverage and readability as separate dimensions. Generate both outputs using documented settings and the same relevant source material.

Hide system identities when those identities could influence judgment, and randomize or balance presentation order. Permit ties or unjudgeable cases where appropriate. Collect reasons linked to observable output features rather than asking only which response feels better.

Aggregate wins, losses and ties using an explicit convention. Inspect disagreements and results by task type. If a model acts as judge, compare its judgments with qualified human review on representative examples and test sensitivity to output order and presentation.

Three practical examples

An editorial team compares two summarization prompts against the same source articles, checking unsupported statements separately from writing quality.
A design team compares two generated illustrations for the same brief, distinguishing instruction adherence from aesthetic preference.
A support team compares draft replies from two system versions while giving reviewers the same customer question and policy reference.

Think of it this way

Think of tasting two recipes prepared for the same brief. Side-by-side tasting can make a difference easy to notice, but the verdict depends on what matters: sweetness, texture, ingredients or suitability for the occasion.

The analogy illustrates relative judgment and explicit criteria. AI evaluation often requires checking facts or instructions against external evidence, which cannot be replaced by taste. A more appealing answer can still be less accurate.

Worked example

Three honest ways to report the same comparisons

In an invented evaluation of 100 independent prompts, candidate A wins 48 comparisons, baseline B wins 32, and 20 are ties.

A's strict win proportion across all prompts is 48/100 = 48%. Among decisive comparisons only, its win proportion is 48/(48 + 32) = 60%. If the predeclared convention awards half a point for each tie, its score is (48 + 10)/100 = 58%.

These figures are compatible, but they answer different reporting questions. Calling all three “the win rate” without defining the denominator can mislead readers.

The counts also do not prove that A is better on every task. If the sample contains mostly short summaries, the result says little about complex document extraction. Uncertainty, prompt selection and dependence between repeated judgments must be considered before generalizing the outcome.

Illustrative case study

Fictional Willow Publishing compares two article-drafting systems. Reviewers initially prefer the more polished version in most pairings. A second pass asks them to verify each factual claim against supplied source notes, revealing unsupported details in several preferred drafts.

The team separates factual support from stylistic preference and permits “neither acceptable.” It also removes model names and balances which version appears first. Reviewers retain short explanations so disagreements can be examined.

The revised evaluation helps the team identify where each version needs improvement. It does not convert the earlier preference percentage into a claim of factual accuracy. The case illustrates how a comparison becomes more useful when its decision criterion matches the intended use.

Common misconceptions and mistakes

Do not confuse offline side-by-side judgment with an online A/B experiment measuring user outcomes. The methods can complement one another, but their evidence is different.

A longer answer can appear more helpful while adding irrelevant or incorrect material. Judge models can show position, verbosity or self-preference biases; human reviewers can also be influenced by presentation and expectations.

Repeated votes on one prompt are not the same as coverage of many independent tasks. Selectively displaying favorable comparisons or repeatedly tuning on the same evaluation set can also make the final result unrepresentative.

Business use

Use side-by-side evaluation to make a concrete change reviewable: a new model, a prompt revision or an updated retrieval process. Define acceptable trade-offs before examining results, including whether improved style can compensate for worse factual support.

Keep the prompt set, rubric, output versions, judge details and aggregation rule with the results. Review difficult and disputed cases, not just the headline preference. Pair relative comparisons with absolute requirements such as source faithfulness, required fields or successful task completion. A system should not pass solely because its alternative is worse.

Check your understanding

Reviewers prefer A on 70% of comparisons, but the rubric asks only which answer is more engaging. Can the team report that A is 70% accurate?

Answer: No. The result concerns engagement preference relative to B, not the fraction of factually correct answers. The criterion, comparison partner and aggregation rule must accompany the number.

Related terms

Human evaluation
Inter-rater agreement
A/B testing
Pairwise evaluation
model-as-judge
evaluation rubric
win rate
position bias

Learn next

Write a short rubric for one real task and independently review a few output pairs. Compare disagreements before expanding the study. Then learn how to report ties, uncertainty and subgroup results, and distinguish relative preference from absolute acceptance requirements.

Sources

Sources: Google Machine Learning Glossary, side-by-side evaluation: https://developers.google.com/machine-learning/glossary#side-by-side-evaluation Zheng and colleagues, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685 Google Cloud, interpreting pairwise evaluation results: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/eval-python-sdk/view-evaluation Google Cloud, evaluating a judge model against human ratings: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/evaluate-judge-model

Previous: Sibyl attack Next: Side sequence