Can a model judge itself?

It can evaluate its own outputs, but shared weaknesses and preferences make independent checking especially important.

Does the judge need a reference answer?

Not always. Some rubrics evaluate qualities directly; factual tasks often benefit from trusted evidence or references.

Is a score of eight out of ten comparable across judges?

Not automatically. Different prompts, models, and examples can produce different interpretations of the scale.

AI Glossary / Generative AI and language / Intermediate

LLM-as-a-judge

Definition

LLM-as-a-judge is an evaluation method in which a large language model assesses an answer, conversation, or other output using specified criteria. The judge may assign a score, identify errors, or compare two candidates. Its judgment is another model-generated output, so it needs validation. A confident explanation from the judge does not make its assessment an objective fact.

What it means

Some language tasks have many acceptable answers. A summary can be accurate without matching a reference sentence word for word, and two explanations may differ in clarity while conveying the same facts. A language-model judge can help evaluate such qualities at a scale that would make fully manual review expensive. However, the method transfers part of the evaluation problem to a second model: does that model understand the rubric, use the evidence properly, and distinguish a polished answer from a correct one? The judge's suitability must be demonstrated for the actual task.

How it works

A typical judging prompt supplies the original task, the candidate response, a rubric, and any relevant reference material. In pointwise evaluation, the judge assesses one response on its own. In pairwise evaluation, it compares two responses, ideally allowing a tie when appropriate. Structured outputs can separate criteria such as factual support, completeness, and clarity. Before large-scale use, compare the judge with carefully reviewed examples and inspect disagreements. When comparing pairs, reversing their presentation order can reveal position sensitivity. Keep model identity, prompt, reference versions, and generation settings in the evaluation record. Changes to any of these can change scores even if the candidate responses stay the same.

Three practical examples

A support team uses a judge to flag responses that omit a required troubleshooting step, then reviews uncertain cases.
A writing team compares two summaries for whether they preserve the document's main conclusions without introducing unsupported claims.
An education team screens explanations for age-appropriate language while subject experts check correctness. The model's role is bounded by the criterion and evidence supplied, rather than treated as universal expertise.

Think of it this way

Think of a second editor reviewing a draft against a checklist. This can uncover problems the writer missed, but the editor may have blind spots or preferences of their own. A detailed editorial note is useful only if it accurately identifies a problem. The analogy has limits: a model judge generates its review from learned patterns and the supplied context, and it can misunderstand evidence or invent a rationale. Its apparent fluency should not be confused with independent verification.

Worked example

Suppose human reviewers establish a reference assessment for 100 answers: eighty are acceptable and twenty are unacceptable. A model judge accepts seventy-four of the eighty acceptable answers and wrongly accepts twelve of the twenty unacceptable answers. It therefore rejects six acceptable answers and correctly rejects eight unacceptable answers. Overall agreement is (74 + 8)/100 = 82%. That sounds promising, but the judge misses 12/20 = 60% of the unacceptable answers. Among the eighty-six answers it accepts, twelve are unacceptable according to the reference assessment, or about 14.0%. The team should examine the error pattern rather than relying on agreement alone. The human reference also needs a clear rubric and disagreement resolution; it is a reviewed comparison standard, not an assumption that every human judgment is infallible.

Illustrative case study

Fictional case: Cedar Knowledge compares two versions of a document assistant. Its judge consistently prefers the longer answer, including when that answer adds unsupported details. The team introduces separate factual-support and completeness checks and supplies the relevant document passages. It tests pairs in both orders and audits cases where the judge's explanation cites text that is absent. The revised evaluation helps identify which responses require expert review. No improvement percentage is assumed here; the case illustrates how an evaluation process can be repaired when a judge rewards the wrong feature.

Common misconceptions and mistakes

Known concerns include position bias, preference for verbosity, and favorable treatment of outputs resembling the judge's own style or model family. A model can also lack the reasoning ability needed to assess a difficult solution. Asking it to explain a score does not prove that the explanation caused or justifies the score. Candidate text may contain instructions aimed at influencing the judge, so evaluation content must be treated as material to assess, not as authority over the grading procedure. Multiple judges can add perspectives, but agreement between similar models may reflect shared errors rather than independent confirmation.

Business use

Use model judging where a clear rubric exists and automated assessment can be checked against representative reviewed examples. Retain direct checks for measurable properties: arithmetic, required fields, executable behavior, and database outcomes may be better tested with dedicated logic. Track false acceptance and false rejection separately when their consequences differ. For recurring evaluations, maintain a stable calibration set and review whether the judge's behavior changes after a model update. Escalate ambiguous or consequential disagreements to qualified reviewers. The goal is a useful measurement process, not simply a large volume of inexpensive scores.

Check your understanding

A judge agrees with human reviewers on 95% of examples, but nearly all examples are easy successes. Does that establish that it reliably detects serious failures?

Answer: No. Test and report its performance on the failure types that matter, including false acceptance rates and sufficiently varied difficult cases.

Related terms

Human evaluation
Inter-rater agreement
Benchmarks
evaluation rubric
pairwise comparison
position bias
benchmark
calibration set
factuality evaluation

Learn next

Select ten candidate answers with known strengths and defects. Write separate criteria for correctness and presentation. Compare a judge's decisions with reviewed assessments, reverse pair order where applicable, and record disagreements by error type before using aggregate scores for a business decision.

Sources

Zheng and colleagues, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685 Anthropic, Demystifying evals for AI agents: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Previous: Ljung-Box test Next: LLM evaluations (evals)