Must reference text be written by a human?

No. Some workflows use model-generated references, but their suitability and correctness still need validation.

Can a response differ from the reference and still be correct?

Yes, especially for open-ended tasks, paraphrases, and translations with several acceptable formulations.

Is a reference required for every evaluation?

No. Some methods use task outcomes, source-grounded checks, rubrics, or other evidence without a prepared reference response.

AI Glossary / Generative AI and language / Beginner

Reference text

Definition

Reference text is a prepared example of an appropriate response used when evaluating generated text. It may be an expert translation, a summary, a transcript, or another task-specific answer. An evaluation metric compares a model's output with this reference under defined rules. The reference is an evaluation aid; it is not automatically the only acceptable wording or a guarantee that every detail is correct.

What it means

Suppose a model translates a sentence. To evaluate its output, a team may compare it with a translation prepared and checked by a qualified reviewer. That translation is reference text; the model's translation is the candidate or generated text.

Some tasks permit many valid responses. A reference summary may emphasize one set of important facts while another equally useful summary expresses them differently. The evaluation must account for that flexibility rather than treating every wording difference as an error.

A task can have one reference, several references, or an evaluation method that does not use reference text. The appropriate design depends on what quality means for the application.

How it works

Begin with a clear input and task specification. Prepare reference responses that follow the required content, language, format, and scope. Review them for errors and record how they were created.

Generate candidate responses under a documented procedure. Apply the chosen comparison method consistently, including tokenization, normalization, handling of multiple references, and aggregation across examples. Metrics such as ROUGE measure particular forms of overlap; other evaluators may assess meaning or task criteria.

Inspect disagreements between scores and human judgments. A low overlap score might reflect a valid paraphrase, while a high score can conceal a changed number or negation. Reference quality and metric choice jointly determine what the resulting evaluation can support.

Three practical examples

1. A translation team prepares reviewed reference translations for a held-out set of customer messages and compares candidate systems against the same material.

2. A transcription project uses checked transcripts as references, with a documented policy for punctuation, fillers, and spelling variants.

3. A summarization team creates reference summaries and a separate factual checklist so that acceptable alternative wording is not the only consideration in evaluation.

Think of it this way

Think of reference text as an example answer supplied to an examiner. It helps make comparisons consistent, but a thoughtful examiner still considers whether a different answer satisfies the question.

The analogy has limits. An automatic comparison may count shared tokens without understanding why two responses differ. It may also fail to notice that the example answer itself contains a mistake. Reference-based evaluation therefore needs an explicit interpretation, not merely an answer key.

Worked example

Take the reference “the parcel arrived on Tuesday” and two candidate outputs. Candidate A says “the parcel arrived on Wednesday”. Candidate B says “delivery took place Tuesday”. For this toy comparison, lowercase all words and use exact unigram matches with clipped counts.

The reference contains five words. Candidate A shares four, giving overlap recall 4/5 = 0.80. Candidate B shares only “Tuesday” after lowercasing, giving recall 1/5 = 0.20.

Yet A changes the day, while B can express the relevant event correctly if “delivery” refers to the parcel. The calculation shows what a simple overlap measure counts and what it misses. It does not establish a complete ranking of summary quality. A factual check and the original source are needed to assess the changed date and the paraphrase.

Illustrative case study

Fictional case: Maple Briefs evaluates a tool that summarizes weekly project notes. Its first reference set contains concise summaries written by one editor. The model sometimes produces accurate summaries with different wording and receives unexpectedly low overlap scores.

The team reviews a sample of these disagreements. It discovers both valid paraphrases and genuine omissions, so it avoids dismissing the metric entirely. Instead, it adds a content checklist and clearer reference-writing guidance.

The revised evaluation reports overlap alongside factual coverage and unsupported claims. The reference remains useful, but the team no longer presents a single similarity score as proof that every summary is ready to publish.

Common misconceptions and mistakes

Reference text is not necessarily ground truth in an absolute sense. Human experts can disagree or make errors, and model-produced references need appropriate validation.

Another mistake is using evaluation references as training examples and then treating the resulting test as independent. Leakage can make performance appear better than it will be on new inputs.

Multiple references can represent some acceptable variation, but they do not enumerate every valid answer. Nor does high similarity prove faithfulness to the original source: both candidate and reference could repeat the same unsupported statement.

Business use

Good reference material is a reusable evaluation asset. It helps teams compare model versions consistently, investigate regressions, and make quality expectations concrete.

Budget for reference creation, review, versioning, and maintenance rather than treating it as a one-time clerical step. If business rules change, determine whether the references must change too. Record the input, reference provenance, evaluation criteria, and known ambiguities so later comparisons remain interpretable. Use references together with checks that directly address the application's costly mistakes.

Check your understanding

A summary has a high word-overlap score against a reference but changes “approved” to “not approved”. Should the score alone establish quality?

Answer: No. A small wording change can reverse the meaning while leaving most words unchanged. Evaluate the relevant fact directly against the source and task requirements.

Related terms

Generated text
Golden response
Ground truth
ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
BLEU (Bilingual Evaluation Understudy)
Human evaluation
Data leakage
Semantic similarity
candidate response
evaluation dataset
ROUGE
BLEU

Learn next

Select five representative prompts and prepare reviewed references with explicit quality criteria. For each, write one valid paraphrase and one subtly incorrect response. Check whether the proposed evaluation distinguishes them. Use the disagreements to refine the reference guidance and add complementary checks before comparing model versions.

Sources

Sources: Google Machine Learning Glossary, https://developers.google.com/machine-learning/glossary — primary definition of reference text and distinctions around generated text and golden responses. Lin, ROUGE: A Package for Automatic Evaluation of Summaries, https://aclanthology.org/W04-1013.pdf — original paper on reference-based overlap metrics and multiple-reference treatment. Relevant sections were read. The parcel calculation is an intentionally simple illustrative comparison, not a claim about every ROUGE implementation.

Previous: Reference prior Next: Referent