AI Glossary / Causal inference and experimentation / Intermediate

BLEU (Bilingual Evaluation Understudy)

Definition

BLEU, short for Bilingual Evaluation Understudy, is an automatic metric originally designed to compare machine translations with human reference translations. It combines clipped word-sequence matching at several lengths with a penalty for outputs that are too short. Higher values indicate more overlap under the specified evaluation setup. BLEU is useful for controlled comparisons, but it does not directly determine whether a translation preserves meaning, is factually correct, or is suitable for a particular reader.

What it means

There can be many valid ways to translate a sentence. BLEU uses one or more reference versions as examples of acceptable wording, then measures how much a system shares with them. A sequence of n adjacent words is an n-gram. Matching longer sequences captures more local word order than matching individual words. However, equivalent expressions can differ substantially in wording, and similar wording can conceal an important error. The score is therefore a limited proxy for translation quality.

How it works

Tokenize candidate and reference texts under a declared procedure. Count candidate n-gram matches while clipping each count to the maximum count found in a single corresponding reference. Pool counts across the corpus to obtain modified precisions. Combine the chosen n-gram precisions with a weighted geometric mean and multiply by a brevity penalty based on candidate and effective reference lengths. A common BLEU configuration uses orders one through four with equal weights. Record tokenization, case treatment, references, smoothing, and score scale so the result can be reproduced.

Three practical examples

• Translation development: compare two system versions on the same held-out source sentences and references.
• Regression checks: identify whether a change unexpectedly lowers reference overlap in a particular language pair, then inspect the affected translations.
• Evaluation reporting: publish the metric configuration alongside scores so another team can distinguish a model change from a scoring change.

Think of it this way

Think of comparing a musical performance with a written arrangement by checking short note sequences. Shared sequences reveal similarity, but do not fully establish musical quality or whether a different arrangement is equally good. Likewise, BLEU observes textual overlap; it cannot independently judge every meaningful variation or detect every consequential change.

Worked example

For a deliberately simplified BLEU-2 example, use the reference “we deliver fresh bread daily” and candidate “we deliver bread daily”. Treat spaces as token boundaries, keep lowercase text, use equal unigram and bigram weights, and apply no smoothing. All four candidate words occur in the reference, so modified unigram precision is 4/4 = 1. The candidate bigrams are “we deliver”, “deliver bread”, and “bread daily”. Two match the reference, giving bigram precision 2/3.
Candidate length c is 4 and reference length r is 5. The brevity penalty is exp(1 - r/c) = exp(-0.25), approximately 0.7788. The geometric mean is the square root of 1 times 2/3, approximately 0.8165. Multiplying gives about 0.6359, or 63.59 on a 0-100 scale. This is a teaching example with two n-gram orders, not a standard corpus BLEU-4 result. The omitted word “fresh” also shows why even a fairly high overlap score does not certify that every part of the source meaning was retained.

Illustrative case study

A fictional localization team finds that a revised translator produces more natural wording but slightly lower BLEU against one reference set. Reviewers inspect a sample and find both legitimate paraphrases and real omissions. The team adds human error analysis instead of dismissing either the metric or the reviewers. It keeps the original references fixed for historical comparisons and clearly labels any separate evaluation with expanded references. This allows the score to remain reproducible while the team learns what the score fails to explain.

Common misconceptions and mistakes

BLEU is not the percentage of sentences translated correctly. A score of 40 does not mean 40% of meaning is preserved. Corpus BLEU is not generally the arithmetic average of sentence BLEU. Without smoothing, a zero precision for an included n-gram order drives the geometric mean to zero. Different tokenizers, reference counts, and case settings can change results. A strong BLEU result also does not establish factual accuracy in open-ended answers, where reference overlap may be a poor fit for the task.

Business use

Use BLEU as one reproducible signal when comparing translation systems under a consistent protocol. Pair it with human assessment of omissions, terminology, meaning changes, and readability where those properties matter. Report results by language pair or domain when an aggregate might hide uneven quality. Preserve the evaluation configuration and test-set identity. A small numerical improvement should prompt uncertainty analysis and example inspection before it becomes a claim about better customer experience or reduced editing effort.

People also ask

Can a good human translation receive a low BLEU score?

Yes. Valid wording can differ from the available references.

Why clip repeated word counts?

Clipping prevents unlimited credit from repeating words that appear only a few times in the references.

Why penalize short outputs?

A candidate can achieve high precision by producing only a small matching fragment, so the length penalty discourages that shortcut.

Check your understanding

If every word in a short candidate appears in the reference, must BLEU equal its maximum?

Answer: No. Longer n-grams may fail to match, and a shorter candidate can receive a brevity penalty. The worked example has perfect unigram precision but a lower combined score.

Related terms

• Machine translation
• N-gram
• Tokenization
• Smoothing
• Human evaluation
• modified precision
• brevity penalty
• reference translation
• corpus evaluation
• chrF

Learn next

Calculate unigram and bigram matches for a few short translation pairs, including repetition and a valid paraphrase. Then compare sentence-level and corpus-level aggregation using a documented implementation. Study character-based and learned translation metrics next, while keeping human error categories visible so metric agreement is not confused with complete quality assurance.

More in Causal inference and experimentation

• Average treatment effect - The average treatment effect, or ATE, is the average difference between the outcome a defined population would experience under one intervention and the...
• Average treatment effect on the treated - The average treatment effect on the treated, or ATT, is the average causal effect of an intervention among the units that actually received it.
• Balanced accuracy score - Balanced accuracy score measures classification performance by averaging the recall of each class with equal class weight.
• Box-Benkhen experiment - A Box - Behnken experiment uses a response-surface design to study how several controllable factors affect an outcome and to fit a quadratic model.
• Brier score - The Brier score measures the quality of probability predictions by averaging squared differences between predicted probabilities and observed outcomes.
• Calinski-Harabasz Index - The Calinski-Harabasz Index is an internal clustering score that compares separation between cluster centers with variation inside clusters, adjusted for...

Sources

Papineni and colleagues, BLEU: a Method for Automatic Evaluation of Machine Translation: https://aclanthology.org/P02-1040/ Original BLEU paper: https://aclanthology.org/P02-1040.pdf SacreBLEU, reference implementation and reporting documentation: https://github.com/mjpost/sacrebleu

Previous: Blank node Next: Blind deconvolution