Is BLEU a percentage of correctly translated meaning?

No. It is an automatic metric based on modified n-gram overlap and a brevity penalty under its defined calculation.

Can evaluation work without a reference translation?

Some quality-estimation approaches use the source and output without a reference, but their predictions still need validation.

Why use human review if automatic metrics are available?

Human review can examine specific meaning errors, context, and suitability that a summary metric may not reveal.

AI Glossary / Natural language and speech / Intermediate

Evaluation of machine translation

Definition

Evaluation of machine translation is the systematic assessment of automatically translated text or translation systems. It examines whether the output preserves source meaning, reads appropriately in the target language, and serves its intended use. Evaluation can combine human judgments, automatic metrics, and task-based evidence; no single score captures every aspect of translation quality.

What it means

A translation can be fluent yet wrong. It may omit a negation, change a number, or assign an action to the wrong participant while sounding natural. Conversely, an awkward sentence may preserve the source’s important meaning.
Evaluation must therefore specify what quality means for the task. Dimensions can include accuracy, fluency, terminology, completeness, style, and consistency across a document.
The unit also matters. A segment-level score assesses an individual item, while an aggregate score summarizes a collection. A good average can conceal a small number of serious mistakes, and performance on one language pair or domain does not establish performance everywhere.

How it works

Define the use case, language direction, test material, and evaluation criteria before comparing systems. Use representative material that has not been used to tune the systems being assessed.
Human evaluators can judge meaning and language quality or annotate specific errors under a shared rubric. Provide appropriate source and document context, and examine disagreement rather than assuming every judgment is interchangeable.
Automatic metrics can compare outputs with references or estimate quality using learned models and available inputs. Their scores depend on the metric, version, preprocessing, and reference setup.
Compare systems on the same examples, report uncertainty where appropriate, and inspect error categories. Task-based evaluation can additionally ask whether users complete the intended activity accurately and efficiently.

Three practical examples

A localization team compares two systems on representative support articles using both error review and automatic scores.
A research group reports corpus-level translation metrics with the exact scoring configuration and test set.
A product team checks whether translated instructions preserve quantities, conditions, and references needed to perform the task.

Think of it this way

Think of evaluating a bridge between two messages: the destination should preserve what mattered at the starting point while remaining usable for its readers. The analogy is limited because translation often permits several valid phrasings, and quality cannot be reduced to matching one fixed sequence of words.

Worked example

Use an invented reference meaning: a meeting starts at 10:30 and registration closes at 10:00. Candidate A expresses both times correctly but uses awkward wording. Candidate B reads smoothly but says registration closes at 11:00.
A rubric that checks only target-language fluency might prefer B, while an accuracy review must identify its changed deadline. A surface-overlap metric could behave differently again depending on the reference wording; no actual metric score is claimed here.
Now imagine reviewers assess 40 segments and mark a major meaning error in 6. The observed proportion is 6/40, or 15%, under that rubric and sample. It is not automatically a prediction for every future translation.
The example shows why the report should retain error type and severity alongside summaries. A single aggregate score would not explain whether the system’s weakness lies in awkward style or incorrect operational facts.

Illustrative case study

In a fictional project, Hazel Manuals evaluates translated installation guides. One system produces smoother text, but reviewers find several omitted conditions in steps that apply only to certain device models.
The team adds explicit checks for conditions, numbers, and terminology, while continuing to assess readability. It reviews complete instruction sequences rather than isolated sentences where references become unclear.
The imagined project illustrates evaluation matched to use. It does not report a real system ranking or imply that a small checklist replaces professional review of every relevant quality dimension.

Common misconceptions and mistakes

A higher automatic metric score does not prove every translated sentence is better.
A reference translation is an example of an acceptable rendering, not necessarily the only valid wording.
Fluency is not the same as preservation of meaning.
Scores from different metric versions, tokenization settings, or test sets are not automatically comparable.
Human evaluation also needs a protocol: reviewer expertise, context, instructions, and disagreement can affect results.
An observed difference should not be treated as a reliable improvement without considering sample variation and the evaluation design.

Business use

Translation evaluation helps decide whether a system fits a workflow and where human editing is needed. It also provides feedback on recurring errors that broad scores can hide.
Create a test set reflecting the organization’s language pairs, content types, and difficult cases. Preserve the scoring configuration and review rubric with the results. Compare quality alongside relevant operational measures, such as editing effort or task success, while keeping those measures distinct. Re-evaluate when the model, source domain, terminology, or intended use changes materially.

Check your understanding

A translation preserves most wording but changes “do not open” to “open.” Should strong lexical overlap settle its quality assessment?

Answer: No. The changed negation reverses the instruction. Meaning-sensitive review must identify that error even if many words match a reference.

Related terms

Adequacy
Fluency
BLEU (Bilingual Evaluation Understudy)
BLEU
COMET
Quality estimation
Error annotation

Learn next

Compare accuracy and fluency on a few short translations before studying automatic metrics. Learn why several valid translations can differ in wording. Then design a small shared rubric and review disagreement, followed by aggregate scoring, uncertainty, and document-level evaluation.

Fluency

Sources

Papineni and colleagues, BLEU: a Method for Automatic Evaluation of Machine Translation: https://aclanthology.org/P02-1040.pdf Rei and colleagues, COMET: A Neural Framework for MT Evaluation: https://aclanthology.org/2020.emnlp-main.213/ Marie, Fujita, and Rubino, Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers: https://aclanthology.org/2021.acl-long.566/ Independent source review: BLEU’s modified-precision and brevity-penalty discussion, COMET’s introduction on learned evaluation and reference-free quality estimation, and Marie–Fujita–Rubino’s original abstract on evaluation comparability were consulted.

Previous: Evaluation of binary classifiers Next: Evaluator agent