Is METEOR limited to exact word matches?

No. Depending on configuration and resources, it can recognize stems, synonyms, and paraphrases as well.

Why penalize fragmentation?

The penalty reflects whether matched words form ordered sequences instead of isolated or rearranged matches, adding information beyond match counts alone.

Can it evaluate without a reference translation?

Standard METEOR is reference-based. Other reference-free evaluation methods answer a different question and should not be labeled METEOR without a clearly defined extension.

AI Glossary / Causal inference and experimentation / Intermediate

METEOR

Definition

METEOR is an automatic text-evaluation metric developed for machine translation. It aligns a candidate translation with one or more reference translations and scores the resulting matches using precision, recall, and a penalty related to fragmented word order. Depending on the version and language resources, matching can include exact words, stems, synonyms, and paraphrases. A higher score generally indicates closer agreement under that configuration, not a guarantee of accurate meaning or a substitute for human review.

What it means

Translation rarely has only one acceptable wording. A metric based solely on identical surface forms can miss reasonable alternatives. METEOR addresses part of this problem through alignment rules that can recognize more than exact identity, while still using reference translations as its comparison basis.

It also considers whether matched words form coherent sequences rather than appearing in a scattered order. These design choices make it a useful perspective on translation quality. Nevertheless, lexical matches and their arrangement cannot capture every distinction in meaning. A small changed word, negation, name, or number can matter greatly even when much of the sentence still aligns.

How it works

Preprocess the candidate and reference according to the chosen implementation, then find permitted matches. From the alignment, calculate precision relative to candidate content and recall relative to reference content. Combine them using the version's weighting scheme.

Apply a fragmentation penalty based on how the matched material breaks into ordered chunks, then compute the final score. Multiple references and system-level aggregation also follow implementation-specific rules.

Record the version, language, resources, tokenization, normalization, and parameter settings when reporting results. The original 2005 formulation provides a clear teaching example, but later METEOR versions add matching resources and tuned parameters. A calculation using the early formula should not be presented as the exact output of every later implementation.

Three practical examples

1. Translation comparison: researchers evaluate alternative systems on the same source sentences and references using an explicitly fixed METEOR configuration.

2. Error investigation: analysts inspect alignments for low-scoring translations to see whether the issue is missing content, differing wording, or fragmented matches.

3. Evaluation education: students compare exact-match-only examples with stem or synonym matching to understand how lexical resources alter what a metric recognizes as similar.

Think of it this way

Think of comparing two descriptions by linking corresponding words and checking how neatly the links preserve their order. More useful links and less fragmentation improve the assessment. The analogy is limited because connected words do not prove equivalent meaning. A fluent but false translation can share many words with a reference, while a valid paraphrase may use wording the matching resources do not recognize.

Worked example

Use the original 2005 scoring formula in a simplified exact-match exercise. The reference is “the robot moved very slowly” and the candidate is “the robot moved slowly.” Assume these are five and four lowercase tokens respectively, with no punctuation.

All four candidate words match, so precision P is 4/4 = 1 and recall R is 4/5 = 0.8. The original weighted combination is 10PR/(R + 9P), giving 8/9.8, approximately 0.8163.

There are two contiguous matching chunks: “the robot moved” and “slowly.” The original penalty is 0.5 × (2/4)³ = 0.0625. The score is 0.8163 × (1 − 0.0625), approximately 0.7653. This illustrates the historical formula and declared alignment, not a universal current-tool output or a complete judgment of translation quality.

Illustrative case study

Fictional case: Linden Translate compares two systems for product instructions. One scores better on METEOR, but bilingual reviewers find a few examples where a short negation changes the instruction's meaning. Much of each sentence still matches the reference, so the aggregate score does not expose the practical seriousness of those errors.

The team retains METEOR for consistent automatic comparison and adds targeted human checks for critical meaning changes. It also fixes its tokenization and language resources before rerunning the comparison. The revised report presents metric evidence with error examples rather than treating a higher score as automatic permission to publish every translation.

Common misconceptions and mistakes

Do not assume that all scores called METEOR are directly comparable. Versions, resources, parameter settings, references, and preprocessing can differ.

Do not treat the metric as a factuality checker or read a score of 0.8 as meaning 80% of sentences are correct. It is a composite alignment-based measure.

Avoid assuming a corpus score is always the arithmetic mean of sentence scores. Aggregation rules must be checked. Also remember that incomplete or narrow references can penalize acceptable alternatives, while extensive overlap can conceal a consequential semantic error.

Business use

METEOR can provide repeatable evidence during translation-system development, helping teams compare changes and find examples worth inspecting. It is most informative when the evaluation data and configuration reflect the language and content being handled.

Business decisions should also account for the costs of specific translation errors, reviewer expertise, and variation across document types. Keep references and scoring settings stable across comparable runs, and disclose changes when they are necessary. Use automatic scores to support prioritization and diagnosis while evaluating whether translated material preserves the meaning needed for the intended audience and task.

Check your understanding

Two teams report different METEOR scores for the same candidate text. What should be checked before concluding that one calculation is wrong?

Answer: Check the references, version, language resources, preprocessing, matching modules, parameters, and aggregation. Different valid configurations can yield different scores.

Related terms

Alignment
Precision
Recall
BLEU (Bilingual Evaluation Understudy)
Human evaluation
Machine translation evaluation
reference translation
fragmentation penalty
BLEU
paraphrase

Learn next

Calculate the declared exact-match example, then add a valid alternative wording and inspect whether the alignment changes. Compare the metric's judgment with a bilingual review of meaning. Record a complete scoring configuration so another person can reproduce the comparison rather than relying on the metric name alone.

Sources

Primary sources: Banerjee and Lavie, METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments, https://www.cs.cmu.edu/~alavie/METEOR/pdf/Banerjee-Lavie-2005-METEOR.pdf ; CMU METEOR project, https://www.cs.cmu.edu/~alavie/METEOR/ . The original formula and project description of later matching capabilities were consulted. The calculation is explicitly historical and illustrative; the fictional deployment scenario is not a reported result.

Previous: Metastore Next: Method error