AI Glossary / Causal inference and experimentation / Intermediate

Character N-gram F-score

Definition

Character N-gram F-score evaluates generated text by combining precision and recall of matching character sequences against reference text. In machine translation, the best-known form is chrF, introduced by Maja Popović. It captures partial matches within words as well as longer shared strings, making it less dependent on word boundaries than word-based overlap measures. Higher scores indicate greater character-sequence agreement under the chosen settings, not a guarantee that meaning or factual content is correct.

What it means

A character n-gram is a sequence of n adjacent characters. The word “cats” contains character bigrams “ca”, “at”, and “ts”. Comparing these pieces can recognize overlap between related word forms even when whole words differ. Precision asks how much of the candidate's character material matches the reference; recall asks how much of the reference is covered. Their combination balances excess material and omissions, although surface resemblance remains different from understanding the text.

How it works

Choose the character orders, preprocessing rules, and reference handling. Count overlapping n-grams in candidate and reference strings, using matching multiplicities rather than giving unlimited credit to repetitions. Compute precision and recall for each selected order, then aggregate according to the stated chrF implementation. The original formulation combines averaged precision P and recall R as (1 + beta squared)PR/(beta squared P + R). Beta controls the precision-recall tradeoff. Character-only chrF and chrF++, which adds word n-grams, are related but distinct configurations that should not be reported interchangeably.

Three practical examples

• Translation comparison: assess two translations against the same references when inflected words may share meaningful character sequences.
• Error exploration: compare character-level and word-level overlap to identify outputs whose scores differ because of spelling or word-form variation.
• Regression evaluation: track a fixed chrF configuration alongside human review when a text-generation system changes its output wording.

Think of it this way

Think of comparing two mosaics by looking for matching small strips of tiles. Shared strips can remain visible even when larger sections differ. The analogy explains partial character overlap, but matching pieces do not establish that the full picture conveys the same message. A small change such as a negation can matter far more than its size suggests.

Worked example

For a deliberately simplified character-bigram-only example, use candidate “cat” and reference “cats”, with lowercase text and no spaces. The candidate has two bigrams: “ca” and “at”. The reference has three: “ca”, “at”, and “ts”. Two match. Precision is 2/2 = 1, and recall is 2/3. With beta = 1, the F-score is 2 times 1 times (2/3), divided by 1 + 2/3, which equals 0.8.
With beta = 2, the formula becomes 5PR/(4P + R), giving (10/3)/(14/3) = 5/7, approximately 0.7143. Because recall is lower than precision here, emphasizing recall lowers the result. This is a teaching calculation for one n-gram order, not a full standard chrF score across several orders. It demonstrates partial credit for shared characters while also exposing a limitation: the candidate omits the plural marker. Whether that omission is harmless or changes an important instruction cannot be decided from overlap alone.

Illustrative case study

A fictional multilingual help-center team evaluates revised translations with chrF and a word-based metric. Some word-form changes receive more partial credit under character scoring, which makes the differences easier to inspect. Reviewers still identify a few short omissions that alter the instruction. The team keeps those errors in a separate review category and records the character order, beta, case treatment, and reference version used for each run. The score becomes a reproducible comparison aid rather than a claim that every high-scoring sentence is safe to publish.

Common misconceptions and mistakes

Character overlap is not semantic equivalence. Similar-looking words can mean different things, while good translations can use entirely different strings. The metric is less dependent on word tokenization, but preprocessing still matters: case, whitespace, Unicode normalization, and the definition of a character can affect results. Different beta values or the addition of word n-grams create different measures. A corpus score need not equal a simple average of independently computed sentence scores, so the aggregation procedure also belongs in the report.

Business use

Use character n-gram F-scores when a lightweight, reproducible reference-based comparison is useful. Keep references and settings fixed when assessing model changes. Review results by language and content type rather than assuming identical meaning across scripts or domains. Pair the score with checks for omissions, terminology, numbers, and source meaning. If it is used outside translation, validate that character agreement tracks the quality dimension of interest; convenient availability of a metric does not establish that it measures task success.

People also ask

Does chrF require a language model?

No. Its core calculation uses character-sequence counts and a precision-recall formula.

Is chrF++ exactly the same as chrF?

No. It extends character matching with word n-gram information, so configuration and naming matter.

Can two scores be compared without knowing beta?

Not reliably. Beta changes the balance between precision and recall, and other settings can change the result too.

Check your understanding

Why does “cat” receive partial credit against “cats” in the worked example?

Answer: Two character bigrams match even though the full words differ. The missing “ts” lowers recall. Partial credit describes shared text fragments, not proof that singular and plural meanings are interchangeable.

Related terms

• Precision
• Recall
• Machine translation
• BLEU (Bilingual Evaluation Understudy)
• chrF
• chrF++
• character n-gram
• F-beta score
• reference-based evaluation
• BLEU
• text normalization
• surface similarity

Learn next

Write down character bigrams for several short string pairs and calculate precision and recall manually. Include a spelling variant, a repetition, and a meaning-changing omission. Then use a documented multi-order implementation and examine how beta and whitespace settings affect results. Study human translation-error analysis to connect surface comparisons with meaning and usability.

• Precision
• Recall

More in Causal inference and experimentation

• Causal positivity - Causal positivity is the requirement that the treatment choices needed for a causal comparison have positive probability within the relevant covariate...
• Causality - Causality concerns relationships in which changes in one part of a system can produce changes in another.
• Causation - Causation is a cause-and-effect relationship in which one event, condition or intervention contributes to producing another.
• Citation precision - Citation precision measures how well the citations an AI system supplies support the statements to which they are attached, under a specified evaluation rule.
• Citation recall - Citation recall measures how much of an answer's evaluated content is supported by the citations provided, under a defined claim or statement-level protocol.
• Cohen's kappa - Cohen's kappa is a statistic measuring agreement between two raters who assign categorical labels to the same items, adjusted for agreement expected under a...

Sources

Popović, chrF: character n-gram F-score for automatic MT evaluation: https://aclanthology.org/W15-3049/ Popović, original chrF and chrF++ implementation: https://github.com/m-popovic/chrF SacreBLEU, chrF configuration and reproducibility documentation: https://github.com/mjpost/sacrebleu

Previous: Character disjunction Next: Characteristic function