AI Glossary / Causal inference and experimentation / Intermediate

ROUGE-S

Definition

ROUGE-S is a ROUGE evaluation measure based on matching skip-bigrams between a candidate text and a reference. A skip-bigram is an ordered pair of words that may have intervening words between them. The measure compares these pairs and can express the overlap as recall, precision, and an F-measure. It preserves word order while allowing gaps, providing a different lexical comparison from adjacent bigrams or a longest common subsequence.

What it means

Two summaries may express shared content with extra words inserted between important terms. Exact adjacent-pair matching can miss that relationship. Skip-bigrams allow a pair such as “robot packed” to match even if one sentence says “robot carefully packed.”

Order still matters. The pair “packed robot” is different from “robot packed.” ROUGE-S therefore does more than count shared individual words, but it does not fully interpret grammatical roles or meaning.

Unrestricted skip-bigrams can connect words far apart in a sentence. Some variants limit the permitted gap to reduce incidental matches. Report the gap convention and statistic because different settings can produce different scores from the same candidate and reference.

How it works

Prepare the texts using a consistent tokenization and normalization protocol. Generate ordered word pairs permitted by the skip rule, then count matching pair occurrences according to the selected implementation's handling of repetitions.

Without a gap limit, a sequence of m token positions contains m(m − 1)/2 ordered-in-sequence position pairs. For a single-reference comparison, divide matching pairs by the reference pair count for recall and by the candidate pair count for precision. An F-measure combines them with a declared weighting; F1 gives equal weighting.

If a gap restriction is applied, count only allowed pairs in both numerators and denominators. Texts too short to contain a pair require an explicit zero-count convention. For multiple references or multiple sentences, state aggregation and boundary handling. These details are part of the evaluation, not incidental formatting choices.

Three practical examples

1. Summary comparison: researchers use skip-bigram overlap to examine shared ordered wording even when candidates insert modifiers between matching terms.

2. Evaluation diagnosis: a team compares adjacent-bigram and skip-bigram scores to investigate whether low local overlap comes from inserted words rather than complete reordering.

3. Short-text experiments: analysts study alternative paraphrases with a clearly stated gap rule, combining the resulting overlap measure with direct judgments of meaning.

Think of it this way

Think of comparing the order of stops on two routes while allowing extra stops between shared ones. A pair of landmarks counts when the first appears before the second on both routes. The analogy is limited because many matching landmark pairs can occur without identical routes, just as matching word pairs do not guarantee that two summaries express the same facts or emphasis.

Worked example

Use lowercase whitespace tokens, a single sentence, no repeated words, and no gap limit. The reference is “robot carefully packed boxes,” with four tokens. Its six skip-bigrams are “robot carefully,” “robot packed,” “robot boxes,” “carefully packed,” “carefully boxes,” and “packed boxes.”

The candidate is “robot packed boxes,” with three tokens and three skip-bigrams: “robot packed,” “robot boxes,” and “packed boxes.” All three match. Skip-bigram recall is 3/6 = 0.50 and precision is 3/3 = 1. Equal-weight F1 is 2 × 0.50 × 1/(0.50 + 1), approximately 0.667.

Now reverse the candidate to “boxes packed robot.” Its pairs are “boxes packed,” “boxes robot,” and “packed robot,” none of which appear in the reference. Its skip-bigram overlap is zero despite sharing three individual words. This illustrates both order sensitivity and a limitation: ROUGE-S alone provides no unigram credit in this example. The arithmetic is a hand calculation under the declared protocol, not an executed software score.

Illustrative case study

Fictional case: Acacia Summaries compares short generated status updates with editor references. A candidate adds a descriptive word between two reference terms, reducing adjacent-bigram overlap. Skip-bigram matching retains some credit for the preserved ordering.

The editors also find candidates with many shared ordered pairs but an incorrect date. They keep direct factual review and document the gap setting used in the automatic comparison. The case illustrates the additional flexibility and remaining limitations of skip-bigrams. It does not describe a real product or prove that ROUGE-S is always preferable to other ROUGE variants.

Common misconceptions and mistakes

ROUGE-S is not ROUGE-SU. ROUGE-SU extends the comparison with unigram information, which can give credit when individual words overlap but no ordered pair matches. Report the actual variant instead of treating the names as interchangeable.

Do not confuse a skip-bigram with any unordered pair. The words must preserve sequence order. Also avoid counting unrestricted pairs in the denominator when the numerator uses a restricted gap; that would mix different protocols.

Large gaps can create incidental matches between common words, while restrictive gaps can miss useful relationships. Repeated words and sentence boundaries add further counting choices. Finally, strong pair overlap does not verify factual accuracy, grammaticality, or completeness. A changed number or negation can matter more than many unchanged word pairs.

Business use

ROUGE-S can supplement reference-based text evaluation when shared order with intervening words is relevant. It provides a repeatable comparison that is more flexible than adjacent matching while retaining some ordering information.

For business reporting, name the gap rule, tokenizer, reference set, and recall or F-score convention. Inspect examples where automatic scores disagree with useful output quality. Select measures because their behavior fits the evaluation purpose, and retain direct checks for requirements such as factual consistency and required content coverage.

People also ask

Must the two words be adjacent?

No. ROUGE-S permits gaps, subject to any stated maximum-gap restriction.

Does an unrestricted four-token sentence have six pairs?

Yes, there are four choose two position pairs. Repeated token values may produce repeated pair types, which requires consistent occurrence handling.

Why add unigrams in ROUGE-SU?

They allow shared individual words to receive credit even when no permitted skip-bigram matches, addressing a limitation of pair-only comparison.

Check your understanding

A five-token reference and four-token candidate have no repeated words and use unrestricted pairs. Four pairs match. What are skip-bigram recall and precision?

Answer: The reference has 5 × 4/2 = 10 pairs and the candidate has 4 × 3/2 = 6. Recall is 4/10 = 0.40 and precision is 4/6, approximately 0.667.

Related terms

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
• ROUGE-N
• ROUGE-L
• Precision
• Recall
• F-measure
• ROUGE
• ROUGE-SU
• skip-bigram
• ordered overlap

Learn next

List all skip-bigrams for two short sentences with the same words in different orders. Count their shared pairs, then impose a small declared gap limit and repeat. Explain why both the match count and possible-pair counts must be reconsidered.

More in Causal inference and experimentation

• Recall at k (recall@k) - Recall at k, written recall@k or R@k, is the fraction of a defined set of relevant items that appears among the first k ranked results.
• Remediation - Remediation is corrective action taken to address an identified weakness, failure or harmful outcome.
• ROC (receiver operating characteristic) Curve - A receiver operating characteristic curve, or ROC curve, plots a classifier's true-positive rate against its false-positive rate as the decision threshold...
• Sequential exchangeability - Sequential exchangeability is a causal assumption that, at each treatment decision, treatment is independent of the relevant future potential outcomes...
• Side-by-side evaluation - Side-by-side evaluation compares two systems by judging their outputs for the same input.
• Silhouette Coefficient - The Silhouette Coefficient measures how well an observation fits its assigned cluster compared with the nearest competing cluster.

Sources

Lin, ROUGE: A Package for Automatic Evaluation of Summaries, ROUGE-S and ROUGE-SU sections: https://aclanthology.org/W04-1013.pdf

The original paper grounds skip-bigram matching, gap restrictions, and the unigram extension. The worked example explicitly uses unrestricted single-sentence pairs and F1. Examples, case, and exercise are original teaching material.

Previous: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) Next: Rough fuzzy hybridization