AI Glossary / Natural language and speech / Advanced
Span-corruption language-model objective
Definition
A span-corruption language-model objective trains a model to reconstruct consecutive pieces of text removed from an input. In the T5 formulation, each removed span is replaced by a sentinel, and the target contains the missing spans separated by corresponding markers.
What it means
Rather than predicting only the next word from its left-hand context, the model receives a passage with several gaps. Visible text on either side can help determine what belongs in each gap.
Removing a span can require recovering a phrase or several related tokens, not merely making isolated word guesses. A sequence-to-sequence decoder predicts the missing content in order and can condition later target tokens on earlier ones.
This entry focuses on the T5-style objective in the worklist's source. Other denoising models may replace spans with a shared mask token and reconstruct the entire original text. Those choices affect both the training format and the amount of output the model must produce.
How it works
Tokenize an original text sequence, choose a corruption rate and select contiguous spans to remove. A span-length policy determines how that missing-token budget is divided among gaps.
Replace each span with a distinct sentinel in the input. Build the target from each sentinel followed by its removed tokens, preserving the order of spans, and include the final marker required by the format.
Train an encoder-decoder model to predict that target autoregressively, using the corrupted input as context. The loss rewards correct missing tokens and structural markers; unchanged input text need not be reproduced in the target.
Corruption rate and average span length influence difficulty and sequence lengths. Longer spans at a fixed removed-token count mean fewer markers. They can also demand more reconstruction from less locally informative boundaries. The appropriate values are experimental choices rather than universal constants.
Three practical examples
- A pretraining pipeline removes short phrases from unlabeled documents and creates reconstruction targets automatically.
- A researcher changes average span length while holding the overall corruption rate fixed to study the effect of gap structure.
- A dataset audit checks that hidden target text has not accidentally remained visible elsewhere through a duplicated preprocessing field.
Think of it this way
Think of restoring missing strips from a page while the remaining paragraphs provide clues. Several strips can be missing, and labels show where each reconstructed piece belongs.
The analogy highlights contextual reconstruction. A model is trained against the actual removed text, even when another completion would also sound plausible. Recovering likely wording does not establish that the original passage, or the model's completion, is factually correct.
Worked example
Consider 40 toy tokens labeled x1 through x40. Remove six tokens in two spans: x5, x6, x7 and x25, x26, x27. The corruption rate is 6/40 = 15%, and the average removed-span length is 6/2 = 3 tokens.
The input replaces the first span with <S1> and the second with <S2>. It contains 40 − 6 + 2 = 36 tokens, ignoring any separate sequence-ending token.
The target is <S1> x5 x6 x7 <S2> x25 x26 x27 <S3>. It has nine tokens: six missing tokens and three sentinels, including the final marker.
If the same six missing tokens were instead arranged as six isolated spans, the input would contain 40 − 6 + 6 = 40 tokens. The target would contain 6 + 6 + 1 = 13. This accounting shows how grouping affects sequence lengths. It does not imply a proportional change in runtime, because model computation depends on more than token counts.
Illustrative case study
The fictional WillowCorpus team prepares a denoising experiment on product manuals. Its initial corruption code selects words before subword tokenization, while its configuration describes a percentage of tokens.
A sample inspection reveals that the actual token-level corruption rate varies unexpectedly across examples. The team makes the unit explicit and verifies input and target lengths using small labeled sequences before running the larger experiment.
It evaluates downstream tasks separately from reconstruction accuracy, including examples with specialized terminology. This invented case demonstrates the importance of matching implementation units to the objective. It reports no measured model improvement or training-cost saving.
Common misconceptions and mistakes
- Span corruption is always independent masked-token prediction. A sequence-to-sequence target can model dependencies among removed tokens.
- Fifteen percent corruption means fifteen percent of words. The rate is defined over the chosen tokenization, which may split words.
- Every denoising objective reconstructs the entire source. T5-style span corruption predicts missing spans and markers rather than repeating all visible text.
Business use
Span corruption provides a self-supervised way to learn from large text collections without a separate human-written output for every input. It is useful for understanding how pretraining examples are generated and why objective details affect computation.
Document tokenization, corruption rate, span-length sampling, sentinel handling and truncation. Inspect whether the data contain shortcuts, such as duplicate passages exposing removed content. Compare downstream quality and full training cost under controlled conditions. A low reconstruction loss is evidence about the training task; it does not directly certify instruction following, factual reliability or suitability for a particular business workflow.
People also ask
Can the model use text after a missing span?
Yes. The encoder sees the uncorrupted surrounding input, including later text.
Does a span always contain the same number of tokens?
No. A policy can sample variable lengths while controlling an average or total removed-token count.
Is this identical to BART text infilling?
They share the idea of corrupting spans, but T5's distinct sentinels and missing-span-only target differ from BART's full-document reconstruction setup.
Check your understanding
A 30-token input loses eight tokens in two spans. How many tokens remain after replacing each span with one sentinel, excluding extra special tokens?
Answer: 30 − 8 + 2 = 24 tokens. Replacing a span does not restore the full number of removed tokens.
Related terms
- Masked language modeling
- Self-supervised learning
- Tokenization
- T5
- sentinel token
- denoising objective
- text infilling
- encoder-decoder model
Learn next
Read sentinel tokens and prefix language modeling. Construct both objectives from the same short passage, then compare which text is visible, which text is predicted and how target length changes.
More in Natural language and speech
- Slot filling - Slot filling is the process of identifying values for named fields in a structured representation of a user’s request or another piece of text.
- SOV language - An SOV language is a language whose basic or dominant clause order places the subject before the object and the object before the verb.
- Span - In natural language processing, a span is a selected stretch of text identified by its boundaries.
- Speaker diarization - Speaker diarization is the process of determining which speaker is active at which times in an audio recording.
- Speaker diarization - Speaker diarization is the process of dividing an audio recording into time intervals and assigning speaker labels to those intervals.
- Speaker identification - Speaker identification is the task of determining which person in a set of candidate speakers best matches a speech sample.
Sources
Colin Raffel and colleagues, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020, Sections 3.1.4 and 3.3.4: https://www.jmlr.org/papers/volume21/20-074/20-074.pdf
The missing-span target format and span-length/corruption-rate discussion were reviewed in the original paper. The 40-token accounting and fictional preprocessing case are original. Example settings are not claimed to be optimal for every task.
Previous: Span Next: Spanning Trees