AI Glossary / Natural language and speech / Intermediate
Sentinel token for span corruption
Definition
A sentinel token for span corruption is a special marker that replaces a removed span of text and links that gap to its reconstruction target. In T5's denoising format, different missing spans within an example receive different sentinel tokens.
What it means
When several parts of a passage are removed, a model needs to know where the gaps are and how the predicted pieces correspond to them. A sentinel provides that structural label without revealing the missing words.
A whole consecutive span can be replaced by one marker, even when several tokens were removed. In the target, the same marker introduces the text belonging to that gap. Additional sentinels separate later reconstructed spans, and the described format ends with another sentinel.
These are reserved vocabulary items, not ordinary words chosen for their meaning. Their role is positional and structural. A sentinel is unique to a span within the example; this does not mean the vocabulary needs a newly invented token for every span in the entire training corpus.
How it works
Choose the token positions to corrupt and group adjacent selected positions into spans. Assign distinct sentinels in order, then replace each span with its assigned marker in the input.
Construct the target by concatenating each sentinel and its missing span, preserving span order. Add the final sentinel specified by the format. The decoder learns to emit these markers and the removed text as a target sequence.
For example, the display label <S1> can stand for a dedicated token ID. Its visible spelling is only an illustration; the tokenizer must recognize the actual reserved token rather than accidentally splitting it into ordinary punctuation and letters.
The markers identify which gap is being reconstructed, but they do not encode the number of missing tokens by themselves. The model determines the content and length of a span until it emits the next marker.
Three practical examples
- A T5-style training example uses one sentinel for a missing two-token phrase and another for a later missing word.
- A preprocessing check verifies that every input gap has the corresponding marker in the target sequence.
- A tokenizer review confirms that reserved sentinel labels map to dedicated token IDs rather than ordinary text fragments.
Think of it this way
Think of numbered blanks in a form and an answer sheet using the same numbers. The labels tell you which answer belongs in which blank without telling you the answer itself.
The analogy explains correspondence. Sentinels are learned model tokens, and a reconstructed span can contain different numbers of tokens. The labels do not prove that the filled-in text is the original or factually correct.
Worked example
For a simplified word-token example, start with: Our team will review the draft tomorrow. Treat the seven words as seven tokens and ignore punctuation for counting.
Remove will review as the first span and tomorrow as the second. The corrupted input is Our team <S1> the draft <S2>. The target is <S1> will review <S2> tomorrow <S3>.
Three original tokens were removed, but only two sentinels replace them in the input. Its length is 7 − 3 + 2 = 6 tokens. The target also has six tokens: three removed words, two matching span markers and one final marker.
Reconstruction inserts will review at <S1> and tomorrow at <S2>. <S3> closes the target format; it is not a third gap in the input. Actual subword tokenization can produce different counts, so these calculations apply only to the stated toy tokenization.
Illustrative case study
The fictional SpruceTraining team builds a span-corruption dataset. Its first version replaces every gap with the same marker but constructs targets using numbered markers.
A review of a two-gap example reveals the inconsistency. The team makes the input and target conventions agree and checks examples containing adjacent removed tokens, several separated spans and the final target marker.
It also verifies reserved-token handling after changing tokenizers. This invented case illustrates format integrity rather than a model-quality result. A correctly formatted training pair is necessary for the intended task, but it does not establish that the trained model will reconstruct every span accurately.
Common misconceptions and mistakes
- Every removed token needs a separate sentinel. A consecutive removed span can use one marker.
- A sentinel tells the model exactly how many tokens are missing. Its basic role is to identify a gap, not encode its length.
- Unique means globally unique across the dataset. The cited format needs distinct markers within an example and reuses its reserved vocabulary across examples.
Business use
Sentinel tokens matter when reviewing denoising datasets, model input formats and tokenizer compatibility. They make the mapping between several missing spans and a compact reconstruction target explicit.
Validate both sides of the training pair, including marker order and the final marker. Ensure there are enough reserved sentinels for the maximum number of spans allowed by preprocessing. Distinguish padding, end-of-sequence tokens and span sentinels, because they serve different roles. When presenting examples to nontechnical readers, explain that visible sentinel strings stand for special tokens rather than instructions embedded in the original document.
People also ask
Are sentinel tokens ordinary words?
No. In the cited T5 setup they are special vocabulary entries reserved for the corruption format.
Can a single sentinel replace multiple missing words?
Yes. It represents a whole consecutive span, whose token length can vary.
Does the target repeat all the uncorrupted input text?
Not in this T5-style objective. The target contains the removed spans and the markers that delimit them.
Check your understanding
Two adjacent tokens are removed as one span, and another token is removed later. How many gap-replacement sentinels are needed in the input?
Answer: Two: one for the adjacent-token span and one for the later span. The target format may additionally use a final sentinel.
Related terms
- Tokenization
- Masked language modeling
- Span corruption
- T5
- special token
- denoising objective
- sequence-to-sequence learning
- reconstruction target
Learn next
Read the span-corruption language-model objective. Build an example with three missing spans, then check that each sentinel appears in the right order on both sides and that the unchanged text is not mistakenly copied into the target.
More in Natural language and speech
- Sentence-permutation denoising - Sentence-permutation denoising is a pretraining task that shuffles a document's sentences and trains a model to reconstruct their original order.
- Sentence separation - In the supplied sequence-generation context, sentence separation means marking the boundary between a source sequence and its target sequence, commonly with...
- Sentiment - In natural language processing, sentiment is an expressed or implied evaluation, commonly described as positive or negative toward a target.
- Sequence labeling - Sequence labeling assigns a label to each unit in an ordered input sequence, producing an aligned label sequence of the same length at the chosen unit level.
- SGNS - SGNS stands for skip-gram with negative sampling, a method for learning word embeddings by distinguishing observed word - context pairs from sampled noise...
- Sibilant sound - A sibilant sound contains prominent hissing or hushing frication produced by a directed stream of air in the front of the mouth.
Sources
Colin Raffel and colleagues, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020, Section 3.1.4 and Figure 2: https://www.jmlr.org/papers/volume21/20-074/20-074.pdf
The original paper's span markers, special-token IDs and target-delimiting convention were reviewed. The seven-word example and length accounting are original and explicitly use simplified word tokenization. Sentinel formatting in other objectives can differ.
Previous: Sentiment analysis Next: Separable filter