AI Glossary / Natural language and speech / Intermediate
Wordpiece
Definition
Definition: WordPiece is a subword tokenization method that represents text using a fixed vocabulary containing whole words and smaller word pieces. In a common BERT-style implementation, it segments each prepared word by repeatedly selecting the longest available piece at the current position.
What it means
What it means: Instead of requiring a separate vocabulary entry for every possible word, a model can reuse pieces across many words. An unfamiliar compound may therefore be represented by several known units. Those pieces are statistical text units, not necessarily syllables or meaningful grammatical components.
WordPiece names both a subword approach and, in practice, tokenizers using a learned piece vocabulary. Learning the vocabulary and applying it are separate stages. The output also depends on case handling, text normalization, preliminary splitting, continuation markers and unknown-token rules. Two systems described as using WordPiece need not produce identical token sequences unless these settings and their vocabularies match.
How it works
How it works: After any required preprocessing, a common implementation starts at the beginning of an input word. It searches for the longest prefix present in its vocabulary, emits that piece and continues with the remaining characters. Continuation pieces may carry a marker such as ## to distinguish them from pieces allowed at a word's beginning.
The procedure is greedy: it chooses the longest match available now rather than exploring every complete segmentation. A BERT-style implementation returns an unknown token if it cannot finish a word under its rules. Other libraries expose configurable unknown handling, so this detail must be checked. The emitted pieces are then mapped to vocabulary IDs for the model.
Three practical examples
Three practical examples:
1. A text classifier represents a new compound using known pieces, allowing it to process the input without a dedicated whole-word vocabulary entry. This representation does not guarantee that the classifier understands the compound's meaning.
2. A document annotation tool maps labels from original words to subword positions. It keeps alignment information so that one word split into three pieces does not accidentally become three separate named entities.
3. A deployment team checks that its tokenizer uses the vocabulary and preprocessing expected by a trained model. Replacing the vocabulary while retaining the same model weights can change which meanings the numeric IDs represent.
Think of it this way
Think of it: WordPiece resembles assembling labels from a box containing both complete labels and reusable fragments. The longest available fragment may be chosen first. The analogy is limited because the vocabulary is a statistical representation, fragments need not carry independent meaning, and the chosen greedy path may fail even when a different assembly would have succeeded.
Worked example
Worked example: Use the invented vocabulary [UNK], sun, flower, ##flower and ##s. Assume lowercase words arrive already separated, and continuation pieces require ##. The word sunflowers is not a whole vocabulary entry. Its longest starting match is sun; the remaining flowers matches ##flower followed by ##s. The result is three pieces: sun, ##flower, ##s.
Now consider a second toy vocabulary: [UNK], ab, a and ##bc. For the input abc, the greedy first choice is ab. The remaining c has no ##c entry, so the specified BERT-style procedure returns [UNK]. It does not backtrack to the otherwise possible sequence a, ##bc.
This example separates longest-match tokenization from a search for any valid segmentation. It also shows why knowing only that every input character appears somewhere in a vocabulary is insufficient: piece position and continuation forms matter. These are deliberately small teaching vocabularies, not claims about the output of a particular pretrained model.
Illustrative case study
Illustrative case study: Fictional Birch Search adapts a document classifier to equipment manuals. Engineers notice that unfamiliar component names are split into more pieces than common words. They inspect the resulting sequence lengths and the alignment between extracted labels and source text.
Rather than replacing token IDs casually, they preserve the trained model's tokenizer and evaluate whether a different model or deliberate retraining is warranted. They also test case-sensitive codes and unusual characters. This invented case illustrates compatibility and coverage checks. It does not claim that shorter tokenization always improves task quality or that more subwords necessarily cause an error.
Common misconceptions and mistakes
Common misconceptions and mistakes: WordPiece is not synonymous with every subword method. Byte-pair encoding and unigram tokenization use related ideas but have different procedures. SentencePiece is a toolkit supporting tokenization models, not another name for WordPiece. A ## marker is a tokenizer convention rather than a literal part of the original word. Subword coverage is not automatically complete, and a piece boundary does not necessarily coincide with a morpheme or syllable. Token counts therefore cannot be inferred reliably from ordinary word counts.
Business use
Business use: WordPiece affects input length, model compatibility and the handling of specialist terminology. Keep the exact vocabulary and preprocessing configuration with each model release. Test representative names, codes, languages and long words, and measure truncation as well as task accuracy. Preserve source offsets when results must be displayed against the original document. Before comparing token counts across systems, confirm that their tokenizers and normalization rules are equivalent; otherwise the counts describe different units.
People also ask
People also ask: Does WordPiece always split a word?
No. A whole word can be emitted as one piece when the applicable vocabulary contains it.
Does ## mean a word is a grammatical suffix?
No. In this convention it marks a continuation piece, which need not be a meaningful linguistic suffix.
Can WordPiece still produce unknown tokens?
Yes. Coverage, length limits and implementation-specific fallback rules can lead to an unknown-token result.
Check your understanding
Check your understanding: In the second toy vocabulary, a and ##bc could cover abc. Must a greedy BERT-style tokenizer find that sequence?
Answer: No. It first chooses the longer starting piece ab and fails on c. The stated procedure does not backtrack to the alternative.
Related terms
- Vocabulary
- Byte-pair encoding
- Preprocessing
- subword tokenisation
- token ID
- unigram language model
- unknown token
- offset alignment
Learn next
construct a tiny vocabulary with beginning and continuation pieces, then segment several words by hand. Include a whole-word match, a successful multi-piece split and a greedy failure. Compare your results with the documented implementation before making assumptions about a model's input lengths.
More in Natural language and speech
- Word sense disambiguation - Definition: Word sense disambiguation, or WSD, is the task of deciding which meaning of a word is intended in a particular context.
- Word-sense induction - Word-sense induction is the automatic discovery of different uses or senses of a word from the contexts in which it occurs.
- Word shape - Definition: Word shape is a representation of a token's character pattern, such as uppercase letters, lowercase letters, digits and punctuation.
- Yarowsky algorithm - The Yarowsky algorithm is an iterative bootstrapping method for word-sense disambiguation.
- Zero anaphor - Definition: A zero anaphor is an unspoken or unwritten referring expression whose interpretation is supplied by linguistic context.
- Zero-width - Definition: In regular expressions, zero-width describes a match or assertion that consumes no characters.
Sources
Sources: TensorFlow Text, WordpieceTokenizer API and configurable continuation/unknown handling: https://www.tensorflow.org/text/api_docs/python/text/WordpieceTokenizer . Google Research BERT, original tokenisation implementation: https://github.com/google-research/bert/blob/master/tokenization.py . The toy vocabularies and Birch Search case are original illustrations; no output from an actual pretrained vocabulary is asserted.
Previous: word2vec-CBOW Next: Work stealing