Are all tokens subword fragments?

No. A subword vocabulary commonly includes whole words and other units as well.

Do two models necessarily split a word the same way?

No. Their tokenizers and vocabularies can differ.

Does splitting an unknown word solve every vocabulary problem?

No. It can improve representational coverage, but coverage and semantic knowledge are different matters.

AI Glossary / Natural language and speech / Intermediate

Subwords

Definition

Subwords are text units smaller than a whole word that can be used as tokens in language-processing systems. A subword tokenizer represents text using a vocabulary that may contain whole words as well as smaller pieces. The pieces can resemble meaningful word parts, but they can also be statistical fragments with no independent linguistic meaning.

What it means

A vocabulary containing every possible word would struggle with new names, unusual spellings, compounds, and inflected forms. Subword tokenization allows many words to be represented as combinations of reusable pieces instead of requiring a separate vocabulary entry for each word.
The resulting boundaries depend on the tokenizer and its training or configuration. Two systems can split the same text differently and therefore produce different token counts.
Subwords are not necessarily syllables or morphemes. A syllable concerns speech structure, while a morpheme is a meaning-bearing unit. A learned token boundary may happen to align with either, but that alignment is not guaranteed.

How it works

A tokenizer is trained or configured to define a vocabulary and a segmentation procedure. Methods such as byte-pair encoding learn recurring combinations, while other approaches use different criteria for choosing pieces.
During encoding, the input is converted into a sequence of vocabulary tokens and their identifiers. The language model processes those identifiers rather than treating every written word as a single indivisible unit.
Smaller units improve coverage of unusual words, provided the tokenizer has suitable underlying character or byte coverage and fallback behavior. An arbitrary subword vocabulary does not automatically cover every possible input.
Decoding reconstructs text according to the tokenizer’s rules. Normalization, whitespace handling, and special tokens also matter, so a simplistic picture of cutting at visible word boundaries does not describe every implementation.

Three practical examples

A language model represents an unfamiliar compound using several known pieces instead of requiring that compound as a whole-word entry.
A document-processing team measures actual tokenizer output when estimating whether a passage fits a model’s input limit.
A multilingual evaluation compares how efficiently the selected tokenizer represents different scripts and language varieties.

Think of it this way

Think of building many labels from a reusable collection of pieces. A whole familiar label may already be available, while an unusual one needs several smaller pieces. The analogy is limited because tokenizer pieces are selected by an algorithm and need not correspond to meaningful building blocks in human language.

Worked example

Imagine a deliberately simplified tokenizer whose vocabulary contains “rain,” “bow,” “s,” and “rainbow.” Assume its stated rule chooses the longest matching token from left to right and that no whitespace or special tokens are involved.
The word “rainbows” is encoded as “rainbow” plus “s,” giving two tokens. If a second tokenizer lacks “rainbow” but contains the other three pieces and uses the same rule, it encodes the word as “rain” plus “bow” plus “s,” giving three tokens.
The written word has not changed, but the token count has. Neither segmentation proves how a person mentally represents the word or where its syllable boundaries lie.
This is an invented segmentation rule for illustration, not a claim that all subword tokenizers use longest matching. It demonstrates why token counts and boundaries must be obtained from the actual tokenizer when they matter to an application.

Illustrative case study

In a fictional project, Summit Notes estimates document capacity by counting spaces and assuming one token per word. The estimate works tolerably on some familiar prose but fails on technical identifiers and names that the selected tokenizer divides into many pieces.
The team changes its planning process to measure the real encoded input, including relevant formatting and special-token overhead. It also tests the languages expected in its documents.
The imagined correction addresses an accounting assumption. It does not claim that more pieces always cause worse model performance or that a tokenizer producing fewer tokens is automatically better for every task.

Common misconceptions and mistakes

Subwords are not necessarily meaningful morphemes, and they are not necessarily pronounceable syllables.
One written word can correspond to several tokens, while some token vocabularies also contain larger units involving spaces or punctuation.
Token counts are not universal properties of a sentence; they depend on the representation.
Finally, representability is not understanding. Being able to encode an unfamiliar word as known pieces does not guarantee that a model knows its meaning, spelling behavior, or correct use.

Business use

Subword representations affect input length, model interfaces, text alignment, and the handling of rare terminology. Teams should understand the selected tokenizer when preparing documents or interpreting token-level outputs.
Use the actual tokenizer for capacity estimates and preserve mappings back to the source text when highlighting or extracting passages. A label attached to one token may cover only part of a word.
Evaluate coverage and segmentation across the language and domain mix that matters. Avoid using a convenient English rule of thumb as a universal estimate, and distinguish efficient encoding from demonstrated quality on the downstream task.

Check your understanding

A word is encoded into four pieces by one tokenizer and two by another. How many tokens does the word have?

Answer: There is no tokenizer-independent answer in this context. It has four or two tokens under the respective encodings.

Related terms

Tokenization
Token
Byte-pair encoding
Vocabulary
Morpheme
Syllable
character encoding
token identifier

Learn next

Inspect the same short passage with two documented tokenizers and compare boundaries and counts. Then study how character or byte coverage, normalization, and decoding influence what text a tokenizer can faithfully represent.

Sources

Source: Jurafsky and Martin, Speech and Language Processing, words and subword tokenization: https://web.stanford.edu/~jurafsky/slp3/2.pdf . Independently consulted section 2.4 on learned subword units, arbitrary versus meaningful pieces, vocabulary training and encoding, including the discussion of pretokenization and tokens spanning spaces.

Previous: Subword token Next: Successor-state axiom