AI Glossary / Deep learning / Intermediate

Vocabulary Reduction

Definition

Vocabulary reduction is a set of techniques for decreasing the number of distinct text units a language system represents or considers during a computation. It may mean replacing a large word vocabulary with reusable subword units, pruning rarely used entries, or restricting output candidates for a particular input. These approaches have different consequences for coverage, model size, sequence length, and prediction quality, so the specific meaning should always be stated.

What it means

A vocabulary is the inventory of token types available to a model, not the total number of tokens in a document. A smaller inventory can reduce some embedding or output-layer costs, but it can also require more tokens to express the same text. Restricting an output shortlist is different again: the model may retain its full vocabulary while scoring only selected candidates at one stage. Neither operation necessarily makes the language itself simpler, and neither guarantees that the system preserves all important distinctions.

How it works

A subword approach learns or defines reusable pieces so uncommon words can be represented by sequences rather than dedicated entries. A pruning approach removes entries according to a criterion and needs a fallback for affected text. A dynamic selection approach chooses a candidate subset for an input or decoding step. Each requires consistency between token identifiers, embeddings, output weights, and reconstruction rules. An existing pretrained model cannot generally accept arbitrary tokenizer changes without corresponding adaptation and evaluation, because its learned parameters are tied to the original representation.

Three practical examples

• Machine translation: use subword units to represent rare word forms within a manageable token inventory.
• Specialized inference: evaluate a context-dependent output shortlist when reducing candidate scoring cost is important.
• Compact text models: compare vocabulary sizes while measuring memory use, sequence length, rare-term handling, and task quality together.

Think of it this way

Think of a kit that stores reusable word-building pieces instead of a separate block for every complete word. Fewer piece types can express many combinations, but assembly may take more steps. The analogy captures the inventory-versus-length tradeoff; it does not imply that all words decompose into meaningful pieces or that the model understands every new combination.

Worked example

Suppose a model uses an embedding dimension of 256. A vocabulary of 50,000 entries requires 50,000 times 256 = 12,800,000 embedding values. A vocabulary of 20,000 requires 5,120,000 values, a reduction of 7,680,000. At two bytes per value, that is 15,360,000 fewer bytes for this table alone, before accounting for metadata, other parameters, or training state. It is not a complete model-memory estimate.
Now suppose the same collection of sentences averages 100 tokens with the first tokenizer and 140 with the second. A computation proportional to sequence length increases by a factor of 1.4. A fully dense pairwise attention calculation proportional to length squared would increase by approximately 1.96 for that sequence-length change, ignoring other effects. Thus a smaller vocabulary can reduce one cost while increasing another. Separately, scoring a shortlist of 2,000 candidates rather than 50,000 reduces the number of candidate scores by a factor of 25, but only if the implementation actually avoids the full computation. It cannot select a needed output token that was excluded from the shortlist, unless a fallback or alternative representation provides it.

Illustrative case study

A fictional translation team reduces its token inventory to fit a device budget. The embedding table becomes smaller, but product names and inflected words split into longer sequences. The team measures complete sentence latency and checks rare terminology across the languages it supports. It also evaluates a shortlist-based alternative separately. The chosen configuration reflects measured end-to-end behavior rather than an assumption that fewer vocabulary entries always produce faster or better translation.

Common misconceptions and mistakes

Reducing token types is not the same as reducing document length. Subword segmentation can increase token count while improving coverage of unseen word forms. A smaller vocabulary does not automatically eliminate unknown characters or scripts; that depends on the tokenizer's fallback design. Frequency pruning can disproportionately affect rare names and specialist terminology. Candidate selection can remove the correct answer before scoring, and aggregate quality metrics can miss particular linguistic failures such as idioms or unusual morphology.

Business use

Vocabulary choices affect memory, training cost, generation latency, and language coverage. Evaluate them on representative text, including uncommon terms, numbers, names, and the scripts and languages expected in use. Record whether the change affects the global tokenizer or only candidate selection. Compare total memory and complete-request latency, not just table size or one kernel. Preserve mapping compatibility when saving and loading models, and treat vocabulary changes as model changes requiring regression checks.

People also ask

Does a smaller vocabulary always make text shorter?

No. More words may need to be expressed as several smaller tokens.

Is vocabulary reduction the same as deleting rare words from documents?

No. Some approaches retain the text through subword representation, while others change the allowed candidate set or require a fallback.

Can a dynamic shortlist hurt quality?

Yes. If necessary candidates are excluded, the remaining scoring process cannot recover them without another mechanism.

Check your understanding

Why can the 20,000-entry vocabulary save embedding memory but increase attention work in the example?

Answer: The table has fewer rows, while the same text becomes 140 tokens rather than 100. Those are different cost drivers. Under a dense pairwise attention calculation, the length change gives roughly 1.96 times as many token pairs.

Related terms

• Tokenization
• Byte-pair encoding
• Decoding
• vocabulary size
• subword unit
• morphological segmentation
• out-of-vocabulary token
• lexical shortlist
• embedding table
• sequence length

Learn next

Compare word, subword, and character representations on a few domain-specific sentences. Count both distinct token types and tokens per sentence. Then examine a candidate-selection method and trace what happens when the desired output is absent. Evaluate representation changes and inference shortlists as separate design choices.

More in Deep learning

• Unsupervised Biclustering - Unsupervised biclustering discovers related subsets of rows and columns in a data matrix without using target class labels to define those groups.
• Vector Autoregression - Vector autoregression, or VAR, is a statistical time-series model that predicts several variables together using their own past values and the past values...
• Vehicle interface layer - A vehicle interface layer is the software boundary that translates between a vehicle's hardware-specific messages and the more general commands and status...
• Winner-take-all (computing) - Winner-take-all, or WTA, is a computational selection mechanism in which one candidate remains active while competing candidates are suppressed or excluded.
• word2vec-CBOW - word2vec-CBOW is the continuous bag-of-words architecture for learning word vectors by predicting a target word from nearby context words.
• You Only Look Once - You Only Look Once, usually abbreviated YOLO, is the name of an influential object-detection approach and a broad family of later detectors.

Sources

Ataman and Federico, An Evaluation of Two Vocabulary Reduction Methods for Neural Machine Translation: https://aclanthology.org/W18-1810/ Sennrich, Haddow, and Birch, Neural Machine Translation of Rare Words with Subword Units: https://aclanthology.org/P16-1162/ Domhan and colleagues, The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation: https://aclanthology.org/2022.naacl-main.136/

Previous: Vocabulary mismatch Next: Voice computing