Is Heaps's law exact?

No. It is an empirical approximation whose fit varies with the data and range.

Why does vocabulary often grow more slowly than text length?

Many additional occurrences repeat terms already observed, while only some introduce new types.

Does a lower type - token ratio prove a text is less diverse?

Not by itself. Length and counting decisions affect that ratio, so comparisons require appropriate controls rather than treating it as an independent quality score.

AI Glossary / Natural language and speech / Intermediate

Heaps's law

Definition

Heaps's law is an empirical relationship describing how the number of distinct terms in a text collection tends to grow as more word occurrences are observed. It is commonly expressed as V = K Nᵝ, where N is the token count and V is the number of different types. Its parameters depend on the collection and processing choices.

What it means

A token is an occurrence, while a type is a distinct item under a specified counting rule. In “blue blue green,” there are three tokens and two types if words are counted as written.
As a collection grows, many new tokens repeat previously seen types, while some introduce new ones. Heaps's law models the resulting vocabulary-growth pattern with a power function.
For the commonly considered range 0 < β < 1, vocabulary grows more slowly than token count. The name law describes a regularity observed in data, not an exact rule that every document must obey. The relationship is also discussed under the name Herdan's law.

How it works

Define tokenization and normalization before counting. Decisions about case, spelling, numbers, punctuation, and word forms affect what counts as a distinct type.
Measure vocabulary size at several collection sizes. Fit K and β over an appropriate range and check how well the estimates match observed growth.
Taking logarithms gives log V = log K + β log N. A roughly straight pattern on a log - log plot supports the power-law approximation, with β describing its slope.
Use the fitted relationship only with its limitations in mind. Changing domains, languages, preprocessing, or sampling conditions can change the growth pattern. A model that works over one range need not remain accurate for very small samples or an entirely different collection.

Three practical examples

A search-engine team estimates how its term dictionary might grow as more similar documents are indexed.
A corpus analyst compares vocabulary-growth curves under case-sensitive counting and case folding.
A language-data project checks whether a new source introduces substantially more distinct terms than its existing collection model predicted.

Think of it this way

Think of collecting postcards: the pile grows with every card, but the number of different designs grows only when a previously unseen design arrives. This captures the distinction between occurrences and distinct types. The analogy stops short of a universal prediction because the supply of designs and how cards are selected determine the actual growth pattern.

Worked example

Suppose a teaching model uses V = 20√N, equivalent to K = 20 and β = 0.5. At N = 10,000 tokens, it predicts V = 20 × 100 = 2,000 types.
At N = 40,000 tokens, it predicts V = 20 × 200 = 4,000 types. The token count has quadrupled while the predicted vocabulary has doubled.
At N = 20,000, the prediction is approximately 20 × 141.42 = 2,828 types. Doubling the original token count therefore increases vocabulary by a factor of √2, about 1.414, rather than by a factor of two.
The type - token ratio falls from 2,000/10,000 = 0.20 to 4,000/40,000 = 0.10 in the first comparison. That decline can arise from collection size alone, so it should not automatically be interpreted as poorer language use.
These are invented parameters for a moderate-size range. At N = 1, the formula would predict 20 types, which is impossible from one token. This exposes why a fitted empirical approximation must not be extrapolated indiscriminately.

Illustrative case study

In a fictional project, Rowan Search estimates the term-dictionary growth of an internal document index. The team fits a curve to one collection using consistent tokenization, then measures a newly added source.
The new source contains many product identifiers and spelling variants, producing more distinct terms than the old curve predicts. Analysts inspect the processing choices and source mix before deciding whether to refit the model.
The imagined case illustrates how a useful estimate can reveal a change in the data. It does not claim that one fitted curve determines the exact storage requirement or that unusual vocabulary growth necessarily indicates a data-quality problem.

Common misconceptions and mistakes

Vocabulary size is not the same as token count. Repeated words increase one without necessarily increasing the other.
β is not universally 0.5, and K is not a constant shared by every language and corpus.
Heaps's law does not prescribe which particular new words will appear.
The model concerns vocabulary growth, whereas Zipf's law concerns the relationship between frequency and rank; related discussions should not make them identical statements.
A fixed subword tokenizer can have a predetermined vocabulary inventory. The number of its types observed in data cannot exceed that inventory, so unlimited power-law growth is not a valid assumption for that setting.

Business use

For a business text platform, the relationship can support approximate capacity planning for the vocabulary component of an index and help compare processing policies. It does not alone estimate postings, document storage, model memory, or total infrastructure costs.
Fit the curve to representative data and inspect deviations as the collection changes. Record whether types mean surface words, normalized terms, or observed subword tokens. Without that definition, vocabulary-growth comparisons can mix fundamentally different quantities.

Check your understanding

Under V = 20√N, what happens to predicted vocabulary if token count increases by a factor of nine?

Answer: Vocabulary increases by a factor of three because √9 = 3, assuming the same fitted relationship remains appropriate over that range.

Related terms

Token
Vocabulary
Zipf's law
Corpus
Tokenization
type
type–token ratio
power law
index dictionary
lexical diversity

Learn next

count types and tokens for successive portions of a small corpus using a documented normalization rule. Plot both ordinary and log - log growth, then test whether parameters fitted to the first part predict later observations reasonably well.

Corpus

Sources

Manning, Raghavan, and Schütze, “Heaps' law: Estimating the number of terms,” Introduction to Information Retrieval: https://nlp.stanford.edu/IR-book/html/htmledition/heaps-law-estimating-the-number-of-terms-1.html. Jurafsky and Martin, Speech and Language Processing, “Words and Tokens”: https://web.stanford.edu/~jurafsky/slp3/2.pdf. The calculations above use original illustrative parameters. Independent source review: The Stanford information-retrieval text was read for the power-law formula, log-log interpretation, and dependence on collection processing choices.

Previous: Healthy user bias Next: Hearst patterns