Can perplexity be below one?

For ordinary discrete-token probabilities, it is at least one. A value of one would mean every evaluated token received probability one.

What happens if an observed token receives probability zero?

Its negative log probability is infinite, making the corresponding perplexity infinite under the mathematical definition.

Is lower perplexity always better?

It is better predictive fit on the same compatible evaluation. It does not settle every comparison of models or applications.

AI Glossary / Causal inference and experimentation / Intermediate

Perplexity

Definition

Perplexity measures how well a probabilistic model predicts an observed sequence, commonly a sequence of words or tokens. For a language model, it is the exponential of the average negative log probability assigned to the evaluated tokens. Lower perplexity means the model assigned greater probability, on average in this logarithmic sense, to those observed tokens. It measures predictive fit to the evaluation text, not general intelligence or guaranteed answer quality.

What it means

A language model assigns probabilities to possible next tokens given a context. Evaluation supplies the actual next token and checks the probability the model gave it. Repeating this across held-out text produces a measure of how surprising that text was to the model.

Perplexity converts average log loss into a positive number with an intuitive relationship to uncertainty. In a simple uniform prediction problem with four equally likely alternatives at every step, perplexity is four. Real language probabilities are uneven, so perplexity need not be a whole number or equal the vocabulary size.

The score depends on both the model and the text. A model can predict familiar domain language better than unfamiliar writing without becoming generally more capable.

How it works

Tokenize the evaluation text according to a defined protocol. At each evaluated position, use the permitted preceding context to obtain the probability assigned to the token that actually occurs. Compute the negative natural logarithm of each probability and average those values over the evaluated tokens.

Exponentiating that average gives perplexity. Equivalently, it is the geometric mean of the inverse token probabilities. Calculating with logs avoids multiplying a long series of tiny probabilities directly.

Specify which tokens count, how document boundaries are treated, and how much context is available. Keep training information separate from the evaluation text. Comparisons require compatible tokenization, vocabulary, data, and evaluation settings; a lower raw token-level score from a different tokenizer is not automatically evidence of a better language model.

Three practical examples

1. Language-model development: researchers compare candidate models on the same held-out corpus and tokenization to assess next-token predictive fit.

2. Domain adaptation: a team measures whether a model predicts its specialist documentation better after adaptation, while separately checking practical question-answering quality.

3. Evaluation diagnosis: analysts compare perplexity across text categories to investigate where the model finds the language less predictable, without treating those differences as a complete quality assessment.

Think of it this way

Think of predicting the next card from a deck whose composition changes with the preceding cards. A predictor that gives more probability to the card actually drawn is less surprised. The analogy is limited because language has complex context and tokens are designed representations rather than fixed physical cards. Perplexity summarizes prediction uncertainty; it does not measure whether the resulting text is useful or true.

Worked example

Suppose a toy model assigns probabilities 1/2, 1/4, and 1/8 to three successive observed tokens. Their joint conditional probability is (1/2) × (1/4) × (1/8) = 1/64.

Perplexity is the inverse joint probability raised to the power 1/3: 64^(1/3) = 4. Using base-two logs gives negative log probabilities of 1, 2, and 3 bits. Their average is 2 bits per token, and 2^2 again gives perplexity 4.

A second model that gives each observed token probability 1/2 has joint probability 1/8 and perplexity 2. Under this exact three-token evaluation, it predicts the observed tokens better. That tiny comparison says nothing about factual accuracy, performance on another corpus, or generated responses. It also illustrates that perplexity uses a geometric calculation: averaging the inverse probabilities 2, 4, and 8 would give approximately 4.67, which is not the correct perplexity.

Illustrative case study

Fictional case: Silverleaf Manuals adapts a language model to equipment documentation. Its held-out manual text receives lower perplexity after adaptation, suggesting improved prediction of that domain's language.

The team then evaluates a separate task in which employees ask how to identify the right replacement component. Some answers remain unhelpful because the model selects a plausible component name without consulting the relevant specification. The team reports the perplexity improvement as a language-modeling result and keeps task success and factual checking separate. The example illustrates a measurement boundary, not an actual organization's experimental outcome.

Common misconceptions and mistakes

Perplexity is not a percentage and is not simply the number of mistakes. A model can assign a substantial probability to the correct token without ranking it first, and perplexity uses that probability rather than only a correct-or-incorrect decision.

Do not compare scores from incompatible tokenizers as if every token represented the same amount of text. Changes in vocabulary, context length, document handling, or evaluated data also matter. Training-set perplexity can be misleading as evidence of generalization.

A lower score does not certify truthful answers or successful downstream work. Also, averaging perplexities from separate segments is generally different from combining their token log losses and then exponentiating. Aggregate at the appropriate token-weighted log-loss level when that is the intended corpus measure.

Business use

Perplexity helps technical teams monitor language-model fitting and compare controlled experiments. For business decisions, pair it with evaluations of the actual feature, such as whether users find the right information or whether summaries preserve important facts.

Ask what text, tokenizer, and context protocol produced a reported score. A vendor's lower number is difficult to interpret without those details. Treat perplexity as evidence about predictive language fit under documented conditions, and use task-based checks to judge whether that fit translates into value.

Check your understanding

A model predicts each evaluated token from four equally likely alternatives, assigning the observed token probability 1/4 every time. What is its perplexity?

Answer: Four. Average negative log probability corresponds to two bits per token, and 2^2 = 4, regardless of the number of evaluated tokens in this toy setup.

Related terms

Learn next

Calculate perplexity for two short lists of assigned token probabilities using log losses. Then change the tokenization conceptually and explain why raw per-token scores would need care before comparison.

Tokenization

Sources

Jurafsky and Martin, Speech and Language Processing, N-gram language models, perplexity section: https://web.stanford.edu/~jurafsky/slp3/3.pdf

The authors' textbook grounds the probability definition and comparison limits. The numerical calculation, fictional case, and exercise are original. This entry concerns discrete-token language-model perplexity.

Previous: Perpetual punishment Next: Persistence arc