Is per-word entropy always a whole number?

No. It is an average and can be fractional.

Does a larger vocabulary always imply a larger measured entropy?

No. Probabilities and dependencies also matter.

Is perplexity measured in bits?

No. Perplexity is obtained by exponentiating the appropriately based entropy or cross-entropy value.

AI Glossary / Statistics and probability / Intermediate

Per-word entropy

Definition

Per-word entropy expresses uncertainty in a word-generating process on an average-per-word basis. For a length-n sequence, it can mean the joint entropy of the sequence divided by n. The entropy rate is the limiting value as n grows, when that limit exists. With base-two logarithms, the unit is bits per word.

What it means

A word that is difficult to predict carries more information when it occurs than a highly predictable word. Per-word entropy summarizes expected uncertainty across possible sequences, taking their probabilities and dependencies into account.
It is not a count of interesting ideas, factual accuracy, or literary quality. Two corpora can differ in predictability for reasons such as vocabulary, repetition, topic, segmentation, or the amount of context available. The definition of a word is part of the measurement.

How it works

Specify the source process and its probability distribution over word sequences. The entropy of an entire sequence is the probability-weighted average of its negative log probability. Dividing by sequence length expresses this uncertainty per word.
The chain rule decomposes sequence entropy into conditional entropies: the uncertainty of each word given the words before it. Dependence matters. An independent-word model ignores the reduction in uncertainty that context may provide.
For an independent, identically distributed word source, per-word entropy equals the entropy of its single-word distribution. For a dependent source, the single-word marginal entropy can be larger than the entropy rate because preceding words contain predictive information.
Real language does not provide its true generating probabilities. Evaluators normally use a model's probabilities on held-out text and calculate average negative log probability. This is an empirical cross-entropy estimate, rather than direct knowledge of the source's true entropy.
For a matched distribution and a normalized model, expected cross-entropy cannot be lower than entropy; their difference measures model mismatch through relative entropy. A finite test sample can fluctuate, so this expectation-level statement is not a guarantee about every observed estimate.
Exponentiating a base-two cross-entropy value gives perplexity. Changing the logarithm base changes the numerical unit: natural logarithms give nats, not bits. Always report the convention.
Use the same evaluation text, segmentation, context rules, and treatment of boundaries for comparisons. Modern language models often operate on subword tokens. A per-token score cannot simply be relabeled per-word entropy. Comparisons across tokenizers need a common normalization and a clearly defined probability calculation.

Three practical examples

Language-model evaluation: researchers compare held-out average log losses under a consistent word definition.
Text compression: engineers study how predictable a source is and how closely a coding scheme approaches an information limit.
Domain analysis: a team examines whether a model finds a new document collection less predictable than its familiar material.

Think of it this way

Think of a word-guessing game. If the previous words make the next word almost certain, little uncertainty remains. Per-word entropy averages the remaining uncertainty across the process rather than judging whether the resulting sentences are useful.

Worked example

Consider an invented independent word source with three possible words having probabilities 0.50, 0.25, and 0.25. Its per-word entropy is -(0.50 log2 0.50 + 0.25 log2 0.25 + 0.25 log2 0.25) = 1.5 bits.
An evaluator that predicts all three words equally has expected cross-entropy log2 3, approximately 1.58496 bits per word. The excess is about 0.08496 bits, reflecting the mismatch between the uniform model and the source. The source entropy corresponds to perplexity 2 raised to 1.5, approximately 2.82843; the uniform model's expected cross-entropy corresponds to 3.
Separately, suppose a model assigns conditional probabilities 0.5, 0.25, 0.125, and 0.5 to four observed words. Their negative log probabilities are 1, 2, 3, and 1 bits. The observed average is 7/4 = 1.75 bits per word. This four-word score is an empirical model loss, not a reliable estimate of an entire language's true entropy.

Illustrative case study

Fictional case: Cedar Language Tools compares two document models on the same reserved collection. The team fixes segmentation and document-boundary rules before scoring. A lower cross-entropy identifies better average probability assignments on that collection, but the team separately evaluates factual answers and user tasks before choosing a product configuration.

Common misconceptions and mistakes

Do not treat entropy and cross-entropy as interchangeable without stating what distribution supplies the probabilities. Avoid comparing scores calculated per word with scores calculated per subword token.
Low uncertainty can arise from repetitive text or data leakage. It is not proof of broad understanding.
A particular word's surprisal is a realized information value; entropy is an expectation. A model assigning zero probability to an observed event produces infinite log loss, which must not be hidden by silently dropping that event.

Business use

The concept helps explain language-model loss, compression limits, and changes in text predictability. For business decisions, pair it with task success, cost, reliability, and domain-specific evaluation. A small improvement in average prediction loss does not automatically translate into a specified commercial benefit.

Check your understanding

Why is the invented source's entropy 1.5 bits rather than log2 3?

Answer: Its three words are not equally probable.

Why is the four-word calculation called a model score?

Answer: It uses the model's assigned probabilities on one observed sequence.

Related terms

Learn next

Learn next how the probability chain rule links sequence likelihood and next-word prediction, and how held-out evaluation distinguishes predictive fit from memorization.

Chain rule

Sources

Sources: Jurafsky and Martin, Speech and Language Processing, Chapter 3, especially Section 3.7, https://web.stanford.edu/~jurafsky/slp3/3.pdf . This author-hosted draft explains normalized sequence entropy, entropy rate, model cross-entropy, and perplexity. The toy source, calculations, and fictional case are original illustrations.

Previous: Per-comparison error rate Next: Perceiver