Can entropy exceed one bit?

Yes. A fair four-outcome variable has two bits of entropy; one bit is the maximum only for a binary variable.

Is entropy a probability?

No. It is an average information quantity measured in units determined by the logarithm.

Can two different distributions have the same entropy?

Yes. Entropy summarizes uncertainty and does not uniquely identify all probabilities.

AI Glossary / Statistics and probability / Intermediate

Entropy

Definition

Entropy, in information theory, measures the average uncertainty associated with a probability distribution. For a discrete random variable with probabilities p_i, Shannon entropy is H = -sum p_i log(p_i), with zero-probability terms treated as zero. The logarithm base determines the units: base two gives bits, while the natural logarithm gives nats. Entropy concerns probabilistic uncertainty, not the truth, importance, or meaning of a message.

What it means

An outcome that is almost certain provides little surprise when it occurs. A rare outcome provides more. Entropy averages that surprise across all possible outcomes according to their probabilities.
For a fixed finite set of k possible outcomes, entropy is largest when they are equally likely, giving log(k) in the chosen units. It is zero when one outcome has probability one. These statements concern a specified distribution and outcome set; changing the categories or measurement precision can change the quantity being measured.

How it works

Start by defining the random variable and its possible values. Obtain probabilities from a model or estimate them from data, making clear which is being used. Probabilities must be nonnegative and sum to one.
For each positive probability p, calculate the information content, or surprisal, -log(p). Multiply by p and sum across outcomes. Zero-probability terms contribute zero through the limiting convention, so implementations should not directly evaluate log(0).
Entropy of a model's predicted class distribution measures how concentrated that prediction is. Low entropy means the model favors a smaller set of outcomes; it does not prove the favored outcome is correct. A poorly calibrated model can be confidently wrong.
In a decision tree, label entropy can quantify impurity at a node. Information gain compares the parent's entropy with the weighted average entropy of the children. In compression, source entropy is connected to achievable average coding rates under appropriate assumptions. Dependencies among successive symbols require considering conditional entropy or entropy rate rather than only single-symbol frequencies.
Distinguish entropy from cross-entropy, which evaluates one distribution using another distribution's log probabilities. Also distinguish discrete entropy from differential entropy for continuous variables. Differential entropy can be negative and changes with measurement units, so it cannot simply inherit every interpretation of the discrete formula.
Empirical entropy calculated from observed frequencies is an estimate. Small samples, rare categories, and changing category definitions can affect comparisons. A difference in entropy does not identify the cause of a distributional change.

Three practical examples

A classifier uses predictive entropy as one input to a reviewed uncertainty-routing policy, with thresholds evaluated on representative cases.
A decision-tree algorithm compares candidate splits by their reduction in class-label entropy.
A data-compression designer uses symbol probabilities to give common symbols shorter codes and rare symbols longer codes.

Think of it this way

Think of a guessing game. If one answer occurs almost every time, guessing is easy. If several answers are equally plausible, there is more uncertainty before the answer is revealed. Entropy measures average uncertainty under the probabilities; it does not measure how useful or interesting the answer is.

Worked example

Consider an original five-category source with probabilities 1/2, 1/4, 1/8, 1/16, and 1/16. Using base-two logarithms, the corresponding surprisals are one, two, three, four, and four bits.

Entropy is (1/2)(1) + (1/4)(2) + (1/8)(3) + (1/16)(4) + (1/16)(4) = 0.5 + 0.5 + 0.375 + 0.25 + 0.25 = 1.875 bits per symbol.

An equally likely five-category source would have entropy log2(5), approximately 2.322 bits. The unequal source is more predictable on average, although each of its least likely outcomes has four bits of surprisal when it occurs.

Assign prefix-free codes 0, 10, 110, 1110, and 1111 to the five categories. No code is the beginning of another code, and the probability-weighted average length is also 1.875 bits. This exact match works for these particular probabilities. It does not mean every individual symbol requires 1.875 physical bits or that every distribution has an integer-length single-symbol code matching its entropy exactly.

Illustrative case study

Fictional case: Larch Assist routes customer messages to four departments. The team proposes sending high-entropy predictions to reviewers because those predictions are spread across several labels.
Testing reveals another problem: some unfamiliar messages receive low-entropy but incorrect predictions. The team retains entropy as a useful signal while adding representative error analysis and monitoring for unfamiliar inputs. It also checks whether the department definitions have changed. The workflow uses concentration of predicted probabilities without equating that concentration with validated knowledge.

Common misconceptions and mistakes

Mistake: saying high entropy means low quality. High uncertainty may accurately describe an ambiguous problem.
Mistake: saying low entropy guarantees correctness. Confidence and accuracy are different properties.
Mistake: comparing entropy values with different logarithm bases or category definitions without adjustment.
Mistake: confusing a rare event's surprisal with the average entropy of the whole distribution.
Mistake: applying the finite-category maximum rule without specifying the fixed outcome set.
Mistake: assuming continuous differential entropy is always nonnegative. It is a different quantity with different scaling behavior.

Business use

Businesses use entropy to summarize diversity or uncertainty, support classification workflows, and understand compression or information-gain methods. The number becomes meaningful only alongside the defined outcomes and the probabilities' origin. When used in operational decisions, validate the relationship between entropy, errors, review costs, and missed cases. A single entropy score cannot replace assessment of calibration, fairness, relevance, or the consequences of a wrong prediction.

Check your understanding

Which example outcome has the greatest surprisal?

Answer: Either outcome with probability one-sixteenth, each contributing four bits when observed.

Why is the source entropy smaller than log2(5)?

Answer: Its five outcomes are not equally likely, so the source is more predictable on average.

Related terms

Learn next

calculate binary entropy as probabilities change, compare entropy with cross-entropy, and investigate why confident model outputs can still be miscalibrated.

Cross-entropy

Sources

Sources: Shannon, A Mathematical Theory of Communication, original corrected reprint: https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf . Aalto Dictionary of Machine Learning, entropy: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . Google Machine Learning Glossary, entropy: https://developers.google.com/machine-learning/glossary . The five-category calculation and prefix code are original teaching examples.

Previous: Entropically regularized optimal transport Next: Entropy coding