AI Glossary / Statistics and probability / Intermediate
Cross-entropy
Definition
Cross-entropy is the expected negative logarithm of probabilities assigned by one distribution to outcomes generated by another. For discrete distributions p and q, it is H(p,q) = -sum p_i log q_i. It is widely used to assess probabilistic predictions and provides the basis of cross-entropy loss in machine learning.
What it means
The reference distribution p supplies the weights: how often outcomes occur or how much target probability they receive. The proposed distribution q supplies the probabilities being assessed. Assigning very little probability to an outcome that occurs often produces a large contribution.
Cross-entropy is directional. Swapping p and q generally changes the value. It is also not a distance that becomes zero whenever distributions match. When q equals p, cross-entropy equals the entropy of p, which can be positive.
This entry focuses on discrete outcomes, such as classes and tokens. Continuous-density versions require additional care and do not share every property of the discrete quantity, including nonnegativity.
How it works
Specify a common outcome space and valid probability distributions. Multiply each reference probability p_i by -log q_i and sum. Terms with p_i = 0 contribute zero under the usual convention. If p_i is positive while q_i is zero, the discrete cross-entropy is infinite.
Natural logarithms give units called nats; base-two logarithms give bits. Use one convention consistently when comparing scores.
The identity H(p,q) = H(p) + D_KL(p||q) separates reference uncertainty from the extra penalty associated with q. For fixed p, minimizing cross-entropy over q is equivalent to minimizing this KL divergence. In prediction tasks, p often represents an empirical target distribution or one-hot observed label rather than a fully known population distribution.
Three practical examples
- A classifier's predicted probabilities are assessed using observed category labels, distinguishing confident mistakes from tentative mistakes.
- A language model is evaluated on how much probability it assigns to the actual next tokens in held-out text under a fixed tokenization scheme.
- A smaller model learns from a teacher's soft probability targets, where matching the full target distribution matters rather than only copying its most likely class.
Think of it this way
Think of preparing a codebook for messages. A codebook designed around q makes expected outcomes cheap to describe and unexpected ones expensive. Cross-entropy is the average idealized coding cost when messages actually follow p. A mismatched codebook adds avoidable cost beyond the uncertainty already present in the messages.
Worked example
Consider an original binary example with reference distribution p = [0.75,0.25] and proposed distribution q = [0.60,0.40]. Using natural logarithms, H(p,q) = -0.75 ln(0.60) - 0.25 ln(0.40), approximately 0.61219 nats.
The entropy of p is -0.75 ln(0.75) - 0.25 ln(0.25), approximately 0.56234 nats. The difference is approximately 0.04986 nats, equal to D_KL(p||q).
If q is changed to match p exactly, cross-entropy falls to 0.56234, not zero. The reference distribution still has uncertainty. If q assigns probability one to the first outcome and zero to the second, cross-entropy becomes infinite because p gives the second outcome positive probability.
Reversing the original distributions gives H(q,p), approximately 0.72713 nats, demonstrating asymmetry. These calculations use specified distributions, not a fitted model or an estimate of business value. They show why direction, support, and logarithm base belong in the definition of a reported score.
Illustrative case study
Fictional case: Maple Routing compares two ticket classifiers with identical top-label accuracy. One model makes occasional extremely confident wrong predictions, while the other distributes probability more cautiously on difficult tickets.
The analysts compare held-out cross-entropy as well as accuracy and category-specific performance. They inspect examples with large loss contributions and review whether the labels themselves are reliable.
The result informs model selection, but the team also checks calibration and the costs of routing errors. A lower average cross-entropy is useful evidence about probabilistic prediction on the evaluation distribution; it does not by itself guarantee that every important customer group receives better service.
Common misconceptions and mistakes
- Cross-entropy and entropy are different quantities unless q equals p.
- Cross-entropy is not symmetric and is not generally zero for matching distributions.
- Lower training cross-entropy does not guarantee lower held-out loss; overfitting remains possible.
- Accuracy uses selected labels, while cross-entropy uses probability assignments. They can rank models differently.
- Comparisons require compatible outcomes, data weighting, logarithm bases, and aggregation conventions.
- A low average score can hide poor performance on rare classes or particular groups.
- In continuous settings, densities can exceed one, so negative differential cross-entropy is not automatically an error.
Business use
Cross-entropy can help assess AI systems whose probability estimates matter, including routing, forecasting categorical events, and language modeling. Record the evaluation sample, target representation, class weighting, and whether the reported figure is a sum or mean.
Use a separate evaluation set and inspect the distribution of individual losses. A small number of mislabeled or unusually difficult examples can have a large influence, especially when predictions are highly confident. Investigate rather than silently deleting inconvenient cases.
Connect the metric to operational decisions. Probability quality matters differently depending on how predictions are used. A decision system may need calibrated thresholds, error-cost analysis, or abstention in addition to an improved average cross-entropy.
People also ask
Is cross-entropy only for neural networks?
No. It applies to probability distributions and many kinds of probabilistic models.
Can it be zero?
For discrete outcomes, it can be zero in a degenerate perfectly predicted case, but not for every pair of matching distributions.
How is it related to perplexity?
For compatible average token cross-entropy, perplexity is its exponential using the matching logarithm base: exp(H) for nats or 2 raised to H for bits.
Check your understanding
Answer: 1.
Which distribution weights the terms in H(p,q)?
Answer: The reference distribution p. 2.
Why is perfect matching not necessarily zero?
Answer: The reference entropy remains. 3.
Why does zero q probability create trouble?
Answer: A positive-reference-probability outcome then receives an infinite negative-log penalty.
Related terms
Learn next
Learn logarithms, probability distributions, and entropy, then derive the entropy-plus-KL identity. Compare accuracy and cross-entropy on identical predictions. Study stable loss computation and held-out evaluation before interpreting training curves as evidence of useful improvement.
More in Statistics and probability
- Critical value - A critical value is a boundary used in a statistical decision rule, commonly separating rejection and nonrejection regions for a hypothesis test.
- Cromwell's rule - Cromwell's rule is advice to avoid assigning exact prior probabilities of zero or one to uncertain empirical propositions.
- Cross-correlation matrix - A cross-correlation matrix describes pairwise relationships between components of two random vectors.
- Cross-sectional regression - Cross-sectional regression models relationships between an outcome and explanatory variables across different units observed for a common point or period in...
- Crossover study - A crossover study compares interventions by having the same participants receive different interventions during successive periods.
- Cumulative distribution - A cumulative distribution describes how probability accumulates up to each value of a variable.
Sources
Google, Machine Learning Glossary, Cross-entropy: https://developers.google.com/machine-learning/glossary#cross-entropy
Zhang, Lipton, Li, and Smola, Dive into Deep Learning, Information Theory: https://classic.d2l.ai/chapter_appendix-mathematics-for-deep-learning/information-theory.html
The primary educational definitions and cross-entropy identity were read. The binary calculations and fictional routing case are original. Perplexity uses the standard positive-exponent convention.
Previous: Cross-dissolve Next: Cross-entropy loss