No. It is nonnegative, but generally asymmetric and does not satisfy all the requirements of a metric.

Yes. Positive probability under P combined with zero probability under Q can make D(P||Q) infinite.

Does zero KL prove a model is useful?

It shows agreement between the compared distributions under the definition's conditions. It does not establish that the selected outcomes, data, or business objective are appropriate.

AI Glossary / Mathematics and optimization / Intermediate

Kullback-Leibler divergence

Definition

Kullback-Leibler divergence measures how one probability distribution differs from another in a particular direction. For discrete outcomes, D(P||Q) is the sum of P(x) times log(P(x)/Q(x)). It is nonnegative and equals zero when the distributions agree, but it is generally asymmetric and is not a mathematical distance.

What it means

The first distribution supplies the weighting. Events that occur often under P have more influence on D(P||Q) than events that rarely occur under P. This direction matters when comparing a target distribution with a model, approximating a complex distribution, or examining changes in the proportions of different outcomes.
An information-theoretic interpretation is the extra expected log loss from using Q instead of P when outcomes actually follow P. This statement concerns an expectation under the specified distribution, not a guarantee about every observation or every finite dataset. The value also depends on the logarithm base: natural logarithms give nats, while base-two logarithms give bits.

How it works

List the shared outcome categories and their probabilities under both distributions. For each category with positive P, calculate the probability ratio, take its logarithm, multiply by P, and add the contributions. Contributions from categories with P=0 are treated as zero.
If P assigns positive probability to an event that Q declares impossible, the divergence is infinite under the standard extended-real definition. Smoothing can avoid a zero estimate in an application, but it changes the distribution being evaluated and must be explained.
For continuous distributions, the corresponding expression uses an integral of p(x) log(p(x)/q(x)). Comparing densities requires a common underlying measure. A histogram calculation is a discretized comparison whose result depends on the bin definitions.

Three practical examples

A language-model team can compare a target next-token distribution with predicted probabilities through expected log loss, while recognizing that the full target distribution is usually unknown.
An analyst can compare category proportions across two datasets to summarize distribution change in a chosen direction.
A probabilistic-modeling team can use a specified KL objective to choose an approximation, remembering that reversing the direction changes the optimization problem.

Think of it this way

Think of a forecasting guide that assigns confidence to possible events. A guide is especially costly when it gives very little probability to something that actually happens frequently. The analogy explains the directional penalty, but KL is a precise distribution-level calculation; it does not directly measure money, fairness, or the usefulness of individual predictions.

Worked example

Consider an original three-category example with P=(0.5,0.3,0.2) and Q=(0.25,0.5,0.25). Using natural logarithms, the three terms in D(P||Q) are 0.5 log(2), 0.3 log(0.6), and 0.2 log(0.8). They are approximately 0.346574, −0.153248, and −0.044629. Their sum is approximately 0.148697 nats.
Two contributions are negative because Q gives those categories more probability than P does. This does not violate nonnegativity: the complete weighted sum is nonnegative, not every individual contribution.
Now reverse the roles. D(Q||P)=0.25 log(0.5)+0.5 log(5/3)+0.25 log(1.25), approximately 0.137912 nats. The unequal answers demonstrate why the direction must be stated.
The entropy of P is approximately 1.029653 nats, and its cross-entropy against Q is approximately 1.178350 nats. Subtracting the entropy from the cross-entropy gives the same 0.148697-nat divergence. Finally, replacing Q's first probability with zero while P's first probability remains 0.5 would make D(P||Q) infinite, rather than merely a large finite number.

Illustrative case study

In a fictional subscription business, analysts compare support-request categories between a reference period and a new period. They report a KL value together with the direction, category definitions, and actual proportions. They also inspect which categories contribute most to the result.
When a newly created category has no recorded reference cases, the team does not silently substitute a small number. It records how zero counts are handled and checks whether the apparent change reflects a revised labeling policy. The divergence becomes one diagnostic in an investigation, rather than an automatic explanation of why customer behavior changed. This scenario illustrates a process and does not describe measured company results.

Common misconceptions and mistakes

Do not describe KL as a symmetric distance or assume it satisfies the triangle inequality. Do not compare values computed with different logarithm bases without conversion. A zero sample count is not proof that the underlying probability is zero.
Distinguish the true divergence from an estimator based on limited data. Sampling, density estimation, and numerical approximation introduce uncertainty; a negative numerical estimate does not mean the exact KL definition permits negative divergence. Also, minimizing cross-entropy is equivalent to minimizing KL only when the target distribution is fixed, so its entropy is a constant.

Business use

Business teams should connect the comparison to a concrete question, such as whether the mix of requests has changed enough to investigate. Record the reference distribution, direction, sample sizes, category mapping, zero-handling rule, and log base. Pair the scalar result with a readable table of the underlying probabilities.
There is no universal KL threshold that separates acceptable change from harmful change. Operational significance depends on the consequences of the affected categories and on estimation uncertainty. A small aggregate divergence can still accompany an important change in a rare, costly event.

Check your understanding

If D(P||Q)=0.2, must D(Q||P) also equal 0.2?

Answer: No. Reversing the distributions changes both the probability weights and the logarithmic ratios, so the divergence can differ substantially.

Related terms

Entropy
Cross-entropy
Probability distribution
relative entropy
Jensen-Shannon divergence
maximum likelihood
variational inference
distribution shift

Learn next

Next, compare entropy, cross-entropy, and KL using one small probability table. Then study Jensen-Shannon divergence to see how a related symmetric comparison is constructed, without assuming it shares every property of KL.

Sources

Primary reference: Imre Csiszár and Paul C. Shields, Information Theory and Statistics: A Tutorial, preliminary definitions and conventions for divergence: https://www.renyi.hu/~csiszar/Publications/Information_Theory_and_Statistics:_A_Tutorial.pdf . The cited tutorial uses base-two logarithms; the original worked example here explicitly uses natural logarithms.

Previous: Kuiper's test Next: Kurtosis