Are all f-divergences symmetric?

No. KL and Pearson chi-squared generally depend on direction; total variation is symmetric.

Does zero always imply equality?

Additional conditions on the generator are needed; strict convexity at one is sufficient in the standard setting.

Is an f-divergence a hypothesis test?

Not by itself. A testing procedure also needs sampling assumptions and a way to calibrate evidence.

AI Glossary / Probabilistic machine learning / Intermediate

f-divergence

Definition

An f-divergence is a measure of difference between probability distributions generated by a chosen convex function f. It compares their probability or density ratios and combines the results into a nonnegative value when f(1) = 0. Familiar members include Kullback - Leibler divergence, total variation and chi-squared divergence, with conventions stated explicitly.

What it means

Two models may assign different probabilities to the same outcomes. An f-divergence offers a systematic way to describe that disagreement. The function f controls how strongly particular probability-ratio discrepancies contribute, so choosing a divergence is part of choosing what kind of mismatch matters.

The letter f is a placeholder for a mathematical function, not a particular fixed score. Different members have different scales, support behavior and symmetry properties. A divergence is not generally an ordinary distance: swapping the distributions can change the answer, and the triangle inequality need not hold. Its usefulness depends on the question being asked.

How it works

For discrete distributions with positive reference probabilities q_i, use D_f(P || Q) = sum over i of q_i f(p_i/q_i). In continuous settings, the sum becomes an integral with densities relative to a common reference measure. Zero probabilities require the appropriate limiting conventions; casually dividing by zero is not a valid implementation.

With f(t) = t log(t), the formula gives KL(P || Q). With f(t) = (t − 1) squared, it gives Pearson chi-squared divergence. With f(t) = one-half |t − 1|, it gives total variation under the convention of one-half the absolute probability differences.

Convexity and normalization give nonnegativity. Strict convexity at one is a sufficient condition for zero divergence to imply identical distributions. Not every convex generator has that stronger identification property. Implementations should state direction, normalization and logarithm base where relevant.

Three practical examples

1. Model assessment: compare an estimated category distribution with a reference distribution using a specified divergence. Inspect individual categories as well as the total score, especially when rare categories matter operationally.

2. Distribution monitoring: track changes in the mix of incoming requests relative to a documented baseline. Distinguish statistical variation from changes large enough to affect staffing or model performance.

3. Probabilistic inference: optimize a chosen divergence between an approximate distribution and a target. The direction and family influence which kinds of approximation error receive the strongest penalty.

Think of it this way

Think of inspecting two recipes using different discrepancy rules. One rule counts total changes in ingredient proportions; another sharply penalizes adding an ingredient that the reference almost excludes. The choice changes the score. The analogy concerns comparison rules, not flavor: an f-divergence compares probability distributions and does not know which outcomes people value.

Worked example

Let P = (0.75, 0.25) and Q = (0.5, 0.5). Their ratios p_i/q_i are 1.5 and 0.5. For Pearson chi-squared divergence, each squared ratio deviation is 0.25. Weighting by Q gives 0.5 × 0.25 + 0.5 × 0.25 = 0.25.

Using the total variation convention above gives one-half × (|0.75 − 0.5| + |0.25 − 0.5|) = 0.25. The equal numerical values are coincidental; these are different measures.

Reverse the chi-squared comparison. The squared probability differences remain 0.0625 each, but their denominators are now 0.75 and 0.25. Thus chi-squared(Q || P) = 0.0625/0.75 + 0.0625/0.25, approximately 0.3333. Direction has changed the answer. Total variation remains 0.25 because its formula is symmetric. This calculation uses exact stipulated probabilities, not noisy frequency estimates from a sample.

Illustrative case study

Fictional case: Ash Service monitors the categories of support requests. Its dashboard shows a growing distribution discrepancy, and a manager proposes an automatic staffing change. The analyst first examines the underlying category counts and discovers that the comparison mixes a holiday week with an ordinary baseline.

The team defines suitable comparison periods, records the divergence convention and uses the score as a signal to investigate. It also checks whether the changing categories require different handling times. This fictional workflow illustrates why a distribution score alone does not determine the correct business response.

Common misconceptions and mistakes

A divergence value is not automatically a probability, a percentage of mistakes or a statistical significance level. Different generators can produce scores on different scales. Even the name total variation may use a factor-of-two convention, so formulas matter more than labels.

Zero counts in a finite sample do not prove that an outcome is impossible. Smoothing or pseudocounts change the estimated distributions and therefore the score. For continuous high-dimensional data, empirical distributions and density estimation need particular care. Comparing separate feature distributions may miss important changes in their joint relationships.

Business use

Choose the comparison around the decision. State what P and Q represent, how probabilities are estimated, and which errors deserve attention. Check whether a change in binning, vocabulary or category definitions creates an artificial jump.

For monitoring, examine score variation under an appropriate stable baseline before setting action thresholds. For model selection, pair distributional comparison with the downstream performance that matters. An overall score can hide a small but consequential category, while an alarming score can reflect a harmless reclassification. Retain the underlying distributions so reviewers can interpret the result.

Check your understanding

Why does reversing Pearson chi-squared divergence change the worked example's answer?

Answer: The reference distribution supplies the denominators. Reversing P and Q changes those denominators, even though the squared probability differences stay the same.

Related terms

Total Variation Distance (Tvd)
Convex function
Probability distribution
Kullback–Leibler divergence
total variation distance
Pearson chi-squared divergence
density ratio
distribution shift
variational inference

Learn next

Calculate several divergences on the same small probability table and then introduce a very small reference probability. Observe how support and direction affect the result before using distribution scores on estimated or high-dimensional data.

Sources

Sason and Verdú, f-Divergence Inequalities, definition and basic properties: https://webee.technion.ac.il/people/sason/f-divergence%20inequalities.pdf

Independent review checked the primary definition and Proposition 1 convexity condition. This entry explicitly uses the half-L1 convention for total variation; the numerical comparison and operational examples are original.

Previous: F distribution Next: F-logic