Is alpha restricted to between zero and one?

Not in every definition. Wider ranges exist, subject to support and finiteness requirements.

Is smaller always better?

Smaller means closer under the stated criterion, but usefulness still depends on the downstream task.

Is the half-alpha value exactly squared Hellinger distance?

Scaling conventions differ. Under the common definition H squared = half the sum of squared square-root differences, this entry's D_0.5 equals four times H squared.

AI Glossary / Probabilistic machine learning / Advanced

Alpha divergence

Definition

Alpha divergence is a parameterized family of measures for comparing probability distributions. Changing alpha changes how discrepancies are penalized. Under one common convention, the family approaches reverse Kullback - Leibler divergence as alpha approaches zero and forward Kullback - Leibler divergence as alpha approaches one. Because conventions differ, the formula, argument order, and normalization must accompany any reported alpha value.

What it means

Suppose a complicated distribution p describes uncertainty, while a simpler distribution q will stand in for it. There are different ways for q to be wrong: it might overlook regions where p places probability, or assign probability where p places very little. A divergence defines the mismatch that an approximation procedure tries to reduce.
Alpha provides a way to vary that criterion. It does not directly set the model's confidence, accuracy, or fairness. A small divergence is meaningful only relative to the distributions, definition, and task being considered, and it need not guarantee good performance for every downstream decision.

How it works

For normalized discrete distributions with strictly positive probabilities, one convention is D_alpha(p || q) = [1 − sum_i p_i^alpha q_i^(1−alpha)] / [alpha(1−alpha)], for alpha other than zero or one. Continuous versions replace the sum with an integral. The exceptional values are defined through limits rather than direct division by zero.
Under this convention, the zero limit is KL(q || p), the one limit is KL(p || q), and alpha = 0.5 gives twice the sum of squared differences between square-root probabilities. The general expression is asymmetric: exchanging p and q corresponds to replacing alpha with 1−alpha.
Unnormalized functions require the appropriate extended definition. Substituting arbitrary scores into the normalized formula is not valid simply because the scores are positive.

Three practical examples

A researcher compares simple posterior approximations using different divergence objectives. Evaluation checks uncertainty in the quantities that matter, rather than assuming one objective always produces the best approximation.
A team compressing a probabilistic model studies how strongly the chosen criterion penalizes overlooking low-probability outcomes. It validates those outcomes separately because their business consequences may be disproportionate to their probability.
An analyst compares probability distributions over a fixed set of demand categories. Before interpreting divergence values, the analyst aligns the categories, checks normalization, and documents the selected alpha convention.

Think of it this way

Think of comparing two maps using a scoring rule that decides which mapping mistakes are most costly. Some rules punish missing a populated region heavily; others strongly penalize drawing populated areas where there are almost none. Changing the rule changes which simplified map looks best, without changing the territory itself.

Worked example

Consider an original two-outcome example with p = (0.8, 0.2) and q = (0.5, 0.5). Both distributions are normalized and positive. At alpha = 0.5, the overlap sum is sqrt(0.8 × 0.5) + sqrt(0.2 × 0.5) = sqrt(0.4) + sqrt(0.1), approximately 0.9486833.
The denominator is 0.5 × 0.5 = 0.25, giving D_0.5 = (1 − 0.9486833)/0.25, approximately 0.2052668. Exchanging p and q leaves this particular value unchanged, because the square-root products are symmetric.
Using natural logarithms, the alpha-to-one limit is KL(p || q) = 0.8 log(1.6) + 0.2 log(0.4), approximately 0.1927448. The alpha-to-zero limit is KL(q || p) = 0.5 log(0.625) + 0.5 log(2.5), approximately 0.2231436. The two limits differ, demonstrating why argument order matters.
If q were replaced by p, the overlap sum would be 0.8 + 0.2 = 1 and the divergence would be zero. None of these values is a percentage of incorrect predictions. They measure distributional discrepancies under specified formulas, not a count of mistakes on labeled cases.

Illustrative case study

Fictional forecasting laboratory RangeWorks replaces an expensive uncertainty model with a smaller approximation. Two candidate approximations look similar around the most common demand level but differ in how much probability they assign to unusually high demand.
The team records the exact divergence objective used to fit each candidate and compares the resulting planning decisions on separate evaluation scenarios. It discovers that a favorable aggregate score does not settle which approximation is preferable for a costly shortage scenario. This fictional example illustrates objective selection and evaluation; it reports no verified implementation, measured savings, or universally superior alpha setting.

Common misconceptions and mistakes

Alpha divergence is generally not a metric: symmetry and the triangle inequality should not be assumed. The symmetric half-alpha case does not make every member of the family symmetric. Nor should alpha divergence be confused with Rényi divergence merely because both use a parameter named alpha and related overlap expressions.
A second common mistake is transferring a numeric alpha setting between references without checking definitions. Some parameterizations place the same special cases at different values. Zero probabilities also require care with limits and support conditions; certain divergences can be infinite. Adding smoothing changes the distributions being compared and should be disclosed.

Business use

The concept helps explain why two probabilistic approximations can behave differently even when they use similar model families. For practical comparisons, document the formula, probability support, normalization, argument order, and numerical procedure. Evaluate the decisions supported by the approximation, including important tail events, rather than selecting a model solely from a divergence number. If the true target distribution is unavailable, distinguish an estimated objective from an exactly computed comparison.

Check your understanding

Why do the two KL limits differ in the example?

Answer: They reverse which distribution weights the log ratio.

Why not substitute alpha = one directly?

Answer: The displayed expression gives a zero denominator; the value comes from a limit.

What must accompany an alpha setting?

Answer: At least the formula convention and the order of its distribution arguments.

Related terms

f-divergence
Rényi divergence
Expectation propagation
Probability distribution
Kullback–Leibler divergence
Hellinger distance
variational inference
support

Learn next

Study forward and reverse KL divergence with small discrete examples. Then compare their support requirements and examine how approximation-family restrictions affect the result of minimizing a chosen divergence.

Sources

Minka, Divergence Measures and Message Passing, Microsoft Research Technical Report MSR-TR-2005-173, Section 2: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2005-173.pdf

Previous: Alpha channel Next: Alpha-expansion