AI Glossary / Probabilistic machine learning / Intermediate

Log-sum-exp trick

Definition

The log-sum-exp trick is a numerically stable way to calculate the logarithm of a sum of exponentials. It subtracts a common reference value before exponentiation and adds that value back after taking the logarithm.

What it means

Probability models often work with logarithms because multiplying many small probabilities otherwise produces numbers too tiny for a computer to represent. Eventually, however, the model may need to add alternatives rather than multiply them. Adding their logarithms would give the wrong answer.

For real values x1 through xn, log-sum-exp means log(exp(x1) + ... + exp(xn)). Direct calculation can overflow when an input is large, even though the final logarithm would be representable. Very negative inputs can instead make every exponential round to zero.

The trick rearranges the same mathematical expression. It does not discard smaller alternatives or replace a sum with its largest member. Ordinary finite-precision rounding still exists, but avoidable extremes in intermediate calculations are reduced.

How it works

Choose m as the largest input. Calculate m + log(sum(exp(xi − m))). Every shifted input is nonpositive, so its exponential is at most one, and at least one exponential equals one for a nonempty list of finite real inputs.

The identity follows by factoring exp(m) out of the original sum. The logarithm of that factor is m. Because the subtraction and later addition cancel algebraically, changing the reference does not change the exact result.

Use a tested library routine for production calculations. It can handle array axes, weighting and special values more carefully than a casual manual implementation. Cases involving positive infinity, all negative infinities, empty inputs or signed weights require explicit semantics; the simple finite-input explanation does not settle every such case.

Three practical examples

• A classifier normalizes large unscaled class scores without first creating enormous exponentials.
• A mixture model adds component likelihood contributions that are already stored as logarithms.
• A sequence model combines alternative path scores in the log domain while preserving contributions from more than one path.

Think of it this way

Think of measuring mountain heights from a nearby elevated platform rather than from sea level. The relative distances become manageable, and the platform height is restored when reporting the final result.

Unlike a physical measurement, this is an exact algebraic rearrangement before computer rounding. The platform analogy explains the reference shift; it does not explain the exponential weighting of different inputs.

Worked example

Three large scores, one finite result

Take x = (1000, 999, 998). Directly exponentiating 1000 overflows common floating-point formats. Choose m = 1000 and work with the shifted scores (0, −1, −2).

Their exponentials are approximately (1, 0.367879, 0.135335), which sum to 1.503214. Taking the logarithm and restoring m gives approximately 1000.407606.

The corresponding normalized probabilities are approximately (0.665241, 0.244728, 0.090031). They can be calculated from the shifted exponentials divided by their sum; the original huge exponentials are unnecessary.

Now subtract 2000 from every original score. The inputs become (−1000, −1001, −1002), and log-sum-exp becomes −999.592394. The normalized probabilities remain unchanged because every score received the same shift.

For three finite inputs, the result lies between their maximum and that maximum plus log(3). Here 1000.407606 lies between 1000 and approximately 1001.098612. This bound provides a useful check, but it does not replace evaluating the expression.

Illustrative case study

A fictional recommendation team notices that a score normalization step sometimes returns invalid probabilities for unusually confident model outputs. The underlying ranking is reasonable, but the implementation exponentiates raw scores first.

The team replaces that calculation with a stable normalization routine and adds examples containing both large positive and large negative scores. It checks that the probabilities sum to one and that shifting every score by the same constant preserves them.

The repair addresses arithmetic failure. It does not establish that confidence estimates are calibrated or that recommendations are useful. Those questions still require separate evaluation against observed outcomes.

Common misconceptions and mistakes

• Adding log probabilities is always correct: that represents multiplication of the original values, not addition of alternatives.
• The result equals the largest input: the maximum is a useful lower bound, but other inputs generally contribute.
• Stability improves model accuracy automatically: it prevents certain numerical failures; it cannot correct a misspecified probability model.
• Exponentiating a stable log result is always safe: the final ordinary-scale value may still be outside the machine's representable range.

Business use

This technique supports dependable numerical infrastructure in probabilistic services, classifiers and ranking systems. It is especially useful when rare events, long sequences or large scores create extreme intermediate values.

Teams reviewing an implementation should distinguish numerical correctness from statistical quality. Check the calculation's axis, treatment of masked alternatives and handling of impossible events. A stable sum across the wrong dimension remains the wrong calculation. Keep values in the log domain when later operations permit it instead of repeatedly converting between representations.

People also ask

Is log-sum-exp the same as softmax?

No. Log-sum-exp returns a scalar normalizing quantity along a chosen axis; softmax returns normalized values. The calculations are closely related.

Does it change which score is largest?

A shared shift preserves score ordering and normalized probabilities for finite inputs.

What if every alternative has log probability negative infinity?

Their total probability is zero, so the log total is negative infinity. A naive subtraction of the maximum needs special handling in this case.

Check your understanding

Two inputs both equal 50. What is their log-sum-exp?

Answer: It is 50 + log(2), approximately 50.693147. Each shifted exponential is one, so both alternatives contribute. Returning 50 would ignore the second contribution.

Related terms

• Log probabilities
• Softmax
• Forward algorithm
• Log probability
• numerical stability
• floating-point arithmetic
• log-softmax
• mixture model
• partition function

Learn next

Practice translating multiplication and addition of positive numbers into log-domain operations. Then derive softmax from a log normalizer and test common-shift invariance. Study special floating-point values before implementing masked or weighted versions of the calculation.

• Softmax

More in Probabilistic machine learning

• Local level model - A local level model represents a time series as noisy observations of an underlying level that changes through a random walk.
• Local linear trend model - A local linear trend model is a state-space time-series model with both a changing level and a changing slope.
• Log-Gaussian Cox process - A log-Gaussian Cox process is a model for randomly located events in which the logarithm of an underlying event intensity follows a Gaussian process.
• Manifestation shift - Manifestation shift is a change in how a target condition produces the observable physical characteristics used to recognize it.
• Masked autoregressive flow - A masked autoregressive flow, or MAF, is a normalizing-flow model that uses masked neural networks to transform observed data into a simpler base distribution.
• Matching network - A matching network is a few-shot classification model that predicts a query's label by attending to labeled examples in a support set.

Sources

SciPy documentation, scipy.special.logsumexp, stable calculation and supported options: https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.logsumexp.html

The algebraic derivation, numerical examples and fictional case are original teaching illustrations. The explanation uses natural logarithms and finite real inputs unless a special case is explicitly identified.

Previous: Log shipping Next: Log trigger