What represents a zero probability?

Its log is negative infinity in an extended-value convention; there is no finite real logarithm of zero.

Can I recover the original probability?

Yes. For natural logs, exponentiate the value; for base-two logs, raise two to that value.

AI Glossary / Natural language and speech / Intermediate

Log probabilities

Definition

Log probabilities are logarithms of probabilities. They express the same probability information on a different numerical scale, making products of probabilities easier to calculate as sums. In language models, they commonly represent the model's assigned probabilities for tokens or complete sequences.

What it means

Using natural logarithms, a probability p becomes ln(p). For probabilities strictly between zero and one, this value is negative. A probability of one has log probability zero, while a probability approaching zero has a log probability that decreases without bound.

A less negative value represents a higher probability. For example, ln(0.5) is about −0.693, while ln(0.1) is about −2.303. The minus sign does not make a probability invalid; it results from the logarithmic scale.

The base must be specified when comparing values. Natural logs use base e, while base-two logs support quantities measured in bits. A model-assigned token probability concerns the model's next-token distribution; it is not automatically the probability that the token forms part of a factually correct answer.

How it works

The key identity is ln(a × b) = ln(a) + ln(b) for positive a and b. A sequence model expresses a sequence probability as a product of conditional probabilities, each conditioned on the relevant preceding context. Its log probability is the sum of their logs.

This does not require treating successive tokens as independent. The conditional terms explicitly account for the context specified by the model. Replacing them with unrelated marginal probabilities would be a different calculation.

Products of many small numbers can underflow in finite-precision arithmetic, effectively becoming zero. Working with their logarithms avoids that particular multiplication problem. When adding probabilities of alternative outcomes, however, adding their log probabilities is wrong; a stable log-sum-exp calculation is commonly used instead.

Log probability and negative log probability also have opposite directions. Higher log probability favors an outcome, whereas lower negative log probability is better when it is used as a loss. Reporting the sign convention prevents misleading comparisons.

Three practical examples

A speech recognizer combines successive conditional scores in log space while comparing candidate transcriptions.
A language-model evaluation sums token log probabilities and normalizes under a stated convention to compare predictive performance.
A debugging interface displays unusually low token probabilities to help reviewers inspect where an input sequence surprised a model.

Think of it this way

Think of replacing repeated multiplication with addition on a transformed ruler. The underlying quantities retain their relationships, but the ruler makes a difficult calculation more manageable.

The analogy does not mean the transformed values are ordinary percentages. A log probability of −2 is not a negative likelihood of success, and it should not be displayed as −200%. Converting back requires the inverse transformation for the chosen logarithm base.

Worked example

Scoring a short sequence

Suppose an invented model assigns three successive conditional probabilities of 0.5, 0.2 and 0.1 to a specified three-token sequence. Their product is 0.5 × 0.2 × 0.1 = 0.01.

The natural logs are approximately −0.693147, −1.609438 and −2.302585. Their sum is −4.605170. Exponentiating that sum gives exp(−4.605170) ≈ 0.01, recovering the sequence probability apart from rounding.

The mean negative log probability is 4.605170 ÷ 3 ≈ 1.535057 nats per token. This normalization uses exactly three scored tokens; including an end marker would change the calculation unless its probability were also supplied.

Now consider two mutually exclusive alternatives with probabilities 0.2 and 0.3. Their combined probability is 0.5, so the combined log probability is ln(0.5), about −0.693147. Adding ln(0.2) and ln(0.3) would instead give ln(0.06), which describes a product and is the wrong operation for that sum.

Illustrative case study

A fictional team compares two answer-generation configurations. Its dashboard initially labels the sum of token log probabilities “answer confidence.” Longer answers tend to receive more negative totals because they contain more scored factors, and users mistake the label for a factual reliability estimate.

The team renames the field to describe exactly what is measured and documents the tokenizer, conditioning and length treatment. It evaluates factual support separately using the task's evidence requirements. When comparing predictive scores, it ensures the scored material and normalization are compatible.

The lesson is that log probabilities are useful model measurements, but their interpretation depends on the event and scoring procedure. A fluent or highly probable answer can still contain an unsupported claim.

Common misconceptions and mistakes

Negative log probabilities are invalid. Negative values are expected when the underlying probability lies between zero and one.
Adding log probabilities adds probabilities. It corresponds to multiplying the underlying positive probabilities.
Sequence scoring assumes independent tokens. Conditional sequence factors do not require that assumption.
A high token probability proves truth. It measures a model's distribution, not independent factual verification.

Business use

Businesses may encounter log probabilities in model evaluations, diagnostic tools and generation services. Ask which tokens were scored, what context was supplied, which logarithm base was used and whether totals were normalized.

Comparisons can become misleading when tokenization, length or the scored event changes. Treat the measurement as one source of evidence about predictive behavior, and pair it with task-level checks such as extraction accuracy or answer support. If a user-facing confidence estimate is needed, it requires its own validation rather than a relabeled sequence score.

Check your understanding

Two mutually exclusive outcomes have probabilities 0.2 and 0.3. Should their log probabilities be added to obtain the log probability of either outcome?

Answer: No. Add the probabilities first and take the log, or use log-sum-exp on their log probabilities.

Related terms

Probability
Logarithm
Conditional probability
Cross-entropy
Perplexity
Nats
sequence likelihood
negative log likelihood
log-sum-exp

Learn next

Calculate a short sequence probability both directly and in log space. Then compare summing alternative probabilities with multiplying conditional factors, keeping the event definition explicit in each calculation.

Probability

Sources

Jurafsky and Martin, Speech and Language Processing, Chapter 3: https://web.stanford.edu/~jurafsky/slp3/3.pdf. Log-space sequence probability calculations. SciPy, logsumexp documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.logsumexp.html. Stable logarithms of sums of exponentials. Numerical examples are original and arithmetically checked.

Previous: Log odds ratio Next: Log shipping