How many bits are in one nat?

Approximately 1.4427 bits, obtained by dividing one by ln(2).

Why are natural logarithms common in AI?

They fit naturally with exponential probability models and calculus, and many numerical libraries use them by default.

Can a lower nat value prove a model is more useful?

Not alone. The quantity and evaluation conditions must match, and predictive loss does not directly measure every practical quality such as factuality or task success.

AI Glossary / Natural language and speech / Intermediate

Nats

Definition

Nats are units of information obtained when information-theoretic quantities use natural logarithms, whose base is e. One nat equals 1/ln(2), approximately 1.4427 bits. Nats commonly appear in entropy, cross-entropy, negative log-likelihood, and Kullback - Leibler divergence calculations.

What it means

Information measures depend on the logarithm's base. Using base two reports bits; using the natural logarithm reports nats. The underlying probability model can be identical even when the displayed numbers differ.

For an event assigned probability p, its information or surprisal is −ln(p) nats. Less probable events receive larger values. A certain event has zero surprisal because −ln(1)=0. This is a mathematical description relative to a probability distribution, not a measure of how interesting or emotionally surprising an event feels to a person.

In AI reports, a loss may be stated in nats per token, nats per example, or as a total over a dataset. These denominators matter as much as the unit itself when comparing results.

How it works

For a discrete distribution with probabilities p_i, entropy is −Σ p_i ln(p_i). Cross-entropy against model probabilities q_i is −Σ p_i ln(q_i), and KL divergence is Σ p_i ln(p_i/q_i), with the usual conventions and support requirements.

To convert a value from nats to bits, divide by ln(2). To convert bits to nats, multiply by ln(2). Changing the log base rescales the quantity by a constant; it does not change the underlying predictions.

For a mean negative log-likelihood H measured in nats per token, perplexity is exp(H). If H is measured in bits per token, perplexity is 2^H instead. Apply the exponential appropriate to the unit, and ensure the averaging and tokenization are comparable.

Three practical examples

Model evaluation: interpret a language-model loss reported with natural logarithms and convert it to bits when a comparison report uses base two.
Classifier training: read average negative log-likelihood in nats per labeled example without mistaking it for an error percentage.
Distribution analysis: report a KL divergence with its log base so another team can reproduce the scale of the result.

Think of it this way

Think of nats and bits as two scales for the same information quantity, like two units used to report a length. Conversion changes the numeral, not the object measured. The analogy is limited because information depends on a specified probability model, and a per-token value also depends on what counts as a token.

Worked example

Suppose a model assigns probabilities 1/2, 1/4, and 1/8 to three observed target tokens, conditional on their respective preceding contexts. Their negative log-likelihoods are ln(2), ln(4), and ln(8), approximately 0.6931, 1.3863, and 2.0794 nats.

The total is ln(64), approximately 4.1589 nats. Dividing by three gives mean loss ln(4), approximately 1.3863 nats per token. Dividing that mean by ln(2) gives 2 bits per token. Perplexity is exp(ln(4))=4, also equal to 2^2 when using the bit-based mean.

The three probabilities need not sum to one because they come from different prediction contexts. Within each context, the full distribution over alternatives must be valid. Also, exponentiating the total instead of the mean would give 64, which is not the per-token perplexity of this example.

Illustrative case study

Fictional case: Elm Metrics compares two evaluation dashboards. One shows a loss of 1.3863 and another shows 2.0, so a reviewer initially concludes that the first model is better.

The team discovers that the reports describe the same probability assignments, with one using nats and the other bits. After aligning units and denominators, the results agree. The team then checks tokenization and test data before making any model comparison. This is an illustrative reporting mistake, not a claim about actual model performance.

Common misconceptions and mistakes

A nat is not a token, byte, or percentage point. A loss of one nat does not mean one error, and changing units does not improve a model. Negative log-likelihood is also not automatically the entropy of the true data distribution; it depends on the model's assigned probabilities and the observed sample.

Do not compare total loss with mean loss. Do not assume entropy statements for discrete variables transfer unchanged to differential entropy for continuous variables, whose value can be negative and depends on the measurement scale. The basic examples here use discrete outcomes.

Business use

In evaluation reports, include the quantity, logarithm base, denominator, dataset, and tokenization. “Cross-entropy: 1.4 nats per token” is more interpretable than an unlabeled “loss: 1.4,” though it still needs the evaluation context.

Check whether a training objective contains regularization, label smoothing, weights, or other terms before treating it as ordinary negative log-likelihood or converting it into perplexity. Unit conversion is straightforward; establishing that two reported metrics measure the same thing can require more care. Preserve enough detail for another analyst to reproduce the calculation.

Check your understanding

A mean negative log-likelihood is ln(8) nats per token. What are its bit-based value and perplexity?

Answer: It is 3 bits per token because ln(8)/ln(2)=3. Its perplexity is exp(ln(8))=8, assuming the loss is the appropriate mean negative log-likelihood.

Related terms

Entropy
Cross-entropy
Perplexity
bit
surprisal
negative log-likelihood
KL divergence
natural logarithm

Learn next

calculate surprisal for two probabilities, then compute a probability-weighted entropy and distinguish it from a simple unweighted average. Convert the result between nats and bits. Finally inspect a model's loss definition before deriving perplexity from its reported value.

Sources

Sources: SciPy, scipy.stats.entropy, formulas and the relationship between logarithm base and information units: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html . Jurafsky and Martin, Speech and Language Processing, Chapter 3, language-model probability and evaluation context: https://web.stanford.edu/~jurafsky/slp3/3.pdf . The three-token calculation and Elm Metrics scenario are original.

Previous: Native-language identification Next: Natural evolution strategy