Why take a logarithm?

It preserves likelihood rankings and turns multiplication into addition.

Must log likelihood be negative?

No. Continuous densities can exceed one, and log likelihood depends on the measurement scale.

Is maximizing it always easy?

No. Some models have multiple optima, problematic boundaries, or no finite maximum.

AI Glossary / Statistics and probability / Intermediate

Log likelihood

Definition

Log likelihood is the logarithm of a likelihood function, usually using the natural logarithm. It measures how well different parameter values account for the observed data within a specified probabilistic model. Taking the logarithm turns products into sums and preserves the ordering of positive likelihood values. Consequently, maximizing log likelihood gives the same parameter estimates as maximizing likelihood, while often making calculation and numerical optimization easier.

What it means

The data are fixed when interpreting a likelihood function; the candidate parameters vary. Log likelihood is therefore not the probability that a parameter value is true. Such a probability requires a Bayesian model with a prior and posterior calculation.

For discrete data, likelihood uses probability masses. For continuous data, it uses probability densities, which are not probabilities of exact observations. This distinction explains why a continuous-data log likelihood can be positive. Its absolute value depends on the measurement convention and on which constants are retained, so comparisons need a consistent basis.

How it works

Write the probability model for the actual observation process, including censoring, missingness, or dependence when relevant. For independent observations with probability or density contributions f(xi; theta), the log likelihood is the sum of log f(xi; theta). Conditional independence can justify a similar factorization after conditioning on specified variables.

Calculate the expression directly in logarithmic form when possible. Multiplying thousands of tiny probabilities first can cause numerical underflow before the logarithm is taken. Optimize over valid parameter values and check convergence, boundaries, and sensitivity to starting values.

Many optimization tools minimize a function, so they use negative log likelihood. A penalty added for regularization produces a different objective; report it separately from the unpenalized likelihood when comparing fitted models.

Three practical examples

A binary classifier is trained by improving the probabilities assigned to the observed labels, using a sum of log-probability contributions.
A demand analyst estimates a count model's parameters by maximizing the log likelihood of observed daily counts under an explicit sampling model.
A reliability team fits a lifetime distribution using different likelihood contributions for observed failures and units still functioning at the observation cutoff.

Think of it this way

Think of a model as making a probability-based prediction for each observation. Log likelihood adds the scores of those predictions into one total. A very confident wrong prediction can contribute a large negative score. The total rewards probabilistic fit, but it cannot tell you whether the underlying model describes the right problem.

Worked example

Consider an original example with five independent binary outcomes: 1, 1, 0, 1, 0. Assume each outcome has the same success probability p. For this particular ordered sequence, L(p) = p cubed times (1 - p) squared, and log L(p) = 3 log p + 2 log(1 - p).

At p = 0.5, the likelihood is 0.03125 and the log likelihood is approximately -3.465736. At p = 0.6, the likelihood is 0.03456 and the log likelihood is approximately -3.365058. The latter is higher, so it fits this sample better within the model. Differentiating gives 3/p - 2/(1 - p) = 0, which yields the maximum at p = 0.6.

The improvement is about 0.100678 log units. Exponentiating it gives a likelihood ratio of 1.10592. This is not a 110.592% probability that the second value is correct. If only the total count of three successes were recorded, the binomial likelihood would include a factor of 10; it would change both log likelihoods by log 10 without changing their maximizing p.

Illustrative case study

Fictional case: Harbor Recommendations compares two models that assign purchase probabilities to the same held-out customer events. The team initially looks only at how many probabilities exceed a classification threshold. It then evaluates log likelihood to distinguish cautious predictions from confident mistakes.

Analysts confirm that both models score exactly the same eligible events with the same outcome definition. They report average negative log likelihood together with calibration and business costs. A better score supports a probabilistic performance claim on that evaluation set, while deployment still depends on whether recommendations improve outcomes in a suitable experiment.

Common misconceptions and mistakes

A higher training log likelihood does not guarantee better future predictions; a flexible model can overfit. Raw totals from datasets of different sizes are generally not directly comparable. Even averages require comparable populations and scoring conventions.

Do not confuse log likelihood with log probability of the parameters, a p-value, or an accuracy percentage. Negative log likelihood reverses the preferred direction: smaller is better. Omitting constants is acceptable for some parameter optimization tasks but can invalidate absolute reporting or comparisons that require a complete, consistent likelihood.

Business use

Log likelihood supports model fitting, probabilistic evaluation, likelihood-ratio calculations, and information criteria. For business users, its main benefit is sensitivity to the quality of probability estimates rather than only yes-or-no predictions.

Ask what observations contribute to the total, which independence assumptions are made, and whether the score was measured on training or held-out data. Monitor subgroup performance as well as the overall average. A few highly confident failures may deserve investigation even when the aggregate score improves, especially when their operational costs differ substantially from ordinary errors.

Check your understanding

1. Which candidate fits the worked sample better, p = 0.5 or p = 0.6?

Answer: 0.6, because its log likelihood is higher. 2.

Does adding a parameter-independent constant change the optimizer?

Answer: No, though it changes the reported value. 3.

Is lower negative log likelihood preferable?

Answer: Yes, under a consistent comparison.

Related terms

Likelihood
Cross-entropy
Overfitting
Calibration
maximum likelihood estimation
negative log likelihood
probability density
likelihood ratio

Learn next

Work through Bernoulli and normal likelihoods, then examine how dependence and censoring change their factorization. Compare training objectives with held-out probabilistic scores and learn when likelihood-based model comparisons require additional assumptions.

Likelihood

Sources

NIST/SEMATECH, “Maximum Likelihood.” https://www.itl.nist.gov/div898/handbook/eda/section3/eda3652.htm — primary technical guidance on estimation and practical limitations. Marco Taboga, “Maximum likelihood estimation,” StatLect. https://www.statlect.com/fundamentals-of-statistics/maximum-likelihood — author's mathematical treatment of likelihood, logarithmic optimization, and regularity assumptions. The worked example and fictional case are original.

Previous: Log-Gaussian Cox process Next: Log likelihood ratio