Is least squares always maximum likelihood?

No. Ordinary least squares coincides with coefficient MLEs under the usual independent Gaussian error model, but the equivalence is model-specific.

Does a higher training likelihood guarantee better predictions?

No; complexity and overfitting can increase training fit without improving new-case performance.

AI Glossary / Statistics and probability / Intermediate

Maximum likelihood estimate

Definition

A maximum likelihood estimate is a parameter value, or set of values, that maximizes the likelihood of the observed data within a specified statistical model. The data are held fixed while candidate parameter values are compared. For discrete outcomes the likelihood uses probabilities; for continuous outcomes it uses densities. An estimate is a fitted value, whereas an estimator is the rule that produces it from data.

What it means

Maximum likelihood asks which allowed parameter values give the greatest support to the observations under the chosen model. It does not ask which parameter values have the highest posterior probability, because ordinary maximum likelihood does not specify a probability distribution over parameters.
The model is crucial. The same observations can produce different maximum likelihood estimates under different distributional assumptions or dependence structures. Maximization cannot make an unsuitable model correct.

How it works

Specify the outcome model, parameter space, and observation mechanism. Identify whether observations are independent, censored, truncated, clustered, or otherwise linked. These features determine the likelihood that should actually be optimized.
For independent observations with density or mass function f(x;theta), the likelihood is the product of f(x[i];theta) over observations. Independence is a modeling assumption, not a mathematical convenience that can be imposed on any dataset without consequence.
Usually maximize the log-likelihood instead. Because logarithms are strictly increasing, this preserves the maximizing parameter values when the likelihood is positive. Products become sums, which are easier to differentiate and less prone to numerical underflow.
In simple models, setting derivatives to zero yields a candidate estimate. Check the parameter boundaries and whether the candidate is a maximum. Some likelihoods have several local maxima, no finite maximizer, or different parameter values that fit equally well.
Numerical optimization requires similar care. A convergence message does not establish global optimality or statistical identifiability. Examine gradients, constraints, starting-value sensitivity, and whether fitted parameters approach a boundary or an implausible region.
The usual large-sample consistency and approximate normality results require regularity conditions, including suitable identification and model assumptions. They are not universal guarantees for every likelihood, finite sample, or high-dimensional model.
A point estimate does not quantify uncertainty by itself. Depending on the setting, use likelihood-based intervals, information-matrix standard errors, bootstrap procedures, or other justified methods. Near boundaries or with weak information, familiar symmetric approximations can be misleading.
Distinguish ordinary maximum likelihood from penalized likelihood and maximum a posteriori estimation. A penalty changes the optimized objective; a prior changes a posterior calculation. These approaches can be useful, but their fitted values should not be described as ordinary unpenalized MLEs without qualification.
Finally, evaluate model fit and predictive behavior separately from optimization. A larger training likelihood can reflect additional flexibility rather than a better generalizing model. Compare models using procedures appropriate to their complexity, the data, and the intended inference.

Three practical examples

A team estimates a common event probability from independent binary outcomes using the observed success proportion.
A reliability analyst fits a lifetime distribution with a likelihood that includes censored observations correctly.
A language model is trained by maximizing conditional log-likelihood, with evaluation on separate text to assess generalization.

Think of it this way

Think of comparing several settings of a probability-generating machine after seeing its output. Maximum likelihood chooses the permitted setting that assigns the most support to that output. It does not prove that the machine description is right or that the chosen setting is known without uncertainty.

Worked example

An original example observes seven successes and three failures in ten independent Bernoulli trials with a common unknown probability p. For the particular observed sequence, the likelihood is L(p) = p^7*(1-p)^3 for 0 <= p <= 1.
For an interior p, the log-likelihood derivative is 7/p - 3/(1-p). Setting it to zero gives 7*(1-p) = 3*p, hence p-hat = 0.7. The second derivative is -7/p^2 - 3/(1-p)^2, which is negative throughout the interior. Both endpoints have zero likelihood, confirming the maximum at 0.7.
At p = 0.7, the likelihood is approximately 0.00222357. At p = 0.5, it is 0.0009765625. Their ratio is approximately 2.277. These are likelihoods for the observed sequence, not posterior probabilities of p.
If only the success count is represented, the binomial likelihood includes the factor choose(10,7) = 120. That factor multiplies every candidate likelihood equally, so it changes neither the maximizer nor the likelihood ratio.
The ten trials provide limited information. An estimate of 0.7 does not establish that the underlying probability is exactly 0.7.

Illustrative case study

In a fictional case, Pinehaven Services estimates the probability that a request needs manual review. Analysts initially apply a common-probability Bernoulli model to every logged request.
They discover many requests come from the same accounts and share operating conditions. The team revisits the dependence assumptions and compares a richer model before presenting uncertainty estimates. The original sample proportion remains a descriptive quantity, but its naive likelihood-based uncertainty may be inappropriate. This case illustrates model review, not an actual measured business outcome.

Common misconceptions and mistakes

Do not interpret maximum likelihood as the probability that an estimated parameter is true. Do not describe a continuous density at an observation as the probability of that exact observation.
An MLE need not be unbiased in finite samples, need not exist as a finite value, and need not be unique. Large-sample properties require conditions; they are not automatic consequences of using an optimizer.

Business use

Maximum likelihood provides a common foundation for fitting many statistical and machine-learning models. It allows teams to connect assumptions, parameter estimates, and uncertainty calculations within one framework. Its practical value depends on valid data handling and model checking. A precisely optimized likelihood is useful only when its relationship to the business question is defensible.

Check your understanding

Why does the binomial coefficient not change the example's estimate?

Answer: It is constant with respect to p, so it rescales the entire likelihood rather than changing where it reaches its maximum.

Related terms

Likelihood function
Log likelihood
Estimator
Maximum a posteriori estimate
Fisher information
Profile likelihood
Penalized likelihood

Learn next

Next, derive MLEs for a simple normal model and compare them with unbiased variance estimates. Then study likelihood intervals, boundary cases, and diagnostics for numerical optimization.

Sources

Frank Harrell, Glossary of Statistical Terms, maximum likelihood estimate and likelihood function: https://hbiostat.org/glossary/ NIST/SEMATECH e-Handbook, Maximum Likelihood: https://www.itl.nist.gov/div898/handbook/eda/section3/eda3652.htm NIST/SEMATECH e-Handbook, Maximum likelihood estimation: https://www.itl.nist.gov/div898/handbook/apr/section4/apr412.htm

Previous: Maximum inner-product search Next: Maximum margin separator