Does EM always find the best possible parameters?

No. It can settle at a local solution or encounter other model-specific difficulties.

Does the E-step fill every missing value with one number?

Not generally. It computes conditional expectations needed for the complete-data log likelihood.

Is EM limited to clustering?

No. Its framework applies to many latent-variable and incomplete-data estimation problems.

AI Glossary / Graphs and symbolic AI / Intermediate

Expectation-maximization

Definition

Expectation-maximization, usually abbreviated EM, is an iterative method for estimating model parameters when some relevant information is unobserved. It alternates between estimating how hidden information is distributed under the current model and updating parameters using those expectations. Common applications include mixture models, hidden-state models, and suitable missing-data problems.

What it means

Imagine observations generated by two groups, but the group labels are missing. Estimating each group's parameters would be easier if the labels were known. Assigning the labels would be easier if the parameters were known.

EM addresses this interdependence through repeated conditional calculations. It starts with provisional parameters, computes probabilistic information about the hidden variables, and uses that information to obtain revised parameters.

The hidden information need not be a missing spreadsheet cell. It can be an unobserved component label or another latent quantity introduced by the model. EM estimates parameters of a chosen probabilistic model; it does not automatically discover the correct number of groups or prove that the assumed data-generating process is accurate.

How it works

The expectation step, or E-step, forms the expected complete-data log likelihood conditional on the observed data and current parameters. In a mixture model, this often involves responsibilities: probabilities that each observation came from each component.

The maximization step, or M-step, chooses parameters that maximize that expected log likelihood. Depending on the model, this can resemble calculating weighted averages or expected counts.

Repeat the two steps until a stopping rule is met. For standard exact EM under its assumptions, the observed-data likelihood does not decrease after a full iteration. This is not a guarantee of reaching the global optimum; initialization and the likelihood's shape can matter.

Practical implementations should monitor the relevant objective, numerical problems, and iteration limits. If an E-step or M-step is approximated, the behavior needs to be assessed for that particular approximation rather than assumed to match exact EM.

Three practical examples

1. A mixture model uses EM to estimate component parameters when observations arrive without component labels.

2. A hidden-state sequence model uses an EM-style procedure to estimate parameters from observed sequences while averaging over possible hidden state paths.

3. An analyst estimates a multivariate model with missing measurements when the model and missingness assumptions justify that treatment.

Think of it this way

Think of sorting mixed deliveries into likely sources while also learning each source's typical contents. Better source descriptions improve the sorting, and the updated sorting improves the descriptions.

The analogy is limited because ordinary sorting often makes hard assignments. EM usually retains uncertainty through conditional expectations rather than deciding every hidden label with certainty. It can also reinforce a poor initial explanation if the model or initialization is unsuitable.

Worked example

Suppose each packet contains two tosses of one of two coins, A or B. The coin is chosen once per packet, and tosses are independent given that coin. Observed packets are HH, HH, and TT; their coin labels are hidden.

Initialize equal mixing probabilities, with head probabilities 0.8 for A and 0.2 for B. For an HH packet, A's responsibility is 0.64 / (0.64 + 0.04) = 16/17. For TT, it is 0.04 / (0.04 + 0.64) = 1/17.

The expected number of A packets is therefore 16/17 + 16/17 + 1/17 = 33/17. The M-step updates A's mixing proportion to (33/17) / 3 = 11/17. B's proportion becomes 6/17.

A receives an expected 64/17 heads across 66/17 tosses, giving a new head probability of 32/33. B receives 4/17 heads across 36/17 tosses, giving 1/9.

These fractional counts are expectations, not claims that a physical packet was divided. A second iteration would recompute responsibilities using the new parameters. This small calculation illustrates one iteration only; three packets are not a basis for strong conclusions about real coins or mixture identifiability.

Illustrative case study

Fictional case: Lumen Services fits a mixture model to task durations, hoping to distinguish routine and complex work. The team runs EM from several initializations and finds that different fits can describe the observations differently.

It compares model fit and held-out behavior, then checks whether the inferred groups correspond to operationally meaningful categories. It does not name a component complex merely because its estimated average is larger. The case illustrates that fitting a latent model and interpreting its components are separate responsibilities.

Common misconceptions and mistakes

Do not replace every hidden variable with its single most likely value and assume the procedure is ordinary EM. Hard assignment is a different update in general.

Do not interpret nondecreasing likelihood as guaranteed global optimality or improved prediction on new data.

Avoid treating EM as a universal cure for missing data. The observation and missingness assumptions still matter.

Do not confuse the expected complete-data log likelihood used in the E-step with the complete-data likelihood evaluated at one guessed completion. The expectation applies to the relevant log-likelihood expression, which can require more than substituting means into missing fields.

Business use

EM can make otherwise difficult parameter-estimation tasks manageable when the hidden-variable structure has convenient conditional calculations. Business value depends on whether the fitted model answers a useful question and remains reliable outside the fitting sample.

Record initialization, convergence criteria, model choices, and any regularization. Inspect weakly supported components and unstable estimates. For decisions involving latent groups, present uncertainty and avoid treating component membership as an established factual label.

Check your understanding

Why does the worked example use 16/17 of an HH packet for coin A rather than assigning the packet completely to A?

Answer: Because the current model leaves a 1/17 probability that B generated it. EM carries that uncertainty into expected counts.

Related terms

Latent variable
Hidden Markov model
Missing data
maximum likelihood
mixture model
responsibility
initialization
local optimum

Learn next

Review conditional probability and log likelihood, then perform a second iteration of the coin example. Track the mixing probabilities and expected counts separately. Follow with Gaussian mixtures to see how the same idea produces weighted means and covariance estimates.

Sources

Primary sources: Dempster, Laird, and Rubin, Maximum Likelihood from Incomplete Data via the EM Algorithm: https://www.maths.usyd.edu.au/u/jchan/GLM/Dempster1977EM.pdf

SAS documentation of the E-step and M-step: https://support.sas.com/documentation/cdl/en/statug/68162/HTML/default/statug_mi_details02.htm

The coin arithmetic and fictional case are original teaching examples.

Previous: Expectation conditional maximization Next: Expectation propagation