AI Glossary / Probabilistic machine learning / Advanced

Stochastic variational inference

Definition

Stochastic variational inference, or SVI, uses stochastic optimization to fit a tractable approximation to a Bayesian posterior distribution. Instead of evaluating every observation for every global update, it uses appropriately scaled information from randomly selected data subsets. The resulting updates optimize a variational objective while reducing the data processed at each step.

What it means

Variational inference replaces a difficult posterior calculation with an optimization problem over a chosen family of distributions. The usual objective is the evidence lower bound, or ELBO. Improving that objective makes the approximation closer in the associated KL sense, within the available family.

SVI changes how the optimization is carried out; it does not eliminate approximation error from the family itself. The original influential formulation separates global latent variables, shared across the dataset, from local latent variables associated with individual observations. Natural-gradient updates give a particularly convenient construction for conditionally conjugate models. Broader modern usage also includes stochastic gradient methods for nonconjugate variational objectives.

How it works

Select a random minibatch. Where local latent variables are present, infer their variational distributions using the current global approximation. Use their expected sufficient statistics to estimate the full dataset's contribution, scaling according to the sampling scheme.

In the conjugate natural-gradient setting, combine this estimated contribution with the prior to form a proposed global variational parameter. Blend that proposal with the current parameter using a learning rate. The prior is included once in the full-data target; it is not blindly multiplied by the minibatch scaling factor.

Repeat with new minibatches. Suitable decreasing learning rates and regularity assumptions support stochastic approximation convergence, but do not guarantee a global optimum in a nonconvex problem. Inexact local inference can affect the gradient estimate. A method using a constant learning rate should not inherit a convergence claim that depends on a diminishing schedule.

Three practical examples

1. Topic analysis: update global topic-word distributions using a subset of documents, while estimating local topic mixtures for those documents. Check held-out predictive fit and topic stability rather than relying only on training objective values.

2. Latent-factor modeling: estimate shared factor structure from sampled records while accounting for record-specific latent variables. Design minibatches so that frequent entities do not silently receive unintended extra weight.

3. Large probabilistic models: optimize a variational posterior when full-data updates are too costly. Compare smaller reference problems to identify errors in likelihood scaling, prior treatment, and uncertainty representation.

Think of it this way

Think of revising a large report by examining randomly chosen sections and estimating what those sections imply for the whole. Each revision is deliberately partial because any one sample may be unrepresentative. The sampling scheme and adjustment size determine whether repeated revisions move toward the intended full-document objective.

Worked example

Consider an original conjugate toy that isolates the global update. There are 100 Bernoulli observations and a Beta(2, 2) prior for their common success probability. Use a Beta variational family; this simple example needs no local latent variables.

Suppose the current approximation is Beta(52, 52). A uniformly selected minibatch contains 10 observations, of which 7 are successes and 3 failures. Multiplying the batch counts by 100 divided by 10 gives estimated full-data counts of 70 and 30. Adding the prior produces the proposed parameters 72 and 32.

With learning rate 0.25, the new parameters are 0.75 times 52 plus 0.25 times 72 = 57, and 0.75 times 52 plus 0.25 times 32 = 47. The updated approximation is Beta(57, 47), whose mean is 57 divided by 104, approximately 0.5481. The previous mean was 0.5.

This is one noisy optimization update, not evidence that 70 successes occurred in the dataset. If the actual full dataset contains 60 successes and 40 failures, its exact posterior is Beta(62, 42), with mean approximately 0.5962. Additional properly constructed updates target that full-data solution. The example illustrates scaling and damping; it does not claim that every variational family can represent an exact posterior.

Illustrative case study

Fictional case: Alder Research organizes a large archive with a probabilistic topic model. The team adopts SVI to avoid processing the entire archive before each global update. Analysts verify that document sampling and count scaling correspond to the intended archive-level objective.

They keep a separate evaluation collection and compare different learning schedules using the same preprocessing. They also inspect whether rare document types are adequately represented. If a topic becomes unstable, they investigate sampling, optimization, and model assumptions separately. This fictional case states an evaluation workflow without inventing a throughput gain or claiming that ELBO improvement proves editorial usefulness.

Common misconceptions and mistakes

SVI is not posterior sampling merely because its optimization is stochastic. Saved optimizer states do not automatically form posterior draws. Minibatch likelihood terms need the correct scaling, and prior terms need separate treatment. More iterations can reduce optimization error but cannot remove restrictions imposed by the chosen variational family. Apparent convergence of a noisy objective should be checked with suitable diagnostics.

Business use

The business benefit is making probabilistic inference feasible under data-processing limits. Its value depends on whether approximate uncertainty remains useful for the intended decisions. Teams should document the variational family, local inference procedure, sampling scheme, scaling, learning schedule, and evaluation method. For continuously changing data, distinguish optimization for a fixed dataset from a deliberate streaming model with changing statistical assumptions.

People also ask

Does SVI always use natural gradients?

The original conjugate formulation does; broader stochastic variational approaches can use other gradients.

Does an unbiased minibatch gradient imply an unbiased posterior approximation?

No. Optimization behavior and variational-family restrictions are separate issues.

Should the prior be scaled like the likelihood minibatch?

Not automatically; it belongs once in the full-data objective.

Check your understanding

Answer: 1.

Why multiply the toy batch counts by 10?

Answer: The uniformly sampled batch has one tenth of the dataset's observations.

Answer: 2.

Why blend rather than replace the current parameters immediately?

Answer: A minibatch provides a noisy estimate, and the learning rate controls its influence.

Answer: 3.

What remains even after perfect optimization in a restricted family?

Answer: Variational approximation error can remain.

Related terms

• Variational inference
• evidence lower bound
• natural gradient
• minibatch
• local latent variable
• global latent variable
• stochastic approximation

Learn next

Derive a conjugate posterior update first. Then replace its sufficient statistics with scaled minibatch estimates and study how learning-rate schedules and local inference affect convergence.

• Learning rate

More in Probabilistic machine learning

• Stein discrepancy - A Stein discrepancy measures how strongly a candidate distribution violates identities that should hold under a target probability distribution.
• Stochastic gradient Hamiltonian Monte Carlo - Stochastic gradient Hamiltonian Monte Carlo, or SGHMC, is a posterior sampling method that combines momentum-based dynamics with gradients estimated from...
• Stochastic neighbor embedding - Stochastic neighbor embedding, or SNE, is a nonlinear dimensionality-reduction method that represents objects in a lower-dimensional space by trying to...
• Stochastic weight averaging - Stochastic weight averaging, or SWA, forms a single neural-network model by taking an equal average of selected parameter vectors along a training trajectory.
• Stochastic weight averaging Gaussian approximation - Stochastic weight averaging Gaussian approximation, usually called SWA-Gaussian or SWAG, fits a Gaussian distribution over neural-network parameters using...
• Straight-through estimator - A straight-through estimator is a training rule that substitutes a usable derivative for the derivative of a hard or discrete operation during backpropagation.

Sources

Hoffman, Blei, Wang and Paisley, Stochastic Variational Inference, develops the global/local formulation and stochastic natural-gradient updates: https://jmlr.org/papers/volume14/hoffman13a/hoffman13a.pdf

The Beta-Bernoulli update and Alder Research scenario are original teaching examples; the scenario is fictional.

Independent review consulted Sections 2.4–2.5, including the natural-gradient update, step-size conditions and minibatch extension.

Previous: Stochastic variance reduction Next: Stochastic weight averaging