AI Glossary / AI and machine learning / Advanced
Wake-sleep algorithm
Definition
The wake-sleep algorithm trains a probabilistic generative model together with a recognition model by alternating two phases. The wake phase improves generation using representations inferred from observed data; the sleep phase improves recognition using examples generated by the model.
What it means
A generative model describes how hidden causes could produce observations. A recognition model works in the other direction, estimating hidden causes from what is observed. For a simple image, hidden variables might represent features that help explain its pixels.
Exact inference can be expensive in a multilayer probabilistic model. Wake-sleep uses a separate recognition network to make that inference more tractable. The two networks teach each other through different sources of examples. Wake and sleep are memorable names for computational phases; they do not imply that software experiences waking, dreaming or biological sleep.
How it works
During wake, an observed data vector is passed through the recognition model. Sampled hidden states provide approximate explanations of that observation. The generative parameters are adjusted to make those sampled states and observed data more probable under the top-down model.
During sleep, the generative model samples hidden states and synthetic observations. Because the sampled hidden states are known, they become targets for training the recognition model to infer causes from generated observations.
The two directions generally have different parameters. The procedure is an approximation: recognition samples may differ from the exact posterior, while sleep examples reflect the generator's current weaknesses. Classical wake-sleep should not be described as exact maximum-likelihood optimization or confused with every modern encoder - decoder training method.
Three practical examples
- Representation-learning research: A researcher studies whether hidden variables capture reusable features of a collection of simple images without class labels.
- Generative-model education: Students compare top-down sampling with bottom-up inference, learning why generating an example and explaining an example are related but different tasks.
- Method comparison: An engineering team uses a small wake-sleep implementation to understand inference approximations before evaluating more suitable contemporary generative modeling methods for a project.
Think of it this way
Think of a model-maker and an investigator. The investigator suggests how a real object might have been built; the model-maker improves its construction process. The model-maker then creates practice objects with known construction histories for the investigator. Unlike people, the networks update numerical parameters and do not exchange conscious explanations.
Worked example
Consider a toy model with one hidden binary variable z and one visible binary variable x. During a wake step, an observed x=1 leads the recognition network to sample z=1. The generative connection is adjusted toward assigning greater probability to x=1 when z=1, using the model's actual update rule.
During a sleep step, the generator samples z=0 and then x=1. The recognition network receives x=1 with the known sampled target z=0 and updates accordingly.
The apparent contradiction is legitimate: a probabilistic generator can produce the same observation from different hidden causes. One sample does not establish a deterministic rule. The example illustrates information flow and training targets, rather than computing a full likelihood gradient or proving convergence.
Illustrative case study
Fictional case: Pattern Studio teaches junior researchers about probabilistic image models. Its demonstration contains an upward recognition network and a downward generator trained on tiny binary patterns.
Learners initially assume that a convincing generated pattern means the hidden-state inference is correct. Separate checks reveal that the recognition network confuses two hidden explanations. The team plots generation quality and inference behavior independently and compares several random seeds. This invented classroom project demonstrates evaluation questions, not a production success claim.
Common misconceptions and mistakes
Wake-sleep is not ordinary supervised training on human-provided labels. Its sleep targets come from the model's own generated hidden states. Nor is it the same as simply reconstructing each input and backpropagating one reconstruction loss through a deterministic autoencoder.
A generator that misses important patterns may provide poor practice examples during sleep. Recognition errors can also affect wake updates. Do not claim that alternating phases must improve every metric monotonically. Later variations and related methods use different objectives, estimators and update schedules; their properties should not be attributed automatically to the original algorithm.
Business use
For most businesses, wake-sleep is more useful as background for understanding generative modeling than as an automatic implementation choice. It highlights two distinct requirements: producing plausible data and inferring useful hidden representations.
If a specialist proposes it for an application, ask which formulation is being used, why its approximations suit the data and how it compares with simpler baselines. Evaluate held-out data, stability and the intended downstream task. Plausible samples alone do not establish that a model captures the distribution or supports reliable decisions.
People also ask
Why use two networks?
One defines a generative direction, while the other provides a tractable approximation for inferring hidden states from observations.
Does the sleep phase use real training examples?
In the classical method it uses samples from the generative model to train recognition.
Is wake-sleep an exact likelihood optimizer?
Generally no. The approximation to inference and the different phase objectives matter.
Check your understanding
During sleep, where do the target hidden states for recognition training come from?
Answer: They are the hidden states sampled by the generative model when it produced the synthetic observations.
Related terms
- Helmholtz machine
- Generative model
- Latent variable
- Approximate inference
- Autoencoder
- recognition network
- variational inference
- maximum likelihood
Learn next
Draw two arrows between an observed variable and a hidden variable. Label the generator and recognition model separately. Trace one wake sample and one sleep sample, recording which parameters change. Then study why exact posterior inference may be difficult and compare wake-sleep with variational autoencoder objectives.
More in AI and machine learning
- Virtual support vector machine - A virtual support vector machine is an SVM trained using artificial examples that represent transformations expected to preserve a class label.
- Voting Classifier - A voting classifier combines predictions from several classification models into one class decision.
- Voting Regressor - A voting regressor combines numerical predictions from several regression models, usually by averaging them.
- Wasserstein GAN - A Wasserstein GAN, or WGAN, is a generative adversarial network trained using an objective based on the Wasserstein-1 distance between distributions.
- Weak artificial intelligence - Weak artificial intelligence has two related but distinct uses.
- Weak supervision - Weak supervision trains machine-learning models using supervision that is less complete, precise or reliable than carefully verified labels for every example.
Sources
Geoffrey Hinton, Peter Dayan, Brendan Frey and Radford Neal, The wake-sleep algorithm for unsupervised neural networks (1995): https://www.cs.utoronto.ca/~hinton/absps/ws.htm ; author paper: https://www.cs.utoronto.ca/~hinton/absps/ws.pdf . Independent review consulted the phase descriptions: recognition-driven wake examples train generation, while generator-driven sleep examples train recognition. The binary-variable example and fictional classroom case are original; exact likelihood optimization or convergence is not claimed.
Previous: Wait list control group Next: Wake word