Are score-based models related to diffusion models?

Yes; many diffusion models admit a score-based interpretation, with relationships depending on the chosen process and parameterization.

Must every sampler add random noise at every step?

No. Probability-flow ODE sampling is deterministic conditional on its initial state.

Does the model learn the original data score at every noise level?

No. Each level has its own perturbed marginal distribution.

AI Glossary / Probabilistic machine learning / Advanced

Score-based generative model

Definition

A score-based generative model learns gradients of log probability density with respect to the data and uses those gradients in a sampling process to create new examples. Modern forms usually learn scores across multiple noise levels, allowing sampling to progress from a simple noisy distribution toward the structure represented by the training data.

What it means

The model learns a vector field: at a particular noisy input, it estimates how log density changes in different directions. This field guides generation without requiring the network to directly output a normalized density.
Here, score is a mathematical derivative, not a judgment of image quality or factual accuracy. Training the score network and running the sampler are distinct steps. Good generation depends on both, as well as the training distribution, noise process, conditioning information, and computational budget.

How it works

Training commonly begins by corrupting data with known noise at different strengths. A network receives the noisy sample and its noise level, then learns through a score-matching objective. With additive Gaussian noise of standard deviation sigma, the conditional target is -(noisy input - clean input)/sigma squared. Averaging these conditional targets appropriately yields the score of the noisy marginal distribution.
Early methods used annealed Langevin dynamics, alternating score-guided movement and random perturbations while reducing the noise level. A continuous-time formulation learns the scores required to reverse a forward stochastic differential equation. A related probability-flow ordinary differential equation provides a deterministic sampling route given its initial random draw.
These formulations connect closely to diffusion models. They do not imply that every implementation has identical objectives or samplers, or that finite numerical steps with an imperfect network produce exact samples.

Three practical examples

Image generation: a learned score field helps transform noisy initial states into new visual samples.
Signal reconstruction: a conditional model combines learned structure with observed signal information, while the reconstruction procedure must respect the observation model.
Simulation data: a team explores synthetic continuous measurements for testing a downstream system, with separate checks for coverage and unrealistic outputs.

Think of it this way

Think of moving through a foggy landscape using a map of local slopes. At first, a heavily smoothed map shows broad structure; increasingly detailed maps reveal finer features. Randomness and a carefully designed movement rule let the process explore rather than simply climb to one peak. This is an analogy for sampling, not a claim that every generated item follows a meaningful path through real objects.

Worked example

Consider an original one-dimensional teaching model. Clean data X follows a normal distribution with mean 2 and variance 4. Add independent Gaussian noise with variance 5 to form Y. The noisy marginal is normal with mean 2 and variance 9, so its score is s(y) = -(y - 2)/9.
At y = 5, that score is -1/3. By contrast, the clean distribution's score at the same number is -3/4. A network conditioned on the noise level must learn the appropriate score, not reuse one field for every level.
For one training pair with clean x = 4 and noisy y = 5, the conditional Gaussian target is -(5 - 4)/5 = -0.2. This differs from the marginal score -1/3. There is no contradiction: training averages over possible clean values associated with the noisy observation.
Indeed, E[X given Y = 5] = 2 + (4/9)(5 - 2) = 10/3. Averaging the conditional target gives -(5 - 10/3)/5 = -1/3, matching the marginal score.
Now illustrate one Langevin step at this fixed noise level using y_new = y + epsilon s(y) + sqrt(2 epsilon) z. Let epsilon = 0.18 and the illustrative standard-normal draw be z = 0.5. Then y_new = 5 - 0.06 + 0.30 = 5.24.
The step moved away from the mean because random exploration exceeded the deterministic pull. One step is not an exact sample or a complete generation process; it simply makes the two contributions visible.

Illustrative case study

Fictional case: Hazel Signals explores a score-based model for synthetic acoustic test signals. The researchers condition the network on the perturbation level and check a small Gaussian example before training on complex recordings.
They compare samplers at several computational budgets and inspect whether generated signals cover rare but important patterns. They also test downstream performance on separately held-out real recordings rather than treating attractive generated examples as sufficient evidence.
The team labels synthetic outputs clearly and investigates suspicious similarity to training material. This invented case describes an evaluation process; it reports no measured accuracy, privacy guarantee, or commercial deployment.

Common misconceptions and mistakes

A score vector is not a probability, and following it deterministically uphill is not generally a sampler for the target distribution. Conditional denoising targets from individual training pairs need not equal the marginal score, as the worked example demonstrates.
A noise-conditioned network cannot ignore which distribution its score represents. More sampling steps do not repair every training error, and outputs that appear plausible may omit important modes. Learning a generative distribution also does not ensure that outputs satisfy external factual, physical, or business constraints.

Business use

These models can support creative production, reconstruction, and simulation when output quality justifies their training and sampling costs. Teams should evaluate latency, diversity, conditioning accuracy, and task-specific failure cases. For synthetic-data projects, usefulness must be tested against real evaluation data, and any privacy claim requires its own evidence. A successful demonstration should not be confused with production readiness.

Check your understanding

Why is the training-pair target -0.2 different from the marginal score -1/3 in the example?

Answer: The pair fixes one clean value, while the marginal score averages the conditional target over the possible clean values given the noisy observation.

Related terms

Score matching
Denoising score matching
Diffusion model
Langevin dynamics
Stochastic differential equation
Probability-flow ODE
Noise-conditioned score network

Learn next

Study Gaussian denoising and the distinction between conditional and marginal distributions. Then compare annealed Langevin sampling, reverse stochastic differential equations, and probability-flow ODEs, paying attention to their approximation errors.

Sources

Yang Song and Stefano Ermon, Generative Modeling by Estimating Gradients of the Data Distribution (2019), Sections 2–4: https://papers.neurips.cc/paper/9361-generative-modeling-by-estimating-gradients-of-the-data-distribution.pdf Yang Song, author tutorial on score-based models, reverse SDEs, and probability-flow ODEs: https://yang-song.net/blog/2021/score/ The author tutorial independently consulted for noise-conditioned scores, annealed sampling, reverse SDEs and probability-flow ODEs.

Previous: Scientific protocol Next: Score function estimator