Must the outcome function be differentiable?

No; the method needs its values and a suitable differentiable probability model.

Is this identical to REINFORCE?

REINFORCE uses this principle in reinforcement-learning settings; the estimator has broader applications.

Does any baseline help?

No. The expected gradient can be preserved while variance increases if the baseline is poorly chosen.

AI Glossary / Probabilistic machine learning / Advanced

Score function estimator

Definition

A score function estimator estimates the derivative of an expected outcome by multiplying sampled outcomes by the parameter gradient of their log probability. Also called a likelihood-ratio gradient estimator, it can work with discrete choices and nondifferentiable outcome functions, provided the distribution and required interchange of differentiation and expectation satisfy appropriate conditions.

What it means

Suppose a simulator produces an outcome after drawing a random configuration. You want to adjust the distribution of configurations to improve the average outcome, but cannot differentiate through the simulator itself.
The estimator uses a different piece of information: how changing a distribution parameter changes the probability of the sampled configuration. The outcome weights that probability sensitivity. Repeating the process estimates a direction for changing the distribution. Here, score means a derivative of log probability with respect to model parameters, not a rating, accuracy measure, or the derivative with respect to the sampled input.

How it works

For J(theta) = E[f(X)] with X drawn from p_theta and f independent of theta, the identity is grad J = E[f(X) grad log p_theta(X)]. An empirical average of these products supplies the estimate. Its derivation writes grad p as p times grad log p.
The relevant differentiation and integration or summation must be interchangeable. Parameter-dependent support can require boundary terms, so the formula cannot be applied mechanically to every distribution.
A baseline independent of the sampled variable can replace f by f - b because the score has zero expectation under the regularity conditions. Suitable baselines can reduce variance. If f also depends directly on theta, its expected direct derivative must be included.
In practice, implementations differentiate the sampled log probability while treating the sampled outcome and baseline as fixed weights for this term. Any separate direct derivative follows its own computation path.

Three practical examples

Discrete design search: a distribution selects component options, while an engineering simulator supplies a performance value without a usable derivative.
Reinforcement learning: sampled actions receive return-based weights when estimating policy gradients, as in REINFORCE.
Variational inference: samples from an approximate distribution support gradient estimates for an expectation involving a probabilistic model.

Think of it this way

Think of adjusting a lottery over design options. An option's result tells you how useful it was, while the score tells you how a small parameter change would alter its chance of being drawn. Combining those signals helps adjust future drawings. A single lucky result can still mislead, which is why repeated sampling and variance reduction matter.

Worked example

Consider an original two-option example. Let X be Bernoulli with probability p, and define f(1) = 7 and f(0) = 2. Then J(p) = 7p + 2(1 - p) = 2 + 5p, so the exact derivative with respect to p is 5.
At p = 0.4, the score is 1/p = 2.5 when X = 1 and -1/(1 - p) = -5/3 when X = 0. The unadjusted sample estimator therefore takes values 17.5 and -10/3. Its expectation is 0.4 times 17.5 plus 0.6 times -10/3, which equals 5.
Its variance is approximately 104.167: unbiased does not mean that each draw is close to the answer. With baseline b = 2, the two estimator values become 12.5 and 0. Their mean remains 5, while their variance falls to 37.5.
In this special example, b = 5 makes both possible estimator values exactly 5, giving zero variance. This unusually convenient baseline uses knowledge of both outcomes; general simulation problems do not offer such a shortcut.
All these derivatives are with respect to p. If a logit theta parameterizes p, multiply by p(1 - p) = 0.24. The exact theta derivative at this point is therefore 1.2. Confusing the two parameters would produce an incorrect update scale.

Illustrative case study

Fictional case: Oak Layout uses a simulator to evaluate discrete shelving arrangements. Its engineers parameterize the sampling probabilities and record both each chosen arrangement's log probability and its simulated packing score.
They first build a tiny version whose expected outcome can be enumerated exactly. This catches a sign error and confirms that their gradient estimate targets the intended parameters. They then compare repeated estimates with and without an independently fitted baseline.
Before accepting a configuration, the team evaluates it separately under realistic constraints. The example illustrates a testing workflow; it does not claim that a score function estimator necessarily finds the best layout or reduces business costs.

Common misconceptions and mistakes

Do not confuse this estimator with score matching, which commonly learns gradients of log density with respect to inputs. Do not omit a direct parameter derivative when the outcome function contains the same parameters.
Unbiasedness does not guarantee low variance or improvement after every update. A baseline chosen carelessly from the same sampled outcome can introduce bias. A support boundary that moves with the parameter can invalidate the simple identity. Finally, differentiating only through a discrete sampled value usually misses the distribution's contribution.

Business use

The method makes some otherwise difficult simulation-based optimization problems accessible. Its value depends on how many costly outcome evaluations are needed for a reliable direction. Teams should measure estimator variance, compare simpler alternatives when available, and retain an independent evaluation stage. Small exact-enumeration problems are particularly useful for checking an implementation before allocating a large simulation budget.

Check your understanding

In the worked example, does subtracting baseline 2 change the expected derivative from 5?

Answer: No. It changes the sample values and reduces their variance, while the expected derivative remains 5. The sample-independent baseline multiplies a score whose expectation is zero.

Related terms

REINFORCE
Policy gradient
Control variates
Monte Carlo gradient estimation
Control variate
Pathwise gradient estimator
Variational inference
Fisher information

Learn next

Derive the estimator for a small categorical distribution and verify it by enumeration. Then compare baseline choices and study pathwise gradients to understand when differentiating through a reparameterized sample is possible.

Sources

Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih, Monte Carlo Gradient Estimation in Machine Learning (2020), Section 4: https://www.jmlr.org/papers/volume21/19-346/19-346.pdf Ronald J. Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (1992): https://gwern.net/doc/reinforcement-learning/model-free/1992-williams.pdf Mohamed and colleagues Section 4 independently consulted for the score identity, baseline and regularity conditions.

Previous: Score-based generative model Next: Score matching