AI Glossary / Probabilistic machine learning / Intermediate
Differentiable cross-entropy method
Definition
The differentiable cross-entropy method, or DCEM, is a differentiable variant of a sampling-based optimization method. It replaces hard selection of the best sampled candidates with a smooth selection mechanism, allowing gradients to pass through the optimization procedure to parameters of the objective being optimized.
What it means
The ordinary cross-entropy method repeatedly samples possible solutions, selects an elite group, and refits a sampling distribution toward that group. It can search a complicated objective without differentiating that objective with respect to the candidate solution.
However, when the objective itself is learned, a larger training system may need to know how its final selected solution changes when objective parameters change. Hard elite selection makes that connection difficult. DCEM softens selection and differentiates the resulting procedure. Its name refers to the optimization method's distribution-fitting principle, not merely to the cross-entropy loss used in classification.
How it works
Sample candidate solutions from a parameterized distribution and evaluate their objective values. For minimization, better candidates have smaller values. Replace the hard top-k membership indicators with smooth weights between zero and one whose sum is k.
The original DCEM uses a temperature-controlled limited multi-label projection for these weights. Refit the sampling distribution using weighted maximum likelihood. With a suitable Gaussian family, weighted means and variances have simple formulas. Reparameterized sampling allows gradients to pass through random draws when the base random numbers are treated as fixed for differentiation.
Repeat the procedure for a chosen number of iterations and differentiate its output with respect to objective parameters. This differentiates the implemented approximation; it does not automatically give the exact derivative of a global optimizer.
Three practical examples
- Learned planning objectives: train parameters of a cost function based on the decisions produced by a sampling-based planner, provided the necessary differentiation path is available.
- Energy-based prediction: incorporate an approximate continuous optimization step inside training instead of treating the prediction solver as an entirely disconnected component.
- Algorithm experiments: compare hard elite selection with several smoothing temperatures to inspect the trade-off between sharp selection and useful gradients.
Think of it this way
Think of an audition where the ordinary method admits only a fixed number of finalists. A tiny score change may suddenly replace one finalist with another. The differentiable version gives graded participation that changes more smoothly. The analogy stops at the mechanics: these are mathematical weights used to refit a search distribution, not probabilities that a candidate is objectively correct.
Worked example
Consider an original one-dimensional refitting example with candidate values 0, 1, 2, and 3. Suppose a soft elite-selection step with k = 2 produces weights 0.8, 0.7, 0.4, and 0.1. Each lies between zero and one, and their sum is 2.
The weighted Gaussian mean is (0.8 × 0 + 0.7 × 1 + 0.4 × 2 + 0.1 × 3) / 2 = 0.9.
The maximum-likelihood variance is [0.8(0 − 0.9)² + 0.7(1 − 0.9)² + 0.4(2 − 0.9)² + 0.1(3 − 0.9)²] / 2. The numerator is 0.648 + 0.007 + 0.484 + 0.441 = 1.58, giving variance 0.79.
Hard selection of candidates 0 and 1 would instead give mean 0.5 and variance 0.25. The soft update allows the other candidates to influence the next search distribution. The weights here are assumed outputs of a soft-selection layer; this calculation verifies refitting, not the projection, a complete optimization run, or a gradient through that run.
Illustrative case study
Fictional case: Spruce Planning trains an objective used by a continuous-action simulator. The original planner selects elite candidates with a hard cutoff, leaving poor gradient feedback to the objective parameters. The team tests DCEM as a differentiable approximation.
It finds that temperature changes both the search behavior and the training signal. Analysts compare final decision quality, gradient stability, and computing cost, then evaluate the trained system using the actual deployment solver. They do not assume that successful differentiation makes the planner globally optimal. This case illustrates a development process, not a reported gain.
Common misconceptions and mistakes
Do not confuse DCEM with differentiating a standard classification cross-entropy loss. The object being differentiated is a sampling-based optimization procedure.
Soft elite weights are not arbitrary independent probabilities. In the original construction, they obey bounds and a fixed-sum constraint tied to the elite count.
Differentiable does not mean deterministic: the procedure can still sample candidates. It also does not mean gradient-free in every relevant sense; training through the solver requires derivatives with respect to the learned quantities.
Finally, finite temperature and finite iterations change the procedure. A gradient through that approximation need not equal the sensitivity of the exact global solution, especially when multiple optima compete.
Business use
DCEM is relevant when a learned model or objective feeds an optimization procedure and the team wants to train the combined system. The benefit must be judged by the quality of the resulting decisions, not solely by whether gradients can be computed.
Track sample count, iteration count, temperature, sampling family, and random-number handling. Compare training-time and deployment-time solver behavior, since replacing a soft solver with a hard one can change decisions. Inspect gradient stability and use simple problems with known solutions to detect implementation mistakes before scaling up.
People also ask
Does DCEM guarantee the best possible solution?
No. It remains a finite sampling-based search with modeling and numerical choices.
Why not keep the hard top-k operation?
Its discrete membership changes make the ordinary selection path unsuitable for straightforward smooth differentiation through objective scores.
Does low temperature recover hard selection?
The original construction approaches hard top-k under the stated distinct-score conditions as temperature tends toward zero; practical gradients can become less useful near a sharp limit.
Check your understanding
Soft elite weights sum to k = 3. When refitting a Gaussian mean using those weights, should the weighted sum be divided by the total number of sampled candidates or by 3?
Answer: Divide by 3, the total weight. Dividing by an unrelated candidate count would not produce the weighted maximum-likelihood mean.
Related terms
- Cross-entropy method
- Reparameterization trick
- Temperature
- Energy-based model
- elite selection
- differentiable optimization
- weighted maximum likelihood
- limited multi-label projection
Learn next
First implement or calculate one ordinary cross-entropy-method update. Then replace binary elite indicators with constrained soft weights and verify the weighted mean and variance. Study how reparameterized sampling and soft selection connect the final output to objective parameters before interpreting the resulting gradients.
More in Probabilistic machine learning
- Deterministic policy gradient - A deterministic policy gradient is the derivative used to improve a policy that chooses a specific continuous action for each state.
- Deterministic training conditional approximation - The deterministic training conditional approximation, or DTC, is a sparse Gaussian process approximation that makes training function values deterministic...
- Diagonal linear discriminant analysis - Diagonal linear discriminant analysis, or DLDA, is a classification method that uses class-specific means and a shared diagonal covariance matrix.
- Dirichlet-multinomial model - A Dirichlet-multinomial model combines a Dirichlet distribution over category probabilities with a multinomial distribution over counts.
- Discrete denoising diffusion probabilistic model - A discrete denoising diffusion probabilistic model, often abbreviated D3PM, generates discrete data by learning to reverse a gradual categorical corruption...
- Discrete variational autoencoder - A discrete variational autoencoder is a variational generative model that uses discrete latent variables, such as categorical choices or binary states, to...
Sources
Amos and Yarats, The Differentiable Cross-Entropy Method, original method and weighted updates: https://proceedings.mlr.press/v119/amos20a/amos20a.pdf Authors' implementation and experiments: https://github.com/facebookresearch/dcem Independent source review: Original DCEM LML constraint, equation 4 and Proposition 3 weighted refitting were read; repository code was not reviewed.
Previous: Differencing Next: Differentiable image pipeline