Must the terminal state be all masks?

No. Absorbing-mask diffusion is one design; other transitions can approach other simple distributions.

Can the model generate multiple tokens in an iteration?

Depending on the reverse-model factorization, it can update multiple positions, but that does not guarantee a universal speed advantage.

Does a mask always mean an originally missing value?

No. During training it can be deliberately introduced by the corruption process, which is different from naturally missing data with an unknown missingness mechanism.

AI Glossary / Probabilistic machine learning / Intermediate

Discrete denoising diffusion probabilistic model

Definition

A discrete denoising diffusion probabilistic model, often abbreviated D3PM, generates discrete data by learning to reverse a gradual categorical corruption process. Instead of adding continuous Gaussian noise directly to category labels, it uses transition probabilities between discrete states. The choice of those transitions determines what “noise” means for the data.

What it means

Discrete values include tokens, category labels, and quantized measurements. Adding 0.3 to an arbitrary token identifier has no natural meaning. A discrete diffusion model instead corrupts a value by probabilistically replacing it with another category or a special mask state.
The learned reverse process predicts distributions over cleaner discrete states. It can use the surrounding sequence or other context even when the forward corruption acts independently on positions. The model remains generative: denoising a corrupted sequence can produce plausible content without proving that it matches an unknown original or a true statement.

How it works

Specify a transition matrix Q_t for each forward step. Its entry in row i and column j gives the probability of moving from state i to state j, with each row summing to one. Products of these matrices describe cumulative corruption.
Possible designs include uniform replacement, transitions favoring nearby or similar categories, and transitions into an absorbing mask state. A suitable terminal distribution supplies the starting point for generation.
Train a neural reverse model using corrupted examples and a variational objective; the original D3PM work also investigates an auxiliary clean-data prediction loss. The model can predict original-state probabilities that are combined with known forward-process posteriors to form reverse transitions. Generation then repeatedly samples cleaner states according to the chosen reverse process.

Three practical examples

Text-generation research: corrupt discrete tokens and learn iterative reconstruction, evaluating coherence and content rather than assuming a token-valid output is correct.
Quantized-image modeling: use categorical pixel values with transitions that respect their ordered intensity structure where appropriate.
Structured categorical experiments: compare uniform corruption with masking to test which information-removal process supports useful learning for a particular representation.

Think of it this way

Think of gradually replacing words in a message with blank cards, then learning how to fill the cards using context. This captures one absorbing-mask design. The analogy has limits: other discrete diffusion processes replace symbols rather than blanking them, and the learned reconstruction may be a plausible new message rather than the exact message that was removed.

Worked example

Consider an original three-state forward process with states A, B, and M, where M is a mask. At each step, A stays A with probability 0.7 and becomes M with probability 0.3. B behaves similarly, and M remains M with probability 1.
The transition rows are therefore A: (0.7, 0, 0.3), B: (0, 0.7, 0.3), and M: (0, 0, 1). Starting from A, after two steps the probability of still being A is 0.7² = 0.49. The probability of being masked is 1 − 0.49 = 0.51.
The masked outcome can arise from masking at the first step, with probability 0.3, or surviving the first step and masking at the second, with probability 0.7 × 0.3 = 0.21. Thus, given the original A and a mask at step two, the probability that step one was already masked is 0.3 / 0.51 = 10/17, approximately 0.588235.
During generation, the original A is not known. The learned model must represent possible originals using context. This calculation verifies the forward process and one training-time posterior; it does not supply a trained reverse model.

Illustrative case study

Fictional case: Cypress Language Lab compares uniform token replacement with absorbing-mask corruption. The team keeps evaluation data separate and tests both generation quality and sampling cost.
Masking gives a clear missing-token interpretation, while uniform replacement can create misleading visible tokens. Neither design is declared universally superior. Analysts inspect outputs for repetition and inconsistent statements, and report those failures alongside numerical scores. The case illustrates how the corruption choice changes a modeling experiment, not a measured product advantage.

Common misconceptions and mistakes

Discrete diffusion is not simply Gaussian noise applied to integer labels followed by rounding. That approach imposes an ordering and geometry that arbitrary categories may not possess.
An absorbing state is absorbing in the forward corruption process. The reverse generative process can leave the mask state as it reconstructs data.
Do not assume that independent forward corruption implies an independent learned language model. A neural reverse model can condition on the full noisy sequence.
Finally, a transition based on embedding similarity is a design hypothesis, not proof that semantic structure will improve results. The original work evaluates different structures and does not support a universal winner.

Business use

Businesses exploring discrete generative models should connect the corruption design to the representation and intended task. Token generation, categorical completion, and quantized measurements impose different requirements on valid transitions and output assessment.
Measure fidelity, diversity, task success, and actual generation cost. Check whether the terminal distribution is adequately reached under the chosen schedule and whether the sampler matches training assumptions. A syntactically valid category or token sequence can still be unsuitable; domain-specific evaluation remains necessary before operational use.

Check your understanding

In the example, what is the probability that an original B remains B after three forward steps?

Answer: It is 0.7³ = 0.343. Once B becomes M it cannot return during the forward process, so remaining B requires surviving all three steps.

Related terms

Masked language model
Discrete diffusion
categorical distribution
transition matrix
absorbing state
variational objective
reverse process
quantization

Learn next

Review categorical Markov transitions and multiply a small transition matrix across two steps. Then compare uniform replacement and absorbing masks. Distinguish the tractable posterior that conditions on a known clean training example from the learned reverse transition used when generating an unknown example.

Sources

Austin, Johnson, Ho, Tarlow, and van den Berg, Structured Denoising Diffusion Models in Discrete State-Spaces, original paper: https://arxiv.org/html/2107.03006v3 Independent source review: Original paper sections 3.1–3.4 checked for categorical transitions, absorbing masks, reverse parameterization and auxiliary loss.

Previous: Discrete cosine transform Next: Discrete diffusion model