Does NADE assume variables are independent?

No; later conditionals can depend on earlier variables.

Does it need a global partition-function estimate?

Not for its normalized autoregressive formulation.

Can the basic binary model handle arbitrary continuous values directly?

No; a suitable continuous-data extension is needed.

AI Glossary / Probabilistic machine learning / Advanced

Neural autoregressive density estimator

Definition

A neural autoregressive density estimator models a joint distribution as a product of conditional distributions predicted by neural networks. The classic NADE model shares computations and parameters across these conditionals, giving tractable probabilities for complete observations. Its original binary formulation is commonly called the Neural Autoregressive Distribution Estimator; later variants extend the approach beyond binary data.

What it means

Suppose a record contains several yes-or-no attributes. Modeling each attribute independently would miss relationships between them. NADE instead predicts each attribute using the attributes that precede it in a chosen order. Multiplying the conditional probabilities yields a probability for the complete record.
The order is a modeling convention, not necessarily a timeline or causal sequence. For example, listing a product's material before its shape does not claim that material causes shape. The learned conditionals reflect statistical patterns in the training data. Exact evaluation under the model also does not mean the model perfectly describes the real population.

How it works

For a binary vector x, use the chain rule p(x) = product of p(x_i given x_1 through x_(i−1)). A sigmoid output gives the probability that each binary variable equals one. The contribution to the joint probability is that value when the observed bit is one and its complement when the bit is zero.
In classic NADE, the hidden activation for each conditional uses only preceding variables. A shared input-weight matrix lets the hidden preactivation be updated incrementally as another observed variable becomes available. Output weights and biases produce the successive conditional probabilities. This sharing makes the basic computation efficient without leaking the predicted variable into its own prediction.
Training minimizes negative log-likelihood, which adds the negative log probabilities of the observed conditional outcomes. Generation proceeds in order: draw the first variable, then draw each later variable conditioned on the values already generated. All conditionals are normalized, so no separately estimated global partition function is needed. Arbitrary missing-variable queries may still require summation or additional inference.

Three practical examples

A product-data team models combinations of binary attributes, such as whether an item has particular features. A joint model can represent patterns that independent attribute frequencies miss. Rare combinations still require investigation before being treated as errors.
An image-modeling experiment represents a small black-and-white image as an ordered vector of binary pixels. The model assigns probabilities to complete images and can generate pixels in sequence. Such generation follows the model's distribution, not a guarantee of meaningful visual content.
A research team models binary indicators in a table and compares held-out log-likelihood against a mixture-of-Bernoullis baseline. This tests how well the fitted distribution predicts complete records, while any practical classification or anomaly-detection use needs its own evaluation.

Think of it this way

Imagine completing a questionnaire one answer at a time. Each new answer is predicted using the answers already given. The chance assigned to the entire answer sheet is the product of those conditional chances. A shared internal summary avoids rebuilding everything learned from earlier answers at each question.

Worked example

Construct a tiny NADE with two binary variables in order x1, x2 and one sigmoid hidden unit. Write sigmoid(t) = 1/(1 + exp(−t)). Set the hidden bias to zero and the input weight for x1 to ln(3). Let the first output have zero weight and zero bias, so p(x1 = 1) = 0.5.
For the second output, choose weight 4*ln(3) and bias −2*ln(3). Its hidden value is sigmoid(ln(3)*x1). When x1 = 0, this equals 0.5, giving output logit 4*ln(3)*0.5 − 2*ln(3) = 0 and probability 0.5. When x1 = 1, the hidden value is 0.75, giving logit ln(3) and probability 0.75.
Thus the joint probabilities are p(0,0) = 0.5 × 0.5 = 0.25; p(0,1) = 0.25; p(1,0) = 0.5 × 0.25 = 0.125; and p(1,1) = 0.5 × 0.75 = 0.375. They sum to one. The negative log-likelihood of record (1,1), using natural logarithms, is −ln(0.375), approximately 0.980829.
Now reverse the prediction question: what is p(x1 = 1 given x2 = 1)? It is 0.375/(0.25 + 0.375) = 0.6. This is neither the first output's unconditional 0.5 nor the second output's forward conditional 0.75. These parameters are stipulated for demonstration; no fitted data or real-world performance is implied.

Illustrative case study

Fictional company Cedar Records proposes modeling binary product attributes with a small NADE. Its analysts reserve later records for evaluation, check coding consistency, and compare against an independent-Bernoulli model. They also test whether attribute ordering and hidden-layer size materially affect results.
For records receiving low probability, the team examines examples manually to distinguish unusual but valid products from coding mistakes. It does not interpret likelihood as a direct probability of an error. This invented scenario describes an evaluation process; it does not claim better data quality, reduced review costs, or a successful deployment.

Common misconceptions and mistakes

Autoregressive does not necessarily mean time-series forecasting. A fixed ordering of non-temporal variables is enough. Hidden units in the classic feed-forward NADE calculation are deterministic activations; they are not sampled latent variables that must be integrated out to evaluate a complete record.
Another mistake is using the probability of one for every binary observation, including zeros. The zero outcome requires the complement. Finally, tractable complete-record likelihood does not make every marginal, conditional, or missing-data calculation equally cheap. Summing over many unobserved earlier variables can be expensive.

Business use

Consider this model when joint probabilities or synthetic samples of structured records have a defined purpose. Compare held-out likelihood, computation cost, and downstream usefulness with simpler alternatives. Keep preprocessing, variable ordering, and the handling of missing observations documented so probability scores can be interpreted consistently across training and deployment.

Check your understanding

What is p(1,0) in the example?

Answer: 0.125.

Why do the four joint probabilities sum to one?

Answer: They combine normalized conditional distributions through the chain rule.

Is p(x2 = 1 given x1 = 1) the reverse conditional?

Answer: No; the reverse value here is 0.6.

Related terms

Density estimation
Sigmoid
Restricted Boltzmann Machine
Autoregressive model
chain rule of probability
negative log-likelihood
ancestral sampling

Learn next

Compare NADE with independent Bernoulli models, then study how autoregressive ordering affects computation, generation, and missing-data queries.

Missing data

Sources

Larochelle and Murray, The Neural Autoregressive Distribution Estimator (2011), Section 4: https://proceedings.mlr.press/v15/larochelle11a/larochelle11a.pdf . The two-variable network and Cedar Records scenario are original illustrations. Section 4 independently consulted for shared hidden computation and conditional outputs; example recalculated.

Previous: Neural Architecture Search Next: Neural cryptography