Must noise be uniformly distributed?

No. A suitable known distribution can be chosen for the problem.

Can the same value appear in both classes?

Yes. Labels identify the sampling source, not an intrinsic property of the value.

Does NCE avoid all probability calculations?

No. It uses the noise density and the model score or density in the classifier.

AI Glossary / Probabilistic machine learning / Advanced

Noise-contrastive estimation

Definition

Noise-contrastive estimation (NCE) learns a statistical model by training a classifier to distinguish observed data from samples drawn from a known noise distribution. The classifier incorporates the model's density and the noise density. This can allow estimation of an otherwise difficult normalization constant alongside the model's other parameters.

What it means

A model may assign an unnormalized score to each possible observation without making it easy to calculate probabilities that sum or integrate to one. NCE turns learning that model into a comparison problem: which observations came from the data source, and which came from a reference generator?
Noise means the chosen comparison distribution. It does not necessarily mean corrupted measurements, meaningless text or a small perturbation added to a real observation. The distinction between source labels is constructed for training; both sources can produce the same observed value.

How it works

Suppose there are k noise samples per data sample. If the model assigns density p_theta(x) and the known noise distribution assigns q(x), the modeled probability that x came from the data source is p_theta(x) / [p_theta(x) + k q(x)]. Its log odds are log p_theta(x) minus log q(x) minus log k.
The training objective rewards data samples receiving high data-source probability and noise samples receiving high noise-source probability. For an unnormalized model, a learnable log-scale parameter can represent the negative log normalization constant. Training need not calculate the normalizing integral at every update.
The noise distribution must adequately cover the data support for the usual identification arguments. Statistical consistency also requires appropriate model and regularity assumptions; it is not a promise that every finite training run produces an exactly normalized or accurate density.

Three practical examples

Discrete model teaching: students estimate a distribution over a small vocabulary by classifying real draws against draws from a known reference distribution. The small example permits exact checks.
Energy-based modeling: researchers use a tractable noise generator to train a model whose normalization would otherwise require a difficult sum or integral. They assess the fitted model separately from classifier accuracy.
Sampling-policy review: a team changes the number of noise draws per observation and updates the classifier's prior-odds correction accordingly. Forgetting that correction would change the implied density relationship.

Think of it this way

Imagine mixing deliveries from a supplier with deliveries from a reference warehouse whose inventory proportions are known. A classifier learns which source likely produced an item. Because the reference proportions and mixing ratio are known, its source odds contain information about the supplier's proportions. Merely attaching an arbitrary real-or-fake label would not provide the same density calculation.

Worked example

Consider an original two-outcome example with possible values A and B. The data distribution is p(A) = 0.75 and p(B) = 0.25. Choose uniform noise q(A) = q(B) = 0.5 and generate two noise samples per data sample, so k = 2.
At the correct model, the data-source probability for A is 0.75 / (0.75 + 2 times 0.5) = 3/7, approximately 0.4286. For B it is 0.25 / (0.25 + 2 times 0.5) = 1/5, or 0.2. Both are below 0.5 because noise is more common in the training mixture. This does not mean the learned data distribution is wrong.
Recover the modeled density using p(x) = k q(x) h(x) / [1-h(x)], where h is the data-source probability. For A, this gives 1 times (3/7)/(4/7) = 3/4. For B, it gives 1 times (1/5)/(4/5) = 1/4. The recovered probabilities sum to one.
If someone mistakenly uses k = 1 with these same classifier probabilities, they recover 3/8 and 1/8, summing to 1/2. The missing mixture correction produces the wrong scale. With balanced sampling and a correctly adjusted classifier, the appropriate probabilities would instead be 0.6 and 1/3.
An unnormalized model with scores 3 and 1 has normalization constant 4. A learned additive log-scale of -log(4) would turn those scores into the desired probabilities. This is an algebraic illustration of the correct solution, not a claim that an optimizer was run. In finite data, fitting this scale does not guarantee exact normalization without checking it.

Illustrative case study

Fictional company Elm Language Lab experiments with a discrete probabilistic model. Its researchers record the noise distribution, its support and the number of noise draws used for every data observation. They first verify that their classifier reproduces the two-outcome calculation.
When a new noise generator makes classification much easier, they avoid declaring the density model better on that basis alone. They examine held-out model behavior and whether relevant outcomes still receive enough comparison samples. This fictional case illustrates experiment design, not a reported performance gain or production deployment.

Common misconceptions and mistakes

NCE is not identical to adding random noise to inputs. It is also not a generic name for every contrastive objective. InfoNCE and negative-sampling objectives are related topics, but their formulas and estimation claims should not be substituted without checking assumptions.
A good source classifier is not automatically a good data-density model. Classification can be easy because the reference distribution is poorly chosen. Omitting the known noise density or the sample-ratio correction changes the objective. Finally, the learned normalization parameter does not make every intermediate or finite-sample model exactly normalized by construction.

Business use

NCE can help research teams train probabilistic models when direct normalization is costly. The practical tradeoff includes the cost of generating and evaluating noise, the quality of that reference distribution and the number of comparison samples. Business evaluation should focus on the intended predictive or modeling use, with source-classification accuracy treated as supporting evidence rather than the final success measure.

Check your understanding

Why is the example's probability for A below one half?

Answer: There are twice as many noise draws as data draws.

What does omitting k change?

Answer: The mixture-prior correction and therefore the recovered density scale.

What makes the normalizer learnable here?

Answer: The classification objective constrains the model's scale through comparison with known noise.

Related terms

Logistic regression
Energy-based model
Negative sampling
Unnormalized model
Partition function
Density ratio estimation
InfoNCE

Learn next

Derive the source-classification probability from Bayes' rule, then vary the noise distribution and sampling ratio in a small discrete example. Compare NCE with ordinary maximum-likelihood estimation.

Sources

Gutmann and Hyvarinen, Noise-contrastive estimation (2010), Sections 2.1–2.3: https://proceedings.mlr.press/v9/gutmann10a/gutmann10a.pdf . Their expanded 2012 account, Section 2.1 Equation 3, independently consulted for unequal sampling ratios: https://www.jmlr.org/papers/volume13/gutmann12a/gutmann12a.pdf . The calculation and fictional case are original illustrations.

Previous: Noise-based logic Next: Noise factor