Is NNGP the same as NTK?

No. The NNGP kernel concerns function covariance under the network prior, while the neural tangent kernel is associated with parameter-gradient relationships and training dynamics.

Is inference always exact?

Closed-form Gaussian regression is available under its assumptions; other likelihoods or approximations can require additional inference methods.

Does the acronym always mean this?

No. NNGP can also refer to nearest-neighbor Gaussian processes in other literature, so the expansion matters.

AI Glossary / Deep learning / Advanced

Neural network Gaussian process

Definition

A neural network Gaussian process, or NNGP, is a Gaussian-process model associated with the infinite-width limit of certain randomly initialized neural networks. Its covariance function is determined by the network architecture, activation functions and parameter-distribution assumptions.

What it means

Imagine drawing many sets of random network weights and examining the functions those networks produce. Under suitable scaling and other conditions, making hidden layers increasingly wide leads to a Gaussian-process description of that distribution over functions.

A Gaussian process is specified by a mean function and a covariance function, or kernel. For any finite collection of input points, its function values have a joint Gaussian distribution. This does not mean the predicted function must be a straight line or that the input data must be Gaussian.

The connection provides a way to analyze neural-network priors and perform Bayesian prediction through a kernel calculation. It is not a statement that every finite, trained neural network behaves identically to a Gaussian process.

How it works

Begin with a specified network family and a probability distribution over its weights and biases. Appropriate scaling keeps activations well behaved as layer widths increase. The limiting covariance between outputs for two inputs can often be computed recursively through the layers.

These covariance values form a kernel matrix for the observed inputs. Combined with a likelihood, the Gaussian-process model can be conditioned on data to produce posterior predictions. For regression with Gaussian observation noise and fixed hyperparameters, the posterior calculations have closed forms.

The kernel captures assumptions inherited from the architecture and prior. Changing the activation, depth or variance settings can change which functions the model favors.

Distinguish NNGP from the neural tangent kernel, which describes a related but different aspect of certain wide-network training dynamics. Also distinguish it from a deep Gaussian process, a composition of Gaussian-process layers rather than simply the NNGP limit of a deep network.

Three practical examples

A researcher compares the prior behavior induced by different activation functions without explicitly constructing an infinitely wide network.
A regression experiment uses an architecture-derived kernel to obtain a predictive mean and model-based uncertainty.
An analysis compares a finite network with its corresponding NNGP to investigate which effects the infinite-width approximation captures or misses.

Think of it this way

Think of considering a vast collection of possible curves and recording how their values tend to vary together at different input points. The kernel summarizes that relationship, allowing predictions without storing every curve individually.

The analogy explains a distribution over functions, not a vote among a finite list of trained models. Its conclusions depend on the chosen prior and observation model. A mathematically precise uncertainty estimate can still be misleading when those assumptions poorly describe the real task.

Worked example

A one-observation prediction

Suppose an illustrative NNGP kernel gives prior variance 1 at both a training input x and a new input x*. Their covariance is 0.5. Use a zero prior mean, observed value y = 2 and Gaussian observation-noise variance 0.25.

With one observation, the posterior mean at x* is 0.5 × 2 / (1 + 0.25) = 0.8. The posterior variance of the latent function is 1 − 0.5² / 1.25 = 0.8.

If predicting a new noisy observation with the same independent noise variance, add 0.25, giving predictive variance 1.05. This distinction separates uncertainty about the function from additional measurement noise.

The kernel values are stipulated for the teaching example; no particular architecture is claimed to have been derived here. The calculation demonstrates conditioning after a valid kernel has been obtained, not the derivation of the infinite-width limit.

Illustrative case study

A fictional modeling team, Willow Sensors, compares a finite neural regressor with an NNGP baseline on laboratory measurements. The Gaussian-process model produces uncertainty intervals, which initially seem to remove the need for further testing.

A held-out experiment under a different operating condition shows that the intervals can be too confident. The team revisits the kernel and noise assumptions and separates interpolation within the measured range from extrapolation beyond it.

It reports both predictive error and interval coverage under clearly defined conditions. The comparison becomes useful evidence about the models' assumptions rather than a claim that Bayesian calculations automatically make every forecast reliable.

Common misconceptions and mistakes

Infinite width requires building infinitely many neurons. The limiting model is evaluated through finite kernel calculations for the chosen data.
NNGP describes every network after ordinary gradient training. The correspondence has specific prior and limiting assumptions.
Gaussian process means Gaussian-shaped input data. The Gaussian statement concerns joint distributions of function values.
Posterior uncertainty covers every possible modeling error. It is conditional on the specified model and inference assumptions.

Business use

NNGPs are mainly useful to research teams studying neural-network behavior, kernel baselines or probabilistic prediction. They can clarify what an architecture implies before finite-network training adds other effects.

A practical assessment should include kernel suitability, the number of observations, matrix-computation costs and the calibration of predictive uncertainty. An infinite-width theoretical model is not automatically cheaper than a finite neural network; kernel operations can become demanding as the dataset grows. Business users should ask what the uncertainty represents and whether it has been checked against the operating conditions in which decisions will be made.

Check your understanding

Answer: A report gives a latent-function variance but a user wants uncertainty for a new noisy measurement.

Are those necessarily the same?

Answer: No. The measurement prediction also includes the observation-noise model. In the worked example, that adds 0.25 to the latent variance.

Related terms

Gaussian process
Bayesian neural network
Prior (regularizer)
Posterior
Neural tangent kernel
Deep Gaussian process
covariance kernel
infinite-width limit
prior
predictive uncertainty

Learn next

Study ordinary Gaussian-process regression before deriving an NNGP kernel. Reproduce the one-observation calculation with a different covariance, then compare the assumptions behind NNGP prediction and neural-tangent-kernel training analysis.

Sources

Lee and colleagues, Deep Neural Networks as Gaussian Processes: https://arxiv.org/abs/1711.00165 Rasmussen and Williams, Gaussian Processes for Machine Learning, regression chapter: https://gaussianprocess.org/gpml/chapters/RW2.pdf Jacot, Gabriel and Hongler, Neural Tangent Kernel: Convergence and Generalization in Neural Networks: https://arxiv.org/abs/1806.07572 Neural Tangents, empirical NNGP kernel documentation: https://neural-tangents.readthedocs.io/en/latest/_autosummary/neural_tangents.empirical_nngp_fn.html

Previous: Neural network Next: Neural network pruning