Does infinite width guarantee better accuracy?

No. A limiting description is not a universal performance ranking.

Is a Gaussian-process limit the same as training a network with gradient descent?

No. A prior distribution over functions and a particular optimization trajectory answer different questions.

Can wide networks still learn features?

Yes. The answer depends on the parameterization and training regime; the familiar fixed-kernel limit does not cover every possible scaling.

AI Glossary / Deep learning / Advanced

Large width limits of neural networks

Definition

Large width limits of neural networks are mathematical descriptions of what happens when the number of units or channels in selected layers grows without bound under specified scaling assumptions. These limits can connect neural networks with Gaussian processes, kernel methods, or other learning dynamics. They are analytical models of network behavior, not instructions to build a physically infinite network.

What it means

Width usually counts units within a layer, while depth counts successive computational stages. A width limit asks whether some property becomes simpler or more predictable as layers become increasingly wide. The property might be the distribution of random outputs before training, the evolution of predictions during training, or the behavior of learned features. There is no single universal infinite-width result: the architecture, initialization, parameter scaling, learning-rate scaling, and training procedure determine which mathematical limit is relevant.

How it works

A common random-network limit scales sums so their variance remains controlled as more units contribute. Under appropriate conditions, outputs across any fixed finite collection of inputs approach a joint Gaussian distribution, defining a Gaussian process. A different analysis uses the neural tangent kernel, formed from inner products of output gradients with respect to parameters. In a standard infinite-width training regime, this kernel approaches a deterministic limit and changes negligibly during training. These connections clarify specific regimes; they do not imply that every wide trained network behaves identically or lacks feature learning.

Three practical examples

Theory research: use a tractable kernel limit to study how predictions respond to training examples.
Model analysis: compare finite networks of increasing width with a limiting prediction to assess whether the approximation is useful.
Baseline design: evaluate a corresponding kernel or Gaussian-process method alongside a neural model, while accounting for their different computational costs.

Think of it this way

Think of averaging the effects of many small, independently generated contributions until their collective behavior becomes predictable. The analogy explains why limiting distributions can simplify analysis. Its limit is that neural contributions may be dependent, transformed, or trained, so the required assumptions and scaling must be checked rather than inferred from a large head count.

Worked example

Consider a toy scalar output f_n = (z1 + z2 + ... + zn) divided by the square root of n, where the contributions are independent, each has mean zero, and each has variance 4. The sum has variance 4n. Dividing by the square root of n divides its variance by n, leaving variance 4 for every n. At n = 100, the unscaled sum has variance 400, while the scaled output still has variance 4. Dividing by n instead would produce variance 4/n, which is 0.04 at n = 100 and tends toward zero.
With suitable distributional conditions, the square-root scaling supports a nontrivial Gaussian limit. This is a simplified illustration of scaling, not a proof for a complete neural network. Across two inputs, contributions from the same unit can be correlated, and those correlations determine the limiting covariance kernel. Knowing that each output has variance 4 alone does not tell us how predictions at different inputs relate.

Illustrative case study

A fictional research team studies whether a kernel approximation explains its small regression network. It trains several widths with explicitly documented initialization and learning-rate choices, then compares predictions with the theoretical baseline. Some settings track the limit closely; others show substantial differences as learned representations change. The team reports those differences and checks both held-out accuracy and resource use. It does not interpret a theorem about one scaling regime as a reason to increase production width indefinitely.

Common misconceptions and mistakes

An infinite-width Gaussian-process prior and neural tangent kernel training dynamics are related but distinct objects. Width does not mean depth, and a theorem at initialization does not automatically describe the trained model. Limits may require fixed depth or particular orders of taking multiple limits. A finite network can depart materially from its asymptotic approximation. Nor does a constant limiting tangent kernel mean that every finite parameter is literally unchanged; predictions may evolve through many individually small parameter updates.

Business use

For most business teams, width-limit theory is useful when interpreting research claims or choosing analytical baselines. Ask which network family, scaling, and training assumptions support the result, and whether experiments cover the widths and data sizes of interest. Kernel calculations can become expensive as the number of examples grows, so replacing an explicit neural model with a limiting method does not automatically save resources. Evaluate accuracy, uncertainty quality where relevant, and operational cost directly. The theory provides explanations and hypotheses, not a blanket deployment recommendation.

Check your understanding

Independent zero-mean contributions each have variance 4. If 100 are summed and divided by 10, what is the output variance?

Answer: The sum's variance is 400, and division by 10 divides variance by 100, giving 4. This demonstrates variance-preserving scaling, not the accuracy or generalization of a trained network.

Related terms

Gaussian process
Neural tangent kernel
Central limit theorem
Feature learning
Asymptotic analysis
Network width
initialization
covariance kernel
parameterization

Learn next

Review variance, covariance, and the difference between a random function prior and a fitted predictor. Then compare neural-network Gaussian-process and neural tangent kernel derivations for the same simple architecture. Keep a checklist of which quantities are held fixed, which grow, and how parameters and learning rates are scaled.

Sources

Lee and colleagues, Deep Neural Networks as Gaussian Processes: https://arxiv.org/abs/1711.00165 Jacot, Gabriel, and Hongler, Neural Tangent Kernel: Convergence and Generalization in Neural Networks: https://arxiv.org/abs/1806.07572 Yang, Scaling Limits of Wide Neural Networks with Weight Sharing: https://arxiv.org/abs/1902.04760

Previous: Large-scale learning Next: LARS Lasso