Why not start all hidden weights at zero?

Identically configured hidden units can remain symmetric and learn the same thing, limiting useful differentiation.

Is He initialization always better than Xavier initialization?

No. Their assumptions and intended activation behavior differ; suitability depends on the model.

Does setting a random seed guarantee identical results everywhere?

No. Other sources of nondeterminism, software versions and hardware operations may also affect results.

AI Glossary / AI and machine learning / Intermediate

Weight initialization

Definition

Weight initialization is the choice of starting values for a neural network's trainable weights before optimization begins. Suitable initial values help signals and gradients travel through the network without becoming excessively small, large or identical across units.

What it means

Training adjusts parameters gradually. Where that process starts can influence whether learning is stable, how quickly it progresses and which solution it reaches. Random initialization is therefore more than filling a table with arbitrary numbers.

Different layers and activation functions call for different scales or structures. Initializing a model from pretrained weights is another starting strategy and should be distinguished from training from scratch. Biases and special parameters may follow different rules from ordinary connection weights; a blanket instruction to make every parameter random is not appropriate.

How it works

In a dense layer, each output combines many weighted inputs. If independent weights have an unsuitable variance, repeated layers can amplify or shrink activation magnitudes. Backward-propagated gradients can face related problems.

Glorot, or Xavier, initialization balances scale using the number of incoming and outgoing connections. He, or Kaiming, initialization accounts for rectifier-style nonlinearities and commonly uses variance approximately 2/fan_in for a ReLU layer under its assumptions. Exact formulas depend on the distribution, gain and selected mode.

Randomly different starting weights also break symmetry between otherwise identical hidden units. Architecture details such as residual connections, normalization and parameter sharing may require specialized choices. The initializer must match the actual tensor layout and forward computation, not just a familiar method name.

Three practical examples

Image classification: An engineer uses a rectifier-appropriate initialization for convolutional layers and checks early activations and gradients before a long training run.
Reproducible experiments: A researcher repeats training with several random seeds to see whether a reported improvement depends heavily on one favorable starting point.
Transfer learning: A team begins with pretrained feature-extractor weights and initializes only a newly added output layer. It records which parameters were reused and which started afresh.

Think of it this way

Think of setting the starting tension in a series of linked springs. Too loose or too tight, and a disturbance may disappear or become difficult to control as it passes along the chain. The analogy captures scale, but neural training also depends on nonlinearities, data and optimization rather than physical spring laws.

Worked example

Suppose a dense ReLU layer has one hundred incoming connections per unit. A common He-normal choice uses weight variance 2/100=0.02, giving standard deviation sqrt(0.02), approximately 0.141.

For comparison, a Glorot-normal layer with one hundred inputs and fifty outputs uses a base variance 2/(100+50), approximately 0.0133, before any applicable gain adjustment. Its corresponding standard deviation is approximately 0.115.

These are starting distribution scales, not fixed values assigned to every weight. Samples differ across connections. The example assumes conventional fan counts and illustrates why copying a standard deviation from a differently shaped layer can be inappropriate.

Illustrative case study

Fictional case: Signal Forge's prototype fails to learn consistently. The team discovers that a custom layer stores its weight matrix in the opposite orientation from the initializer's assumption, causing the wrong fan count to determine scale.

After correcting the calculation, it inspects activation distributions and repeats a short training experiment across multiple seeds. The report separates the initialization correction from later changes to the optimizer. This fictional case illustrates diagnosis and reproducibility, not a promised improvement for every network.

Common misconceptions and mistakes

Initializing every hidden weight to the same value can leave symmetric units learning identical features. However, that does not mean zero initialization is always wrong for every parameter: biases and architecture-specific gates may legitimately start at zero.

Larger random values do not mean a more expressive starting model. They may saturate activations or destabilise gradients. A popular initializer also cannot compensate for incorrect labels, a broken loss function or an unsuitable learning rate. Finally, one successful seed is not evidence that a method trains reliably; variability deserves measurement.

Business use

Good initialization can reduce wasted experimentation and make model development more dependable. Teams should retain framework defaults when they match the architecture, and justify custom changes with diagnostics rather than intuition alone.

Record initialization settings, random seeds, pretrained checkpoints and any parameters reset during fine-tuning. Compare methods under consistent training budgets. For users purchasing AI services, initialization is usually an implementation detail, but reproducibility and robustness across training runs remain relevant evidence of engineering quality.

Check your understanding

A ReLU layer doubles its number of incoming connections. Under the basic He-normal rule, should its starting weight variance remain unchanged?

Answer: No. Since variance is approximately 2/fan_in, doubling fan_in halves the variance.

Related terms

Vanishing gradient
Transfer learning
Xavier initialization
He initialization
fan-in
fan-out
vanishing gradients
exploding gradients
symmetry breaking

Learn next

Build a small multilayer network and compare activation scales before training under several initializers. Keep inputs and architecture fixed. Inspect both forward activations and backward gradients, then repeat short training runs across seeds. Read the framework's fan-count convention before applying an initializer to a custom layer.

Sources

Glorot and Bengio, Understanding the difficulty of training deep feedforward neural networks: https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf (normalized initialization formula consulted). He, Zhang, Ren and Sun, Delving Deep into Rectifiers: https://arxiv.org/pdf/1502.01852 (Section 2.2, Equation 10 and initialization discussion consulted). These primary papers support the stated fan and activation assumptions. The PyTorch documentation URL returned a redirect-only response during this review; its current API details were not verified.

Previous: Weight decay Next: Weight space