AI Glossary / Deep learning / Intermediate

GELU Function

Definition

The GELU function, short for Gaussian error linear unit, is a smooth neural-network activation defined by GELU(x) = x times Phi(x), where Phi is the cumulative distribution function of a standard normal variable. It scales an input by a value determined by that input. Large positive values pass through almost unchanged, while strongly negative values approach zero from below. GELU is different from the gated linear unit, or GLU.

What it means

An activation function introduces a nonlinear transformation between learned layers. GELU makes this transition smoothly instead of cutting every negative input to zero as ReLU does. The normal cumulative distribution supplies a weighting factor between zero and one, but the final result is not itself a probability: it can be negative or exceed one. Using a Gaussian function in the formula does not require the network's actual inputs to be normally distributed, nor does it mean random noise is sampled on each ordinary evaluation.

How it works

The exact expression can be written using the error function as 0.5 times x times [1 plus erf(x divided by the square root of 2)]. It is applied separately to each element, preserving the tensor's shape. Some implementations offer a tanh-based approximation; others evaluate the normal-CDF form through numerical special functions. Backpropagation uses the derivative Phi(x) plus x times phi(x), where phi is the standard normal density. This derivative can be negative over part of the negative-input region, so smoothness should not be confused with monotonicity.

Three practical examples

• Representation learning: a dense hidden layer can use GELU to transform intermediate features before another learned projection.
• Model reproduction: an engineer checks whether a published model specifies exact GELU or a particular approximation before matching an implementation.
• Activation comparison: a research team evaluates GELU, ReLU, and another suitable activation under a controlled training setup, comparing task quality and computational cost rather than assuming one always wins.

Think of it this way

Think of a soft filter that gradually changes how much of each signed input passes through. Unlike a hard switch, nearby inputs receive smoothly changing treatment. The analogy is limited because GELU's final curve is not simply an always-increasing filter setting: multiplying the signed value by its Gaussian weighting produces a small negative dip.

Worked example

For the input vector [-2, -1, 0, 1, 2], the standard normal CDF values are approximately [0.02275, 0.15866, 0.5, 0.84134, 0.97725]. Multiplying each input by its corresponding factor gives GELU outputs approximately [-0.04550, -0.15866, 0, 0.84134, 1.95450]. ReLU would instead return [0, 0, 0, 1, 2]. The GELU output at -2 is less negative than at -1, illustrating that the whole activation is not monotonic.
At x = -1, phi(x) is approximately 0.24197, so the derivative is 0.15866 minus 0.24197 = -0.08331. At x = 0, the derivative is 0.5. If an upstream loss derivative at zero were 4, the chain rule would pass a derivative of 2 through GELU at that point. These calculations use the exact mathematical definition with rounded values. A chosen approximation can differ slightly in outputs and derivatives, especially when numerical precision and implementation details are considered.

Illustrative case study

A fictional analytics team ports a trained model between inference systems and notices small output differences. It checks preprocessing, weight conversion, and numerical precision, then discovers that the two systems use different GELU approximation settings. The team aligns the intended operation and tests end-to-end prediction tolerances. The example illustrates that reproducing a model requires matching mathematical components as well as copying its weights; it does not claim that every small discrepancy is caused by GELU.

Common misconceptions and mistakes

GELU does not set all negative inputs to zero, and it is not a random dropout mask. Its CDF factor should not be displayed as confidence in a business prediction. The function is smooth but not everywhere increasing, and its name does not make it equivalent to ELU, SiLU, or GLU. Replacing another activation with GELU may require retraining and evaluation; it is not a guaranteed improvement or a generally safe way to alter an already trained model without checking behavior.

Business use

For applied projects, activation selection should fit the architecture and deployment environment. If reproducing or adapting a pretrained model, preserve the documented activation behavior unless a deliberate experiment supports changing it. If designing a model, compare alternatives under consistent validation and training conditions, and measure actual inference cost on the intended system. Record approximation mode and numerical precision where they matter for reproducibility. An activation is one component of performance, alongside data quality, architecture, optimization, and evaluation design.

People also ask

Is GELU's output between zero and one?

No. The weighting factor is bounded, but multiplying by x gives a signed output that can exceed one.

Does GELU draw Gaussian random numbers?

Ordinary deterministic GELU evaluation does not; it evaluates a function involving the Gaussian CDF.

Is exact GELU identical to its tanh approximation?

No. The approximation is designed to be close, and the intended version should be specified when matching a model.

Check your understanding

Why is GELU(-1) negative even though Phi(-1) is positive?

Answer: GELU multiplies the negative input by the positive CDF value, giving approximately -0.15866 rather than a probability or a hard zero.

Related terms

• Activation function
• Cumulative distribution function
• Backpropagation
• ReLU
• ELU
• SiLU
• gated linear unit
• Gaussian distribution

Learn next

plot GELU, ReLU, and SiLU over the same signed input range and inspect their derivatives. Then compare exact GELU with the approximation used in a chosen framework before interpreting numerical differences in exported models.

More in Deep learning

• Frequency principle/spectral bias - The frequency principle, also called spectral bias in closely related research, describes a tendency of many neural networks and training setups to learn...
• Fully connected layer - A fully connected layer is a neural-network layer in which each output unit has a trainable weight for every input feature in the vector it receives.
• Gating mechanism - A gating mechanism is a component that regulates how information contributes to a neural-network calculation, commonly by multiplying a signal by an...
• Generalized Autoregressive Conditional Heteroskedasticity - Generalized autoregressive conditional heteroskedasticity, usually abbreviated GARCH, is a family of statistical time-series models for variance that...
• Generalized Hebbian algorithm - The generalized Hebbian algorithm, or GHA, is an unsupervised learning rule for a linear neural network that learns leading directions of variation in its...
• Generative topographic map - A generative topographic map, usually discussed as generative topographic mapping or GTM, is a probabilistic nonlinear latent-variable model for...

Sources

Hendrycks and Gimpel, Gaussian Error Linear Units: https://arxiv.org/abs/1606.08415 Original authors' implementation and approximation notes: https://github.com/hendrycks/GELUs PyTorch, GELU reference: https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.activation.GELU.html

Previous: Gelbach decomposition Next: GenDICE