What does alpha control?

It scales the negative branch and sets its limiting value at minus alpha for positive alpha.

Does ELU return zero for every negative input?

No. It returns negative values that approach its lower saturation level.

Is ELU differentiable at zero for every alpha?

No. In the original formula, the left and right derivatives match at zero when alpha equals one.

AI Glossary / Deep learning / Intermediate

ELU Function

Definition

The ELU function, short for exponential linear unit, is a neural-network activation function that is linear for positive inputs and follows a saturating exponential curve for negative inputs. With parameter alpha greater than zero, it returns x when x is positive and alpha times (exp(x) minus 1) when x is zero or negative. It introduces nonlinearity while allowing negative activation values.

What it means

An activation function transforms a neuron's numerical input before the result is used by later layers. ELU preserves positive values directly, while negative values approach a lower limit of minus alpha instead of being set to zero as in an ordinary ReLU. This changes both the values passed forward and the gradients used during training. The choice can affect learning behavior, but no activation function is automatically best for every architecture or dataset.

How it works

For x greater than zero, ELU has derivative one. For x less than zero, its derivative is alpha times exp(x), which is positive but becomes small for very negative inputs. The function is continuous at zero. Its left derivative at zero is alpha and its right derivative is one, so it is differentiable there when alpha equals one. The parameter controls the negative saturation level. Original ELU should be distinguished from related functions such as CELU and SELU, which use different formulations or scaling.

Three practical examples

Hidden-layer design: a model developer compares ELU with ReLU or another activation under a consistent validation procedure.
Gradient analysis: students trace how positive and negative inputs produce different local derivatives during backpropagation.
Deployment evaluation: an engineering team measures whether an ELU-based model's predictive performance justifies its runtime characteristics on the intended hardware.

Think of it this way

Think of a signal rule that passes positive values straight through but gradually compresses negative values toward a lower floor. The floor is approached rather than crossed for positive alpha. The analogy is limited because the transformation is mathematical, and compressing a signal does not by itself establish what that signal means to the task.

Worked example

Take alpha = 1 and inputs minus 2, minus 1, 0, and 2. ELU(minus 2) is exp(minus 2) minus 1, approximately minus 0.8647. ELU(minus 1) is approximately minus 0.6321. ELU(0) is zero, and ELU(2) is 2. At x = minus 1, the derivative is exp(minus 1), approximately 0.3679; at x = 2, the derivative is one.
Suppose the gradient arriving from later computations is 3. At x = minus 1, the chain rule multiplies it by 0.3679, giving an input gradient of approximately 1.1036. At x = 2, the input gradient is 3. These are local gradient calculations, not complete weight updates. If alpha were 2, the negative output at minus 1 would be approximately minus 1.2642 and its derivative approximately 0.7358. The positive branch would remain unchanged, but the two one-sided derivatives at zero would no longer match.

Illustrative case study

In a fictional classification experiment, a team replaces ReLU with ELU while leaving the rest of its comparison protocol fixed. Training curves change, but the team evaluates held-out accuracy and runtime before judging the change. It records alpha and checks whether an exported model reproduces the same activation behavior. The case illustrates controlled model comparison; it does not claim that ELU necessarily improves accuracy or reduces training time.

Common misconceptions and mistakes

ELU is not purely exponential because its positive branch is linear. Its negative outputs are not probabilities or necessarily errors. Although its negative derivative is nonzero for finite negative inputs in exact arithmetic, it can become very small in the saturated region. ELU therefore does not eliminate every vanishing-gradient issue. Smoothness at zero depends on alpha for the original formulation. Allowing negative outputs does not guarantee zero-mean activations, and an activation choice alone does not establish a self-normalizing network.

Business use

Treat ELU as an architectural choice to evaluate, not a promised performance upgrade. Record the parameter and the precise function used by the implementation. Compare models using the same data partitions and relevant metrics, and inspect numerical behavior when inputs have extreme scales. Measure inference cost where deployment constraints matter. If changing the activation also requires different initialization or training settings, describe the complete configuration so the comparison remains interpretable and reproducible.

Check your understanding

With alpha = 1, why does an incoming gradient of 3 become approximately 1.1036 at input minus 1?

Answer: Backpropagation multiplies the incoming gradient by ELU's local derivative, exp(minus 1), approximately 0.3679.

Related terms

Learn next

Evaluate ELU and its derivative for positive, near-zero, and strongly negative inputs. Compare the graph with ReLU and leaky ReLU. Then inspect how alpha affects continuity and differentiability before comparing trained models under a controlled evaluation design.

Sources

Clevert, Unterthiner, and Hochreiter, original ELU paper: https://arxiv.org/abs/1511.07289 PyTorch, ELU implementation definition: https://docs.pytorch.org/docs/stable/generated/torch.nn.ELU.html Barron, original CELU comparison and parametrization: https://arxiv.org/abs/1704.07483

Previous: ELT Next: Embarrassingly parallel