Does ReLU convert negative inputs to positive ones?

No. It replaces them with zero. That differs from taking an absolute value.

Is ReLU's positive output capped?

Standard ReLU has no upper cap. Named variants may behave differently, so check their definitions.

How is leaky ReLU different?

It retains a nonzero slope on the negative side. That changes both the output and the local gradient for negative inputs.

AI Glossary / Graphs and symbolic AI / Beginner

Rectified Linear Unit

Definition

A Rectified Linear Unit, usually abbreviated ReLU, is a neural-network activation function defined by ReLU(x) = max(0, x). It returns zero for a negative input and leaves a positive input unchanged. At zero, its output is zero. Applied between learned transformations, it helps a network represent relationships that a stack of purely linear transformations could not express.

What it means

A neural-network layer commonly forms weighted combinations of incoming values and adds biases. ReLU then transforms each resulting number separately. For example, inputs −3, 0, and 2.5 become outputs 0, 0, and 2.5.

The word “linear” can be misleading. ReLU is linear on either side of zero separately, but the complete function has a bend and is not a single linear mapping. That change in behavior matters: different inputs can activate different groups of units. ReLU itself has no learned slope or threshold in its standard form; learning changes the weights and biases that determine what reaches it.

How it works

For a unit with input vector x, weights w, and bias b, compute z = w·x + b. The activation is then a = max(0, z). In a layer with many units, apply this operation independently to each pre-activation value.

During backpropagation, the local derivative is one when z is positive and zero when z is negative. The function is not differentiable at exactly zero, so an implementation needs a convention there. Do not confuse that computational convention with a classical derivative that exists at the bend.

ReLU's simple operation is inexpensive, but training behavior depends on the whole model. A unit that stays negative across the training inputs can stop receiving useful gradient through this activation. Monitoring activation patterns can help distinguish healthy input-dependent zeros from persistently inactive units.

Three practical examples

Image classification: a hidden convolutional layer can apply ReLU to its feature responses before passing them to later layers.
Tabular prediction: a small feedforward network can use ReLU hidden units to model changes in response across input ranges.
Learning exercises: students can calculate a ReLU layer by hand and inspect how changing a bias changes which units become active.

Think of it this way

Think of a measuring device with a floor at zero: values above the floor retain their size, while values below it are displayed as zero. This captures the numerical transformation, but it does not mean negative evidence is universally unimportant. Learned transformations and other units determine how information is represented before and after that floor is applied.

Worked example

Consider a one-input hidden unit with z = 2x − 3 and activation a = ReLU(z). Let a simple output calculation be y = 4a + 1.

For x = 1, z = −1, so a = 0 and y = 1. For x = 2, z = 1, so a = 1 and y = 5. For x = 3, z = 3, so a = 3 and y = 13.

The activation changes regime at x = 1.5. Below that point, this particular output remains one; above it, y = 8x − 11. The model therefore has a flat region and a rising region instead of one straight line across the entire input range.

Away from the bend, the derivative of y with respect to x is zero in the inactive region and eight in the active region. These are exact calculations for the stated toy model, not evidence about a trained application. Additional hidden units can create other bends, and a different output layer can combine activations with negative as well as positive weights.

Illustrative case study

Fictional case: Bracken Analytics trains a small demonstration network on synthetic machine readings. Its team uses ReLU in the hidden layer and plots each unit's activation across the training data.

Some units produce zero for only part of the input range, which is expected. Others remain zero throughout the observed training examples after a large update. The team investigates learning rate, initialization, and input scaling, then compares revised training runs using the same validation procedure.

The case illustrates diagnosis rather than a claim that every inactive unit must be repaired or that ReLU guarantees better predictions. The team evaluates the complete model rather than treating an activation choice as a performance result.

Common misconceptions and mistakes

ReLU is not a probability function. Its outputs can exceed one, and the activations in a layer need not sum to one.

It also does not make the final network output nonnegative unless the final architecture imposes that restriction. A subsequent weighted sum can produce negative values.

Zero activations are not automatically a fault: input-dependent inactivity is part of the function. Persistent inactivity across relevant inputs is a different concern. Finally, ReLU does not remove every difficulty associated with gradients or optimization. Architecture, initialization, data, and training settings still affect whether the model learns useful behavior.

Business use

Understanding ReLU helps teams interpret neural-network designs and diagnose training problems. It is a component choice, not a business capability on its own. A proposal that merely names ReLU has not established accuracy, reliability, or suitability for a task.

When comparing models, track validation performance, training stability, and operational requirements. Check whether the output layer matches the target being predicted. For example, a quantity that can be negative should not accidentally be forced through a final standard ReLU without a justified reason.

Check your understanding

A standard ReLU receives −4 and 6. Are its outputs 4 and 6?

Answer: No. Its outputs are zero and six. ReLU clamps the negative value to zero instead of changing its sign.

Related terms

Activation function
Backpropagation
Gradient
Bias
artificial neuron
leaky ReLU
feedforward network
piecewise linear function

Learn next

plot the worked unit while changing its bias from −3 to −1. Identify where its bend moves, then combine two units to see how several simple activations form a more flexible function.

Bias

Sources

Primary sources: PyTorch documents the standard element-wise function: https://docs.pytorch.org/docs/stable/generated/torch.nn.ReLU.html . Stanford's CS231n course notes discuss neural activations and inactive ReLU units: https://cs231n.github.io/neural-networks-1/ . This entry explains standard ReLU rather than every named variant. The numerical model and fictional case are original.

Previous: Rectified Gaussian distribution Next: Recurrence period density entropy