Can tanh produce negative numbers?

Yes. Negative inputs produce negative outputs, with tanh(−x) = −tanh(x).

Is tanh a probability function?

Its direct output can be negative, so it is not an event probability. A transformation into a probability range still requires appropriate modeling and validation.

Does replacing tanh with ReLU always help?

No. The activation's role, architecture and training setup determine the consequences of a replacement.

AI Glossary / Deep learning / Advanced

Tanh

Definition

Tanh, short for hyperbolic tangent, is a mathematical function commonly used as a neural-network activation. It maps finite real inputs to values strictly between −1 and 1. Its formula is tanh(x) = (exp(x) − exp(−x))/(exp(x) + exp(−x)). The function is smooth, increases continuously and passes through zero, making it useful when a model needs signed, bounded activations.

What it means

A neuron can produce a weighted sum with any magnitude. Tanh compresses that sum: small inputs remain relatively close to their original values, while large positive or negative inputs approach the respective boundaries.
The sign is preserved, allowing both positive and negative internal signals. However, a negative activation does not mean that the input is bad, and a value near one does not automatically mean high confidence. Those interpretations depend on the model.
Tanh is a fixed transformation rather than a learning algorithm. Training changes the parameters that feed into or follow it; the standard tanh formula itself has no adjustable parameters.

How it works

The function can also be expressed as 2 × sigmoid(2x) − 1. This relationship explains its similar S-shaped appearance while making clear that its range differs from sigmoid's.
Its derivative is 1 − tanh(x)². At zero, the derivative is one. As the output approaches either boundary, the derivative approaches zero. This is saturation: changes in an already extreme input produce progressively smaller changes in the output.
Tanh appears in recurrent-network designs as well as other architectures. For example, Keras distinguishes the LSTM activation, whose default is tanh, from the recurrent gate activation, whose default is sigmoid. Their roles should not be conflated. The full architecture, not the presence of tanh alone, determines how information and gradients travel.

Three practical examples

A hidden layer uses tanh to create positive and negative internal features. The team checks activation distributions to see whether most values have been pushed into saturated regions.
A model predicts a correction constrained to a chosen interval by scaling a tanh output. The interval must be compatible with the corrections the application actually needs.
A recurrent-model exercise compares candidate-state values with gate values. Students observe that a signed tanh output serves a different purpose from a sigmoid value used to modulate information flow.

Think of it this way

Imagine a gauge that moves freely near its middle but responds less as it approaches either end. Further pressure still changes its mathematical position, but by increasingly small amounts.
That picture helps explain saturation. It does not mean the gauge has judged an extreme signal unimportant. The compression follows the formula, regardless of whether the input represents useful evidence, an unusual event or an error in preprocessing.

Worked example

Evaluate tanh at inputs 0, 1 and 3. The outputs are approximately 0, 0.7616 and 0.9951. The corresponding derivatives are 1, 0.4200 and 0.009866.
For an illustrative local calculation, suppose the loss derivative arriving from the next operation is 0.2. Multiplying by the activation derivative gives input-score derivatives of 0.2, approximately 0.0840 and approximately 0.001973. This isolates tanh's contribution to the chain rule; actual weight derivatives also depend on the preceding calculation.
At input 3, the output is near its upper bound, yet the gradient contribution is much smaller than at zero. A bounded output therefore does not guarantee a strong learning signal.
Now consider a model output defined as five times tanh(z). At z = 1, it produces approximately 3.8080. Its mathematical range is between −5 and 5, so it cannot represent a target value of 7. No adjustment to the weights producing z can overcome that final output restriction. These calculations are teaching examples, not measured model performance.

Illustrative case study

Fictional Maple Lessons uses a small network to estimate adjustments to lesson-duration forecasts. The prototype's hidden tanh activations frequently sit near their boundaries, and learning progresses slowly.
A review finds that one input is supplied in seconds even though the preprocessing specification expected minutes. The team fixes the input transformation, retrains and compares results on separate evaluation data. It also checks whether the final bounded adjustment range excludes legitimate cases.
The diagnosis does not establish that tanh is unsuitable. It shows that activation behavior must be interpreted alongside units, scaling and output design. The team documents those choices so a future data-feed change does not silently recreate the mismatch.

Common misconceptions and mistakes

Tanh being symmetric around zero does not guarantee a zero average activation on every dataset. An asymmetric input distribution can produce a nonzero average output.
Tanh is not the ordinary tangent function. It has no periodic poles on the real line, and its output stays bounded.
A standard tanh layer is not a replacement for data standardization. It compresses values nonlinearly rather than simply centering and scaling their distribution.
Mathematical outputs approach but do not reach ±1 at finite real inputs. Numerical software can round to those endpoints. Conversely, smoothness does not prevent very small gradients or guarantee successful training.

Business use

The business question is whether the model's mathematical choices match the quantity being predicted. Signed internal features can be useful, but bounded final outputs create firm representational limits that should be documented.
Ask developers to explain input units, preprocessing, output scaling and the observed error patterns. If performance changes after a data update, inspect those assumptions before attributing the problem solely to model capacity.
When comparing tanh with another activation, evaluate the complete trained model and operational requirements. An attractive curve or a familiar default is not evidence of better accuracy, reliability or user outcomes.

Check your understanding

What is the derivative at zero?

Answer: One.

Why does tanh(3) provide a smaller local gradient than tanh(0)?

Answer: The curve is saturated near 3, where 1 − tanh(3)² is small.

Can five times tanh(z) predict 7?

Answer: No; seven lies outside its output range.

Related terms

Learn next

Plot tanh and its derivative together. Compare outputs before and after changing an input's units, then compensate with appropriate preprocessing. Finally, check whether a scaled tanh output can represent every target value relevant to a proposed application.

Sources

Keras, layer activation functions: hyperbolic tangent definition and related activations. https://keras.io/api/layers/activations/ Keras, LSTM layer: separate activation and recurrent-activation settings. https://keras.io/api/layers/recurrent_layers/lstm/ The derivative calculations, scaled-output example and fictional case are original teaching illustrations.

Previous: Tall matrix Next: Target network