AI Glossary / Deep learning / Advanced

Softsign Function

Definition

The Softsign function is a neural-network activation defined by softsign(x) = x/(1 + |x|), where |x| means the absolute value of x. It compresses any finite real input into the interval between −1 and 1, retaining its sign. It is continuous and differentiable, including at zero, but is not a probability function.

What it means

A network can generate large positive or negative internal scores. Softsign converts them into bounded responses while allowing small scores near zero to pass through with relatively little compression. Larger magnitudes are compressed more strongly.
The operation works separately on each value. It neither compares competing classes nor forces several outputs to sum to one. Its negative and positive outputs may be useful as internal features or as part of a deliberately scaled prediction, but their meaning comes from the surrounding model.
Softsign is a mathematical transformation, not a method for deciding whether training data are trustworthy or predictions are accurate.

How it works

For a positive input, divide x by 1 + x. For a negative input, divide x by 1 − x. At zero, the result is zero. The denominator is always positive for real inputs, so the output has the same sign as the input.
Differentiating gives a slope of 1/(1 + |x|)². At zero, both one-sided derivatives equal one. Far from zero, the slope becomes small: the function increasingly compresses changes in large inputs.
Unlike the discontinuous sign function, Softsign changes gradually. Unlike hard clipping, it does not become exactly flat outside a fixed interval in exact arithmetic. Nevertheless, very small gradients can still make learning difficult. Differentiability also should not be overstated as infinite smoothness: the second derivative has different one-sided limits at zero.

Three practical examples

• A researcher tests Softsign in a hidden layer of a model that categorizes educational resources. Signed, bounded internal features are available, but the final class-output construction remains a separate choice.
• A forecasting experiment scales an adjustment target to lie inside a chosen interval, then uses a scaled Softsign output. The team checks whether this restriction excludes real observations before adopting it.
• A classroom tool compares inputs of 1, 10 and 100. Their outputs move toward one without growing proportionally, illustrating why bounded activations can hide large differences in raw scores.

Think of it this way

Picture a flexible gauge that moves easily near its center but becomes progressively harder to push toward either end. The direction of movement remains visible, while extreme pushes produce increasingly small additional movement.
The gauge does not know whether a push is sensible or mistaken. Similarly, Softsign compresses unusual values without identifying their cause. An input error may become less conspicuous while still affecting the model.

Worked example

Suppose an illustrative hidden layer produces scores −3, −1, 0, 1 and 3. Applying Softsign gives −0.75, −0.50, 0, 0.50 and 0.75. For example, −3/(1 + 3) equals −0.75.
The corresponding slopes are 1/16, 1/4, 1, 1/4 and 1/16. A small change near zero therefore passes through more strongly than the same small change near either extreme in this list.
Now compare scores 9 and 99. Their outputs are 0.9 and 0.99. A difference of 90 in raw scores becomes only 0.09 after transformation. The inverse calculation, x = y/(1 − |y|), recovers 9 from 0.9 and 99 from 0.99 in exact arithmetic.
This also shows that rounding near the endpoints can discard substantial information about the original magnitude. An output of 0.99 is not automatically 99% confidence. These numbers illustrate the function rather than performance on a trained model.

Illustrative case study

Fictional Cedar Learning experiments with predicting adjustments to estimated lesson durations. Its prototype uses an output of ten times Softsign(z), allowing adjustments between −10 and 10 minutes.
During validation, the team discovers that several lessons need adjustments larger than ten minutes. The model repeatedly approaches its output boundary but cannot represent the required corrections. More training does not remove that architectural restriction.
The team revisits the target definition and output range, then compares a revised model with an unrestricted baseline. It also checks residual errors by lesson type. The lesson is that bounded outputs can encode a useful constraint only when that constraint matches the actual task.

Common misconceptions and mistakes

Softsign is not softmax, sigmoid or the sign function. Their names sound related, but their ranges and purposes differ.
An output centered around zero does not guarantee that the average output across a dataset is zero. An asymmetric input distribution can produce an asymmetric output distribution.
Slower saturation than tanh at large magnitudes does not guarantee better training or eliminate vanishing gradients. Network depth, weights, losses and input scale still matter.
A bounded activation also does not make the whole network robust to outliers. Other pathways, learned parameters and downstream decisions can remain sensitive.

Business use

For teams reviewing a model design, Softsign raises a practical question: what should the output be allowed to represent? If a model predicts an adjustment, signed values may make sense. If it predicts an event probability, a direct Softsign output is unsuitable because it can be negative.
Document any scaling around the activation and check the resulting range against real cases. Compare held-out errors and boundary behavior rather than choosing an activation solely because its formula is simple. Monitor changes in input units, since they can shift many values into the compressed parts of the curve.

People also ask

Can Softsign return exactly one mathematically?

Not for a finite real input; one is approached as the input grows without bound. Computer rounding can behave differently.

Does Softsign have a derivative at zero?

Yes. The derivative there is one, even though the formula contains an absolute value.

Does it learn a threshold?

The standard function has no learned parameter. Surrounding layers can learn weights and biases.

Check your understanding

What is softsign(4)?

Answer: It is 4/5, or 0.8.

What input produces −0.8?

Answer: The inverse gives −0.8/(1 − 0.8) = −4.

Why can ten times Softsign(z) fail to predict a 15-minute adjustment?

Answer: Its mathematical output range excludes 15, regardless of how its preceding weights are trained.

Related terms

• Activation function
• Tanh
• Sigmoid
• Softmax
• Saturation (point operation)
• Vanishing gradient
• saturation
• output constraint
• feature scaling

Learn next

Plot Softsign and tanh on identical axes. Compare their values and slopes near zero and at large magnitudes. Then design a bounded prediction target and check whether its range covers the observations you actually need to model.

• Tanh

More in Deep learning

• Sigmoid layer - A sigmoid layer applies the logistic sigmoid function to each element of an input tensor, producing outputs between zero and one for finite inputs in exact...
• Sigmoid perceptron - A sigmoid perceptron is a neural unit that forms a weighted sum of its inputs, adds a bias, and applies a sigmoid activation to produce a smooth output.
• Softplus Function - The Softplus function is a smooth activation function defined by softplus(x) = ln(1 + exp(x)).
• Sparse Auto Encoder - A sparse autoencoder is a neural network trained to reconstruct its input while encouraging a sparse internal representation.
• Sparse Learning - Sparse learning is a family of machine-learning approaches that favor models or representations in which relatively few components are nonzero or active.
• Stacked RNNs - Stacked RNNs are recurrent neural networks arranged in multiple recurrent layers, with the output sequence from one layer feeding the next.

Sources

Keras, layer activation functions: the Softsign definition and related activation formulas. https://keras.io/api/layers/activations/ The derivative, inverse calculation, worked example and fictional case are original explanations derived from the stated formula.

Previous: Softplus Function Next: Software agent