Are Swish and SiLU identical?

The common fixed Swish with beta equal to one is SiLU; the broader parameterized formula allows other beta values.

Does Swish keep negative values?

Yes, finite negative inputs produce negative outputs in the standard fixed form.

Is a non-monotonic activation necessarily a problem?

No. Suitability depends on the full model and evidence from evaluation, not monotonicity alone.

Swish Function

Definition

The Swish function is a neural-network activation that multiplies an input by a sigmoid gate. Its parameterized form is Swish(x) = x × sigmoid(beta × x). The common beta-equals-one version is also called SiLU. It produces a smooth curve that can retain small negative outputs, unlike ReLU, which sets negative inputs to zero.

What it means

An activation changes how a neuron's weighted input contributes to later computation. Swish uses the input itself to determine the strength of its gate, which is why it is described as self-gated.
The gate is a number between zero and one, but the complete output is not restricted to that interval. Multiplication by x means that outputs can be negative or greater than one. Consequently, Swish should not be interpreted as an event probability.
The parameterized family and the fixed version should be distinguished when describing a model. Saying “Swish” without specifying the implementation may leave an important detail unclear.

How it works

First compute sigmoid(beta × x), then multiply the result by x. With beta equal to one, large positive inputs mostly pass through, while strongly negative inputs approach zero from below. Moderate negative inputs produce a shallow dip before the curve rises through the origin.
For the fixed version, let s = sigmoid(x). Its derivative is s + x × s × (1 − s). At zero, the slope is 0.5. The derivative can be negative on part of the negative-input range, explaining why the function is not monotonic.
The original Swish research explored the function through activation search and reported improvements in particular experiments. Those results are evidence for the tested settings, not a guarantee that replacing another activation improves every model.

Three practical examples

A developer compares Swish and ReLU inside an image classifier for educational diagrams. The useful question is whether held-out classification performance improves enough to justify any change in computation.
A model reviewer sees “SiLU” in a configuration and checks that the implementation is x × sigmoid(x). This avoids assuming that a separate adjustable beta parameter is present.
A classroom demonstration plots the activation alongside its sigmoid gate. Students see that the gate remains between zero and one while the resulting activation can lie outside that range.

Think of it this way

Imagine a dimmer whose setting is controlled by the signal passing through it. Small and large signals receive different amounts of attenuation, and negative signals retain their sign.
The analogy explains multiplication by a gate, but not all the curve's behavior. In particular, Swish's negative dip means that increasing a negative input can sometimes make the output more negative. A plotted curve or numerical example reveals this more clearly than the dimmer image alone.

Worked example

Set beta to one and consider inputs −3, −1, 0, 1 and 3. Their sigmoid gates are approximately 0.0474, 0.2689, 0.5000, 0.7311 and 0.9526. Multiplying by the inputs gives Swish outputs approximately −0.1423, −0.2689, 0, 0.7311 and 2.8577.
Moving from −3 to −1 increases the input but decreases the output, from −0.1423 to −0.2689. This is a concrete demonstration of non-monotonic behavior. At the positive end, the output 2.8577 also shows why the result is not a probability.
At x = 1, the derivative is approximately 0.7311 + 1 × 0.7311 × 0.2689 = 0.9277. At x = −3, it is approximately 0.0474 − 3 × 0.0474 × 0.9526 = −0.0881.
These local slopes describe sensitivity of the activation, not overall model accuracy. The calculation uses selected inputs and a fixed parameter, rather than measurements from a trained network.

Illustrative case study

Fictional Rowan Learning experiments with an image model that sorts uploaded lesson illustrations. An engineer replaces hidden ReLU activations with Swish and observes a better result in one training run.
The team repeats the comparison under its established evaluation procedure and checks inference time on the device used by the application. It also examines whether the apparent improvement is concentrated in easy categories while diagrams with small text remain problematic.
The final decision considers repeated results, operational cost and the errors that matter to users. The team records the exact activation configuration so the experiment can be reproduced. The case illustrates how to evaluate an architectural change; it does not claim an actual performance improvement for Swish on this application.

Common misconceptions and mistakes

Swish is not sigmoid alone. Sigmoid supplies the gate; multiplying by the input changes the output range and shape.
Swish is also different from Softplus. For example, fixed Swish equals zero at input zero, whereas standard Softplus equals ln(2), approximately 0.6931.
Smoothness does not guarantee easy optimization or remove all small-gradient regions. The network's other operations and its training setup still influence learning.
The fixed SiLU function has no independently learned gate parameters. A parameterized Swish implementation may handle beta differently, so inspect the actual configuration before counting parameters or reproducing results.

Business use

An activation function is a component choice within a larger predictive system. Business teams usually benefit more from a controlled comparison than from selecting an activation by reputation.
Ask for the baseline, evaluation data, measured effect and deployment cost. Keep the output's business meaning separate from the activation's mathematical shape. A hidden feature value of 2.8 is not a percentage, score out of ten or confidence estimate unless the model explicitly defines such an interpretation elsewhere.
When migrating or exporting a model, verify that the destination supports the intended operation and that outputs remain consistent within the chosen numerical tolerance.

Check your understanding

What is fixed Swish at zero?

Answer: Zero, because the input multiplies the gate.

Why is Swish(3) greater than one?

Answer: The gate is approximately 0.9526, and multiplying it by three gives approximately 2.8577.

What does the −3 to −1 comparison demonstrate?

Answer: Increasing the input can decrease the output in part of the negative region.

Related terms

Sigmoid
Activation function
SiLU
ReLU
self-gating
derivative
Softplus
neural-network optimization

Learn next

Plot the fixed function and its derivative on the same input range. Compare them with ReLU, then vary beta in the parameterized formula. Record which properties change before considering a controlled model-level comparison.

Sources

Ramachandran, Zoph and Le, Searching for Activation Functions, 2017: parameterized Swish and its experimental context. https://arxiv.org/abs/1710.05941 Keras, layer activation functions: fixed Swish/SiLU formula and mathematical properties. https://keras.io/api/layers/activations/ The numerical example and fictional case are original illustrations.

Previous: SWIM Protocol Next: Switching Hidden Markov Model