Is the add gate the same as the input gate?

In the textbook's LSTM description, yes: it selects candidate information entering cell memory.

Does “add” mean memory values always increase?

No. Candidate components can be negative, so their admitted contribution can reduce a component.

Does an open gate guarantee permanent memory?

No. Later updates and forget gates can alter or remove that contribution, and useful long-range retention still depends on training and data.

AI Glossary / Natural language and speech / Intermediate

Add gate

Definition

An add gate is the learned control in an LSTM network that regulates how much newly proposed information enters its memory state. In the supplied language-processing textbook, this is the gate commonly called the input gate. It scales candidate memory values before they are combined with the retained part of the previous memory.

What it means

An LSTM processes a sequence while maintaining numerical state. At each step, it proposes a possible memory update and determines how strongly each component should contribute. The add gate supplies those component-by-component weights.

The term here names a particular LSTM mechanism. It does not mean a generic arithmetic addition node in a computation graph. Although the resulting contribution is added to memory, the gate itself performs learned selection through multiplication. That distinction matters when reading diagrams or comparing descriptions that use different names for the same input gate.

How it works

Let x be the current input and h_previous the previous hidden state. A standard input gate computes i = sigmoid(W_i x + U_i h_previous + b_i). Its entries lie between zero and one for finite inputs. Separately, a candidate g = tanh(W_g x + U_g h_previous + b_g) proposes positive or negative memory values.

The contribution is i multiplied elementwise by g. With forget gate f and previous cell state c_previous, the update is c = f multiplied elementwise by c_previous + i multiplied elementwise by g. An output gate then controls what is exposed through the hidden state, commonly h = o multiplied elementwise by tanh(c).

Training learns the weights and biases from a task loss. The gate is continuous, not necessarily a binary switch, and no person has to write a rule specifying every useful memory update.

Three practical examples

1. Language modeling: An LSTM may use its gated state to carry information from earlier words while predicting a following word. The add gate controls new candidate contributions at each token.

2. Speech processing: As acoustic frames arrive, a recurrent model can update its internal representation incrementally. Gating provides one mechanism for balancing new frame information with retained context.

3. Sequential classification: A model reading a customer message can revise its numerical representation as later words qualify earlier ones. Gate values alone do not prove which human concept each component represents.

Think of it this way

Think of a mixing desk with a separate slider for each incoming channel. The sliders determine how strongly the new channels contribute to an existing mix. This resembles elementwise scaling, but an LSTM's channels are learned numerical features, not labeled facts. A slider analogy also does not capture training, the forget gate, or the nonlinear output transformation.

Worked example

Use an invented two-component LSTM update. Let the previous cell state be (0.8, -0.4), the forget gate (0.5, 0.75), the candidate (0.6, -0.8), and the add gate (0.25, 0.9).

The retained old memory is (0.4, -0.3). The admitted candidate contribution is (0.15, -0.72), obtained by multiplying corresponding components. Adding these gives the new cell state (0.55, -1.02).

Notice that the second memory component decreases, even though this is called the add gate: the admitted candidate is negative. Also, -1.02 is outside the range of tanh. That is allowed for the cell state because it is an accumulated sum; the candidate's bounded range does not bound the cell state to the same interval.

If the output gate is (0.8, 0.5), the resulting hidden state is approximately (0.400, -0.385), using tanh(0.55) and tanh(-1.02). These are supplied illustrative gate values, not outputs measured from a trained language model. Their arithmetic explains the update without claiming that either coordinate has a particular semantic meaning.

Illustrative case study

Fictional case: Pine Support maintains a small recurrent classifier for incoming messages. A developer labels a diagram's input gate “add gate,” while a reviewer assumes the label refers to the final plus sign in the cell update.

The team resolves the confusion by documenting the gate vector, candidate vector, elementwise product, and memory addition separately. It also adds a numerical example to its internal explanation. The fictional case illustrates clearer technical communication; it does not claim an accuracy gain or establish that LSTMs are the best architecture for this task.

Common misconceptions and mistakes

Do not treat a large gate value as confidence that an input is true. It is a learned scaling value, not a calibrated truth probability. Nor does a small value prove that the current word was ignored everywhere in the network.

The input and forget gates need not sum to one in a standard LSTM. Consequently, the update is not necessarily a convex average of old and new memory. Different recurrent architectures and LSTM variants alter these relationships, so equations should accompany gate terminology.

Business use

For teams maintaining recurrent systems, understanding the add gate helps explain model structure, inspect implementation compatibility, and diagnose mistaken assumptions about state updates. It is particularly useful when moving between educational diagrams and software interfaces that use “input gate.”

Business evaluation should still focus on task outcomes, error patterns, processing cost, and stability on representative sequences. Attractive gate visualizations cannot replace evidence that the complete model performs the intended job.

Check your understanding

A candidate component is -0.8 and its add-gate value is 0.25. What contribution enters the memory update?

Answer: -0.2. The gate scales the candidate; it does not change the negative value into a positive addition or replace the entire previous cell state.

Related terms

Forget gate
Output gate
Sigmoid
Recurrent neural network
LSTM
input gate
cell state
hidden state

Learn next

Trace a complete LSTM step, then compare its independent input and forget gates with the update mechanism of a gated recurrent unit. Keep cell state and exposed hidden state distinct throughout the calculation.

Gated recurrent unit

Sources

Source: Jurafsky and Martin, Speech and Language Processing, December 2020 draft, Section 9.3.1, read for the explicit “add gate” terminology and update equations: https://web.stanford.edu/~jurafsky/slp3/old_dec20/9.pdf . The current chapter and supplied full-book retrieval failed, so this accessible author-hosted edition was used. Bias terms are shown explicitly here. The numerical example is original; architecture-specific naming should be checked when adapting the entry.

Previous: Adaptive utility Next: Add-k