What does LSTM stand for?

Long short-term memory: short-term information kept for a long time, which is the design's whole point.

What are the other gates?

The input gate controls what enters memory and the output gate controls what is read from it.

Are LSTMs still used?

Yes in many sequence applications, though transformers with attention now dominate language tasks.

Can the gates be visualized?

Yes; plotting gate values over a sequence shows what the network chose to keep, a rough but real window into its memory policy.

AI Glossary / AI and machine learning / Intermediate

Forget gate

Definition

A forget gate is the mechanism inside an LSTM network that decides what to erase from its memory: how much of each remembered thing to keep at every step.

What it means

Recurrent networks read sequences one step at a time, carrying a memory forward. Plain versions forget badly: by the end of a long document, the beginning has faded to nothing, which is why they lose track of early context.

LSTMs, long short-term memory networks, solve this with gates, little learned valves on the memory. The forget gate is the one that decides, at each step, what proportion of each stored value survives.

Because the gate is learned, the network discovers what deserves remembering: it can keep the invoice number from paragraph one fully open while flushing the pleasantries, all without anyone programming the rule.

The gate is a number between zero and one per memory slot, computed from the current input and the recent past: near one means keep, near zero means flush, and everything between is emphasis.

For a business, forget gates are the reason sequence models can handle long histories at all: customer timelines, machine logs and documents all need selective memory, and selective memory is what the gate is.

The same gating idea now appears across architectures, not only in LSTMs: learned valves on information flow turned out to be a broadly useful invention, and the forget gate is the original example.

How it works

Maintain a memory vector that travels along the sequence, carrying distilled information from every past step.

At each step, compute the forget gate from the new input and the recent state: one keep-or-flush number per memory slot, learned during training.

Multiply the memory by the gate values, so slots with low gate values fade and slots with high values persist.

Add new information through companion gates: an input gate decides what enters, an output gate decides what is read out.

Train the whole arrangement end-to-end on the task, so the gates learn which memories the task rewards keeping.

Three practical examples

Document processing: keeping the parties and dates from page one available while reading page twenty.
Machine monitoring: retaining the context of an earlier warning while tracking hours of subsequent readings.
Conversation systems: remembering the customer's stated constraint across a long dialogue.

Think of it this way

Think of a skilled minute-taker in a long meeting. They cannot write everything, so they constantly decide: this commitment goes in the record, this anecdote does not. The forget gate is that editorial instinct, learned from what past meetings turned out to matter.

The limit: the minute-taker's judgment was trained on past meetings. A wholly new kind of meeting may flush exactly what should have been kept, silently.

Worked example

keeping the constraint across a dialogue

Suppose a support bot's customer says in their first message that they cannot receive deliveries on Fridays, then discusses products for thirty messages. The figures are illustrative.

A plain recurrent network has forgotten the constraint by message thirty-one; the LSTM's forget gate has kept the relevant memory slot open.

When the bot proposes delivery dates, the Friday constraint is still active, and no Friday is offered.

The training data never contained a rule about Fridays; the gate learned that early stated constraints predict later needs.

Illustrative case study

A fictional logistics firm replaced a plain recurrent forecaster with an LSTM and saw its biggest gains on exactly one pattern: demand surges echoing an event from many weeks earlier, which the old model always forgot.

The forget gate had learned to keep the echo-relevant memory alive far longer than the engineers expected.

The invented lesson: the value of learned memory is discovering which memories pay. You will be surprised by what the network refuses to forget.

Common misconceptions and mistakes

The forget gate deletes data. It scales memory values within the network's internal state; the underlying records are untouched.
Gates are programmed rules. They are learned parameters; their behavior emerges from training, which is why they can surprise their designers.
LSTMs made attention mechanisms unnecessary. Attention later offered a different route to long memory, but gated recurrence was the breakthrough that made long sequences workable first.

Business use

When sequence length matters, ask whether the model uses gated recurrence or attention; both solve selective memory, differently.

Interpret gate behavior when debugging long-sequence failures: what the model forgets early is often where the error begins.

Do not assume longer memory is trained memory; the gate learns what the task rewards, so task design sets the memory policy.

In vendor conversations about long-sequence handling, the forget gate is a useful concrete question: how does the system decide what to keep over long histories, and can that behavior be inspected.

Check your understanding

What problem does the forget gate solve that plain recurrent networks suffer from?

Answer: It lets the network selectively preserve important early information across long sequences instead of fading everything at the same rate.

Related terms

Recurrent neural network
Sequence model
LSTM
attention mechanism
memory
gate

Learn next

AI Agents and Automation for Small Business explains how systems keep and use context over long interactions: https://aiforbusinessandlife.com/academy/ai-agents-automation-small-business. [C4]

Sources

[S1] Russell and Norvig, AIMA glossary; [S2] Google, Machine Learning Glossary; [S8] Jurafsky and Martin, Speech and Language Processing (3rd ed. draft). Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Foreshortening Next: Fork - join model