Does an LSTM keep every past input?

No. It maintains learned numerical states that summarize selected information.

Is an LSTM the same as a GRU?

No. Both are gated recurrent architectures, but their gates and state updates differ.

Do its weights change after every incoming observation?

Not during ordinary inference. Its states change; weight updates require a training or adaptation procedure.

AI Glossary / Data science and analytics / Intermediate

Long Short-Term Memory

Definition

Long Short-Term Memory, usually abbreviated LSTM, is a type of recurrent neural network designed to learn relationships across sequences. Its memory cells and learned gates control how information is carried forward, updated, and exposed as output. LSTMs address difficulties that ordinary recurrent networks can encounter when learning dependencies across many steps, but they do not guarantee perfect recall or accurate predictions.

What it means

A sequence contains an order that matters: words in a sentence, successive sensor readings, or daily observations. An LSTM processes that order while maintaining internal numerical states. Earlier inputs can therefore influence later calculations without every earlier observation being presented again as a separate input at each step. These states are learned representations, not a readable archive of past records. Their usefulness depends on the training task, available data, architecture, and evaluation conditions.

How it works

A common modern LSTM cell combines the current input with its previous hidden state to calculate an input gate, forget gate, output gate, and candidate update. Gate values typically lie between zero and one. The cell state retains a gated portion of its previous value and adds a gated portion of the candidate. The hidden state exposes a transformed, gated version of the updated cell state. Training adjusts weights using prediction errors across sequences. During ordinary inference, states change while trained weights remain fixed. The original 1997 LSTM and later forget-gate developments should not be treated as identical historical designs.

Three practical examples

Equipment monitoring: an LSTM can use a sequence of sensor readings to predict the next reading or classify an operating pattern, subject to representative training data.
Text classification: a model can combine information across tokens to classify a message, rather than treating each token as an unrelated observation.
Demand forecasting: past observations and genuinely available explanatory variables can feed a forecast model that is compared with simpler seasonal baselines.

Think of it this way

Think of a running notebook whose update rules decide what to retain, add, and summarize at each step. The analogy helps explain controlled state updates. It breaks down if the notebook is imagined as a complete transcript: an LSTM stores numerical representations and can lose or distort information.

Worked example

Consider one scalar cell-state component in a simplified modern LSTM. Suppose its previous value is 0.8, its forget gate is 0.75, its input gate is 0.2, and its candidate update is 0.5. The new cell value is 0.75 times 0.8 plus 0.2 times 0.5, which equals 0.6 plus 0.1, or 0.7. If the output gate is 0.6, the corresponding hidden value is 0.6 times tanh(0.7), approximately 0.363.
These values demonstrate the update arithmetic; they do not show how a trained model obtained its gates. The 0.75 gate is not a 75% probability that a forecast is correct. It scales the previous cell component. If subsequent forget gates repeatedly stay below one, the contribution of that old component can diminish. Real cells perform these operations across many components using learned weights and inputs, and a separate output layer may convert hidden states into predictions.

Illustrative case study

In a fictional maintenance project, analysts build an LSTM using machine readings recorded in time order. Initial evaluation looks promising until they discover that preprocessing used statistics from the future test period. They rebuild preprocessing using training data only, evaluate on later periods, and compare with a straightforward forecast. They also reset states when switching between unrelated machines. This case illustrates evaluation discipline; it does not assert that an LSTM necessarily outperforms the simpler model.

Common misconceptions and mistakes

The word memory does not imply human understanding or permanent storage. An LSTM does not automatically discover every long-range relationship. More layers or units can increase computation and overfitting. Randomly mixing overlapping time windows across training and testing can leak information. Carrying hidden states between unrelated sequences can create unintended dependencies. A bidirectional LSTM uses both directions of its supplied sequence, so future observations must not be accidentally made available in a real-time forecasting task.

Business use

Consider an LSTM when sequence order is substantively relevant and sufficient representative data are available. Define the prediction horizon, inputs available at decision time, and how sequences begin and end. Evaluate chronological performance, robustness to missing readings, and operational latency. Compare against simpler methods and other sequence architectures on the same evidence. Document when states are initialized or retained. A useful model must support the operational decision, not merely fit historical sequences or produce an impressive training score.

Check your understanding

With previous cell value 0.8, forget gate 0.75, input gate 0.2, and candidate 0.5, what is the updated cell value?

Answer: It is 0.7. That is an internal state component, not a probability of correct prediction.

Related terms

Recurrent neural network
Gated recurrent unit
Vanishing gradient
Time Series Forecasting
hidden state
cell state
sequence modeling
backpropagation through time

Learn next

Trace the cell and hidden states through two small updates. Then distinguish training weights from inference states, study chronological validation, and compare LSTM and GRU performance with a simple baseline on a well-defined sequence task.

Sources

Hochreiter and Schmidhuber, original LSTM paper: https://people.idsia.ch/~juergen/lstm1997-2024head.pdf Gers, Schmidhuber, and Cummins, later forget-gate development: https://pubmed.ncbi.nlm.nih.gov/11032042/ PyTorch, modern LSTM equations and implementation: https://docs.pytorch.org/docs/2.14/generated/torch.nn.LSTM.html

Previous: Long-running transaction Next: Long tail