Is the previous output the defining feedback signal?

In the Elman design, the defining recurrence carries the previous hidden state, not simply the output prediction.

Does it remember every earlier input exactly?

No. The state is a learned numerical summary with practical limitations.

Is an Elman network the same as an LSTM?

No. An LSTM is a more elaborate recurrent architecture with gates and a cell-state mechanism; the simple Elman recurrence does not include those components.

AI Glossary / Natural language and speech / Intermediate

Elman Networks

Definition

Elman networks are simple recurrent neural networks that feed the previous hidden state into the computation of the next hidden state. This recurrence allows the network's response to depend on earlier inputs as well as the current one. They are named after Jeffrey Elman and are a foundational architecture for modeling sequences.

What it means

A feedforward network presented with one isolated input has no built-in memory of the previous example. An Elman network carries a hidden-state representation forward through a sequence, giving each new computation access to a learned summary of prior processing.
In the original presentation, hidden activations are copied into context units for the next time step. Modern descriptions commonly express the same central idea directly as a recurrent hidden-state equation.
The state is not a literal transcript or a guaranteed record of everything seen. What it retains depends on its size, parameters, training objective, and the sequence. A recurrent connection creates a route for past information; it does not guarantee effective long-term memory.

How it works

A common formulation is h_t = f(Wx_t + Uh_(t−1) + b), where x_t is the current input, h_(t−1) is the previous hidden state, and f is a nonlinear activation. An output layer maps h_t to the required prediction.
The same parameter matrices are reused at successive time steps. An initial hidden state must be specified, and state-reset behavior defines where independent sequences begin.
Training adjusts parameters to improve the task objective. Modern simple RNN implementations often use backpropagation through time; this should not be confused with claiming that every detail of a current implementation reproduces Elman's original experimental training setup.

Three practical examples

A teaching model predicts the next item in a small symbolic sequence using the hidden state to represent earlier items.
A sequence-classification exercise reads a sentence one token at a time and uses its recurrent representation for a final label.
A researcher compares a simple recurrent baseline with a gated architecture to study how each handles dependencies across a sequence.

Think of it this way

Think of reading a story while carrying forward a compact set of notes that is updated after every sentence. The notes influence how the next sentence is interpreted. The analogy captures accumulated context, but an Elman network's state is a numerical representation, not explicit written notes with guaranteed factual recall.

Worked example

Consider an invented one-dimensional recurrence h_t = tanh(0.5x_t + 0.5h_(t−1)), with h_0 = 0 and no bias. This is a small modern simple-RNN illustration, not a reproduction of Elman's original experiment.
For sequence A = [1, 0], the first state is tanh(0.5), approximately 0.4621. The second is tanh(0.5 × 0 + 0.5 × 0.4621), approximately 0.2270.
For sequence B = [0, 0], both states remain zero. At the second step, both sequences present the same current input, zero, but their hidden states differ because their histories differ.
If an output layer reads the hidden state, it can therefore produce different predictions for the same current symbol in different contexts. This demonstrates the mechanism, not a trained capability. The chosen weights are arbitrary, no learning has occurred, and the example does not show that the network can preserve all relevant information over a long sequence.

Illustrative case study

Fictional case: SequenceLab builds an educational word-prediction demo. Its first baseline processes each token independently, so two identical current tokens always produce the same internal representation even when their preceding contexts differ.
The team adds a simple recurrent hidden state and visualizes how it changes across two short sentences. Reviewers also reset the state between unrelated examples so one demonstration does not unintentionally influence the next.
The lesson compares context-sensitive computation with context-free processing and then introduces memory limitations. This fictional case illustrates architecture behavior; it does not claim that the demo understands language or matches the performance of a modern production model.

Common misconceptions and mistakes

An Elman network is not a synonym for every recurrent architecture. LSTMs and other gated models add mechanisms beyond the basic hidden-state recurrence.
Do not confuse hidden-state feedback with output feedback, associated with a different recurrent design tradition.
Avoid assuming that a state can reliably preserve an unlimited history. Also distinguish the fixed copying of hidden values into original context units from learned weights that control how the previous state influences the next hidden computation.

Business use

The architecture is useful for understanding how sequential context entered neural language modeling and for building interpretable educational or experimental baselines. It gives teams a concrete vocabulary for hidden state, recurrence, parameter sharing, and sequence boundaries.
When reviewing a recurrent component, ask how its state is initialized, when it is reset, and how long dependencies were tested. State-management mistakes can affect predictions even when the model's mathematical equation is correct. Architecture choice should follow the task and evaluation evidence rather than historical familiarity alone.

Check your understanding

Two sequences end with the same input symbol. Must an Elman network have the same final hidden state for both?

Answer: No. Different preceding states can produce different current states even when the current input matches. That dependence on prior processing is the key recurrent mechanism.

Related terms

Recurrent neural network
hidden state
context units
sequence modeling
parameter sharing
backpropagation through time
LSTM
Jordan network

Learn next

Unroll a simple recurrence across three time steps and track which parameters are shared. Then compare its state update with an LSTM, focusing on how the additional gates change information retention rather than assuming recurrence alone guarantees memory.

Sources

Sources: Jeffrey L. Elman, Finding Structure in Time, original paper: https://web.stanford.edu/class/psych209a/ReadingsByDate/02_20/Elman90.pdf ; Jurafsky and Martin, Speech and Language Processing, RNNs and LSTMs: https://web.stanford.edu/~jurafsky/slp3/14.pdf Independently checked Elman’s original context-unit copying and learned upward connections, alongside the modern recurrent formulation.

Previous: Elliptical weighted average (EWA) Next: ELMo