Is token-mixing always self-attention?

No. Self-attention is a common mechanism, but other architectures mix positions through different operations, including MLPs.

Does mixing change the number of tokens?

Not necessarily. A layer can preserve the number of positions while changing the representation stored at each one.

Can a causal model mix information from future tokens?

Its causal attention mask prevents that access at the relevant position. Other architectures or masks may have different permitted connections.

AI Glossary / Generative AI and language / Intermediate

Token-mixing

Definition

Token-mixing is the exchange or combination of information across token positions inside a neural network. In a standard Transformer block, attention performs this role by allowing the representation at one position to incorporate information from other permitted positions. Token-mixing describes a computational function, not the act of shuffling written words or changing how a tokenizer splits text.

What it means

A sequence model holds a numerical representation at each token position. If every operation processed each position in isolation, the representation of a word could not take account of surrounding words. Token-mixing provides a route for that contextual information to enter.

The term contrasts with operations that transform features within a position. In the usual Transformer, a position-wise feed-forward network processes each token representation separately, while self-attention connects positions.

Attention is not the only possible token-mixing mechanism. For example, MLP-Mixer uses multilayer perceptrons across image-patch positions. That broader usage matters when comparing architectures: “token-mixing” names the cross-position role, not one mandatory mathematical implementation.

How it works

In self-attention, a token's query is compared with keys at permitted positions. The resulting scores are normalized to weights, which combine value vectors into an attention output. Output projections and residual connections integrate that information into the representation used by subsequent layers.

A causal language model restricts access to future positions, while other model designs may permit different attention patterns. Information exchange therefore depends on the architecture and mask, not simply on whether all tokens are present in memory.

In an MLP-based mixer, learned transformations can instead operate across the token dimension of a token-by-feature table. The distinction between positions and features helps explain why changing the cross-position operation can produce a substantially different model even when the overall input and output shapes are similar.

Three practical examples

1. A language model can represent “bank” differently when earlier permitted context refers to a river instead of a savings account, because contextual information can reach that position.

2. An image model combines information across patch tokens to recognize a pattern that extends beyond one small image region.

3. A model engineer inspects a causal mask to ensure that a position cannot obtain information from later answer tokens during next-token training.

Think of it this way

Think of each person in a study group keeping a private page of notes. Token-mixing is the stage when selected information from other pages is incorporated into a person's notes. A separate stage then reorganizes that person's own page.

The analogy is bounded: token representations are learned vectors, not readable notebooks, and information is combined through numerical operations. A large mixing weight does not by itself prove that a source token supplied a complete human-interpretable reason for the final answer.

Worked example

Consider a toy attention calculation with three one-dimensional value representations: 2, 6, and 10. At a selected position, suppose the allowed attention weights are 0.25, 0.50, and 0.25. The mixed value is 0.25 × 2 + 0.50 × 6 + 0.25 × 10 = 6.

If the second value changes from 6 to 8 while the illustrative weights stay fixed, the mixed value becomes 7. Information from that second position has changed the selected position's result.

This isolates the cross-position combination. Real attention computes its weights from representations, so changing an input can change weights as well as values. Real blocks also use multidimensional vectors, projections, multiple heads, normalization, and residual paths. The scalar example is not a complete Transformer block and does not claim that model features correspond to simple visible quantities.

Illustrative case study

Fictional case: Valley Notes builds a document classifier and compares two experimental architectures. One processes token representations independently before pooling; the other includes an explicit contextual mixing stage.

The team creates examples where the same key word appears in different surrounding phrases and checks whether the models distinguish the intended categories. It records performance and resource use on held-out documents rather than choosing an architecture because its diagram looks more sophisticated.

This fictional test illustrates why cross-position information can matter. It does not guarantee that adding a mixing layer improves a particular dataset or that every observed gain has the same explanation.

Common misconceptions and mistakes

Do not confuse token-mixing with tokenization. Tokenization defines the units supplied to the model; mixing operates on representations within the network.

Do not assume mixing means random reordering. Positions can remain in the same order while their representations incorporate contextual information.

Avoid describing every neural-network component as equally responsible for cross-position exchange. In a standard Transformer block, the position-wise feed-forward operation has a different role from attention. Also avoid assuming that a mixing mechanism has unrestricted access: causal and other masks can limit which positions communicate.

Business use

Understanding token-mixing helps teams ask better questions about model architecture, context use, and computational cost. A claim that a model has a long input window does not by itself establish that it uses every relevant part effectively.

When comparing systems for document or sequence tasks, include examples that require combining information across positions. Measure whether the system solves those tasks and at what resource cost. Architectural descriptions explain possible mechanisms; operational evaluation establishes whether the chosen implementation meets the actual need.

Check your understanding

A layer applies the same feed-forward function independently to every token vector. Does that operation alone exchange information between positions?

Answer: No. It transforms each vector separately. The vectors may already contain context from an earlier mixing operation, but the independent operation itself does not read another position's vector.

Related terms

Self-attention
Residual stream
Feed-forward network
Tokenization
Transformer block
causal mask
channel mixing
MLP-Mixer
contextual representation

Learn next

Learn self-attention next to see one concrete mixing mechanism. Then compare position-wise feed-forward layers and residual connections, tracking which operation reads other positions and which only transforms information already present at the current position.

Self-attention

Sources

Jurafsky and Martin, Speech and Language Processing, Transformers and Pretraining chapter: https://web.stanford.edu/~jurafsky/slp3/7.pdf

Tolstikhin and colleagues, MLP-Mixer: An all-MLP Architecture for Vision: https://arxiv.org/pdf/2105.01601

The accessible primary chapter's token-mixing discussion and the original Mixer architecture were read. The supplied full-book file exceeded retrieval limits; the chapter supplies the relevant text. The scalar calculation and Valley Notes case are original illustrations.

Previous: Token Next: Tokenization