How does self-attention differ from cross-attention?

In self-attention, queries, keys, and values originate from the same sequence representation; cross-attention draws queries from a different source than keys and values.

Can a position attend to itself?

Usually yes, unless a specific mask or architecture prevents it.

Does a causal model ignore earlier text?

No. It can use permitted earlier positions; the restriction prevents access to later positions.

AI Glossary / Generative AI and language / Intermediate

Multi-head self-attention

Definition

Multi-head self-attention is an attention operation in which several heads build contextual representations by comparing positions within the same input sequence. Queries, keys, and values are derived from that sequence, using learned projections. Each head forms its own weighted combinations, and the head outputs are combined. “Self” identifies the source of the representations, while “multi-head” identifies the use of several attention computations.

What it means

A token's meaning often depends on other tokens. In “The battery beside the cables is damaged,” the representation of “damaged” should reflect the relevant object and surrounding wording. Self-attention provides a mechanism for exchanging information between positions, while multiple heads permit several learned patterns of exchange. It does not guarantee that a particular grammatical relationship will be represented in a particular head. The same principle can operate on image patches or other sequences of representations, so self-attention is not restricted to written words.

How it works

For a sequence representation X, each head produces queries, keys, and values through different learned projections of X. It compares each query with the permitted keys, scales the scores, applies any attention mask, and uses softmax to obtain weights. Those weights mix the value vectors to produce updated representations. Head outputs are concatenated and projected back into the model's working representation. In bidirectional self-attention, a position can usually access earlier and later non-padding positions. In causal self-attention, later positions are blocked so that predictions do not use unavailable future tokens. Positional information is also needed to help the model distinguish order; attention over content alone does not supply all the sequencing information a language task requires.

Three practical examples

A bidirectional document encoder uses the full visible sentence to create token representations for identifying names and dates.
A text-generating decoder uses causal self-attention over the prompt and already available generated tokens while producing the next token.
A vision Transformer applies self-attention to a sequence of image-patch representations. Multiple heads can relate different patches, with position information and training determining how those relationships are used.

Think of it this way

Think of participants in one meeting consulting one another's notes through several different discussion channels. Everyone draws from the same meeting's material, which captures the “self” distinction. If participants are only allowed to consult notes already written, the analogy resembles a causal mask. The limits are substantial: attention heads are numerical transformations, and no human conversation or deliberate reasoning is implied. Moreover, a high consultation weight is not a complete account of how the final answer was produced.

Worked example

Take a sequence of four positions in a causal self-attention layer. Position one may attend to position one; position two may attend to positions one and two; position three to the first three; and position four to all four. There are 1 + 2 + 3 + 4 = 10 permitted query-key connections per head, compared with sixteen in an unrestricted four-position pattern. With three heads, there are thirty permitted weights in this conceptual count. The implementation may still allocate a larger dense score array, so permitted connections are not automatically equal to memory saved. For one query at position three, suppose a head gives weights 0.2, 0.3, and 0.5 to scalar values 2, 4, and 8. Its output is 0.4 + 1.2 + 4 = 5.6. Position four contributes nothing because it is masked. The weights are illustrative, and other heads can use different projections and weights.

Illustrative case study

Fictional case: Maple Notes trains a small system to predict the next part of a maintenance note. An evaluation initially looks unusually strong because a masking error lets the model inspect text that would not yet be available during generation. The team corrects the causal mask and retests using the intended input conditions. The lower but valid score gives a more realistic basis for development. The case demonstrates why attention permissions are part of task correctness, not merely an internal optimization detail.

Common misconceptions and mistakes

Do not interpret self-attention as a model thinking about itself. It means the query, key, and value sources are the same sequence representation. A causal mask also differs from masking words for a reconstruction task: one controls information flow, while the other alters input content. Another mistake is assuming multiple heads eliminate the quadratic number of pairwise comparisons in ordinary dense attention. They provide multiple representation pathways but do not by themselves change that sequence-length scaling. Finally, attention weights are not a complete explanation of an output because the rest of the network transforms and combines information further.

Business use

The practical importance is contextual modeling with clear information boundaries. For an extraction application, access to both sides of a token can be useful. For next-token generation, access to future target content would invalidate the intended prediction setup. Technical teams should test padding and causal masks, sequence boundaries, and positional conventions. For long documents, evaluate whether truncation or chunking prevents necessary context from reaching the model. Business evaluation should focus on the completed task, since more heads or a larger context window does not alone establish better answers.

Check your understanding

An encoder can attend to every non-padding position, while a decoder can attend only to available earlier positions and itself. Can both use multi-head self-attention?

Answer: Yes. They differ in attention permissions, while both derive queries, keys, and values from their own sequence representations and use multiple heads.

Related terms

Multi-head attention
Cross-attention
Positional encoding
causal mask
bidirectional encoder
contextual representation
padding mask
Transformer decoder

Learn next

Draw the allowed attention connections for a five-position sequence under bidirectional and causal masks. Mark padding separately. Then explain which information is available when predicting the next token, rather than relying solely on the visual shape of the mask.

Sources

Vaswani and colleagues, Attention Is All You Need, Sections 3.1–3.3: https://arxiv.org/pdf/1706.03762 Devlin and colleagues, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding: https://aclanthology.org/N19-1423.pdf

Previous: Multi-head attention Next: Multi-label