Must every head attend to different positions?

No. Different learned projections permit different behavior, but heads may overlap in what they emphasize.

Are head outputs averaged?

In the original Transformer formulation, they are concatenated and passed through an output projection. Other designs may combine them differently.

Can multi-head attention be causal?

Yes. A causal mask can prevent each position from accessing later positions while retaining multiple heads.

AI Glossary / Generative AI and language / Intermediate

Multi-head attention

Definition

Multi-head attention is an attention mechanism that performs several learned attention operations in parallel and combines their outputs. Each head uses its own projected representations to compare queries with keys and mix the associated values. Multiple heads give a model several ways to gather context within one layer, rather than forcing all relationships through a single attention-weighted mixture.

What it means

Attention lets a representation draw information from other representations. The query expresses what is being matched, the keys provide comparison features, and the values contain information to combine. A head is one such learned attention computation. Multiple heads can respond to different relationships or features, although they are not guaranteed to become neatly labeled specialists. Multi-head attention can be used for self-attention, where the representations come from the same sequence, or cross-attention, where queries come from one source and keys and values from another.

How it works

For each head, learned linear projections create query, key, and value vectors. In scaled dot-product attention, query-key dot products are divided by the square root of the key dimension. A mask can exclude disallowed connections, and softmax converts the remaining scores into attention weights. Those weights form a weighted sum of value vectors. The outputs from all heads are concatenated and passed through a learned output projection. Heads commonly use smaller dimensions than the full model representation, so adding heads does not simply mean duplicating the entire full-width operation many times. The chosen dimensions and implementation determine the actual computation and memory requirements.

Three practical examples

A translation decoder uses several cross-attention heads to combine information from the source sentence while producing target-language text.
A document encoder uses multi-head self-attention to construct contextual representations useful for classification or extraction.
A vision-language system uses attention between representations of text and image regions. Multiple heads can provide different learned comparisons, but the model still needs appropriate training and evaluation to connect those representations usefully.

Think of it this way

Think of several readers examining the same material using different questions, then combining their notes. One might track names while another tracks timing. This is only an analogy: attention heads are learned numerical operations, not people assigned explicit roles. Some heads may overlap, and their behavior may change across inputs. The final output also mixes their contributions, so a head's attention pattern is not by itself a complete explanation of the model's decision.

Worked example

Consider two heads attending to two positions. In head one, the attention weights are 0.8 and 0.2, and the corresponding scalar value projections are 2 and 6. Its output is 0.8 times 2 plus 0.2 times 6 = 2.8. In head two, the weights are 0.25 and 0.75, and its separately projected values are 10 and -2. Its output is 0.25 times 10 plus 0.75 times -2 = 1.0. Concatenating gives [2.8, 1.0]. If an illustrative output projection maps this pair to the scalar twice the first component minus the second, the final result is 4.6. The example supplies weights directly to focus on combination; a real attention layer calculates them from queries and keys. Separately, a model width of 512 split across eight equal-width heads gives 64 dimensions per head in a common configuration, not eight independent 512-dimensional copies.

Illustrative case study

Fictional case: Juniper Search experiments with an encoder containing multiple attention heads. A visualization suggests that one head often links product names with nearby attributes. The team treats this as a useful observation and tests whether altering that head affects a carefully chosen extraction task. It avoids labeling the head a permanent “product expert” from a few examples. The case illustrates how attention visualizations can generate hypotheses while task evaluations and interventions provide stronger evidence about a component's role.

Common misconceptions and mistakes

Multi-head attention is not the same as running several independent models and voting on their answers. The heads are components within a jointly trained network. More heads also do not guarantee better performance: changing head count at fixed model width changes per-head dimensions and can affect what each head can represent. A large attention weight does not prove causal importance to the final answer, because value projections, output mixing, residual connections, and later layers also matter. Finally, multi-head attention is not necessarily self-attention; the sources of queries, keys, and values determine that distinction.

Business use

For most business users, head count is an architectural detail rather than a buying criterion. Evaluate the full system on the required tasks and workload. For technical teams modifying attention, check dimension compatibility, masking, numerical stability, and memory behavior. Compare quality and runtime under the same context lengths and batch sizes. If attention diagrams are shown to users, explain what they visualize and avoid presenting them as definitive reasons for a prediction. An appealing diagram cannot replace evidence that the system performs the intended task reliably.

Check your understanding

Two attention heads assign identical weights to positions. Must they produce identical outputs?

Answer: No. Their value projections can differ, so applying the same weights to different projected values can produce different results.

Related terms

Attention head
P-value
Self-attention
Cross-attention
Transformer
query
key
value
scaled dot-product attention
output projection

Learn next

Work through a two-head calculation with small vectors and explicitly separate score calculation, softmax weights, value mixing, concatenation, and output projection. Then change only the value projection to see why an attention-weight picture does not capture the whole computation.

Sources

Vaswani and colleagues, Attention Is All You Need, Sections 3.2.1–3.2.3: https://arxiv.org/abs/1706.03762 Original paper PDF: https://arxiv.org/pdf/1706.03762

Previous: Multi expression programming Next: Multi-head self-attention