Why is it called self-attention?

Because the representations being updated attend to positions within the same input sequence or stream.

Does self-attention always look ahead?

No. Causal masking prevents access to later positions in autoregressive settings, while bidirectional encoders may use context on both sides.

Are attention weights fixed after training?

The projection parameters are learned, but the attention weights are computed from the current input and can change from one example to another.

AI Glossary / Generative AI and language / Intermediate

Self-attention

Definition

Self-attention is a neural-network mechanism that updates representations of positions in a sequence by combining information from positions in that same sequence. The model computes weights describing how strongly different positions contribute to an output representation. It is a central component of transformer architectures. The word “self” refers to the sequence attending within itself, not to self-awareness or a model consciously examining its own thoughts.

What it means

A word's useful representation depends on context. “Bank” in a sentence about a river differs from “bank” in a sentence about an account. Self-attention provides a way for a position's representation to incorporate information from surrounding or preceding positions, depending on the allowed attention pattern.

The mechanism does not simply choose one relevant word. It typically forms a weighted combination of value vectors. Multiple attention heads can perform different learned combinations in parallel.

Self-attention is one part of a larger model. Feed-forward layers, normalization, residual connections, positional information, and training objectives also contribute to the system's behavior.

How it works

In standard scaled dot-product self-attention, the input representations are transformed into query, key, and value vectors. A query at one position is compared with permitted keys to obtain scores. The scores are scaled, masked where necessary, and normalized using softmax.

The resulting weights are used to combine the value vectors. Repeating this for each position produces updated representations. In self-attention, queries, keys, and values originate from the same sequence, although their learned transformations differ.

A causal mask prevents a position from using later positions in an autoregressive model. Other settings may allow bidirectional attention. Positional information is needed because the basic attention operation alone does not encode sequence order in the way an ordered language task requires.

Three practical examples

1. A text encoder uses sentence context to create different representations for the same word in different passages.

2. An autoregressive writing model uses masked self-attention over available context while predicting its next token.

3. A document-classification system combines information across token positions before producing a category, with its actual usefulness assessed on representative documents.

Think of it this way

Think of each position as preparing a summary by consulting a shared set of notes and assigning different weights to each note. Every position can prepare its own mixture from the same collection.

The analogy has limits. Queries, keys, and values are learned numerical vectors, not explicit questions and facts. A large attention weight does not by itself provide a complete causal explanation of a model's final answer or establish that the attended information is true.

Worked example

Consider one attention head with three scalar value entries: 10, 20, and 40. Suppose a particular position has already computed normalized attention weights of 0.5, 0.3, and 0.2 over them.

Its weighted value is 0.5 × 10 + 0.3 × 20 + 0.2 × 40 = 19. This is a mixture, not a selection of the single value 10 with the greatest weight.

Now suppose a causal mask disallows the third position and, for illustration, the remaining unnormalized relative weights stay proportional to 0.5 and 0.3. Renormalization gives 0.625 and 0.375, producing 13.75. Real attention starts from query-key scores and uses vector values; this example isolates weighting and masking rather than reproducing a complete transformer layer.

Illustrative case study

Fictional case: Linden Search evaluates a text encoder on internal policy questions. Some queries contain a familiar keyword but use it in an unusual context. The team investigates whether contextual representations improve retrieval over a simple keyword baseline.

It includes difficult near-matches in the evaluation rather than judging the model from a few attention visualizations. It also checks longer documents, where important context may fall outside the actual input window.

The team treats self-attention as a mechanism that can support contextual processing. Whether the resulting search is useful depends on the trained model, document preparation, retrieval design, and evaluation, not the mechanism's name alone.

Common misconceptions and mistakes

Self-attention is not the same as cross-attention. Cross-attention typically obtains queries from one representation stream and keys and values from another; self-attention uses one sequence as their source.

Attention weights are not automatically explanations of all model reasoning. Later computations and other heads can affect the final output.

Another mistake is assuming every position always attends to every other position. Masks and specialized attention patterns can restrict access. Also, standard dense attention involves pairwise position comparisons, which can be costly for long sequences; efficient implementations and alternative patterns change practical resource use.

Business use

Self-attention helps explain why transformer systems can combine information across a provided context. For business use, that capability may support language understanding, generation, or multimodal applications.

The practical questions are whether the model can access the necessary information and use it reliably. Evaluate realistic document lengths, references across sections, and ambiguous wording. Consider context-window limits and latency alongside accuracy. An advertised attention mechanism does not remove the need for retrieval, source verification, or a well-designed task workflow.

Check your understanding

A position gives its largest attention weight to one token. Does its output necessarily equal that token's value vector?

Answer: No. Standard attention combines all permitted value vectors according to their weights. The largest contributor may dominate, but the output is generally a weighted mixture.

Related terms

Learn next

Recompute the worked weighted sum with different weights that still total one. Then distinguish the input embeddings, learned projection matrices, input-dependent attention weights, and output vectors in a transformer diagram. This helps separate what training learns from what the model calculates for each new input.

Sources

Sources: Vaswani and colleagues, Attention Is All You Need, https://arxiv.org/pdf/1706.03762 — original paper sections on scaled dot-product attention, multiple heads, and masking read. Google Machine Learning Glossary, https://developers.google.com/machine-learning/glossary — primary self-attention explanation. Hugging Face Transformers glossary, https://huggingface.co/docs/transformers/main/en/glossary — supplied terminology context. Scalar calculations and the search case are original teaching illustrations rather than measurements of a particular trained head.

Previous: Selective Adherence Bias Next: Self-correction