Why divide by the square root of key dimension?

Under a common independent-component assumption, dot-product variance grows with dimension; scaling helps avoid excessively large softmax inputs.

Are keys and values always identical?

No. They often come from separate learned projections even when derived from the same input sequence.

Does the largest weight mean a source is true?

No. It describes relative weighting inside a computation, not a factual verification result.

AI Glossary / Generative AI and language / Intermediate

Dot-product attention

Definition

Dot-product attention is a mechanism that scores the compatibility of a query vector with key vectors using their dot products, converts those scores into weights, and combines the corresponding value vectors. In transformers, scaled dot-product attention divides scores by the square root of the key dimension before applying softmax. The output is a context-dependent mixture of values. The mechanism helps a model combine information across positions, but its weights are not direct guarantees of importance, truth, or a human-like focus.

What it means

A query represents what a position is looking for; keys provide features used to compare available information; values contain the information to be mixed. These names are functional roles for learned vectors, not literal database fields or natural-language questions. A larger query-key score usually gives that key's value more weight after softmax. Because scores compete through normalization, changing one key can affect all the weights for the query, even if the other raw scores stay unchanged.

How it works

Compute a score q dot k for every allowed key. In the scaled form, divide by square root of d_k, where d_k is the query-key dimension. Apply any attention mask, then softmax over the allowed key positions to obtain nonnegative weights summing to one before any training-time attention dropout. Multiply each value vector by its weight and add the results. Matrix notation is softmax(QK-transpose/square-root(d_k))V. Query and key dimensions must match, while value dimension can differ. Learned projections and multiple heads extend this basic operation.

Three practical examples

Language understanding: combine information from other token positions when building a representation of an ambiguous word.
Encoder-decoder generation: let a decoder query representations of source text while producing an output sequence.
Multimodal systems: connect queries from one representation to keys and values from another, when the architecture is trained to make those interactions useful.

Think of it this way

Think of a researcher assigning weights to several notes according to how well their headings match the current question, then combining the notes' contents. Keys resemble headings and values resemble contents. The analogy is limited: learned dot products do not independently verify evidence, and combining vectors is not the same as consciously reading or understanding notes.

Worked example

Use one query q = [1, 0] and two keys k1 = [1, 0] and k2 = [0, 1]. Their dot products are 1 and 0. With key dimension 2, scaled scores are 1/square-root(2), approximately 0.7071, and 0. Softmax gives weights approximately 0.6698 and 0.3302. Let the values be v1 = [2, 0] and v2 = [0, 4]. The attention output is 0.6698[2, 0] + 0.3302[0, 4], approximately [1.3395, 1.3210].
Without the scaling factor, scores 1 and 0 would yield weights approximately 0.7311 and 0.2689, producing [1.4621, 1.0758]. Both are dot-product attention variants, but they are numerically different. If a mask forbids the second key, the remaining allowed key receives weight 1 and the output becomes [2, 0]. This small example isolates attention mixing; it omits projections, multiple heads, residual connections, and other transformer components.

Illustrative case study

A fictional document-model team inspects attention weights and notices a token receiving a large weight. It initially assumes that token is the main reason for the answer. Further tests show that changing other layers or nearby context also changes the output. The team uses attention visualizations as a diagnostic aid, then tests behavior through controlled input changes and task evaluation. It does not present a bright attention cell as a complete explanation of the model's reasoning or as proof that the model verified a source.

Common misconceptions and mistakes

Dot-product attention is not necessarily self-attention: queries can come from a different sequence than keys and values. A dot product is not cosine similarity unless normalization makes that relationship appropriate. Scaling helps control score magnitude but does not calibrate the weights as probabilities of factual relevance. Causal masks restrict access to future positions; ordinary unmasked attention does not impose that restriction automatically. Attention alone also does not encode every aspect of position or sequence order without suitable surrounding design.

Business use

Use the concept to understand transformer architecture, context mixing, and computational tradeoffs. Standard dense self-attention compares all query-key pairs, so sequence length strongly affects work and memory, though implementations can reduce memory traffic and avoid storing the full matrix. Inspect masks, dimensions, and normalization when diagnosing an implementation. Evaluate the complete model on its actual task rather than assuming that adding attention guarantees useful long-range reasoning, accurate retrieval, or reliable attribution.

Check your understanding

Why does masking the second key change the output to [2, 0] in the worked example?

Answer: Only the first key remains eligible. Softmax over that single allowed score gives weight one, so the output is exactly its value vector.

Related terms

Learn next

Calculate two-key attention by hand, then change one key, one value, and the mask separately to see their different effects. Study self-attention and cross-attention next, followed by multiple heads and positional information. Keep attention-weight interpretation separate from evidence about the model's actual decisions.

Sources

Vaswani and colleagues, Attention Is All You Need: https://arxiv.org/abs/1706.03762 Original transformer paper: https://arxiv.org/pdf/1706.03762 Dive into Deep Learning, attention scoring functions: https://d2l.ai/chapter_attention-mechanisms-and-transformers/attention-scoring-functions.html

Previous: Dot product Next: Double deep Q-network