Does cross-attention change the source values themselves?

The attention output updates the query-side stream in the usual block; the source representations may be updated elsewhere in the architecture.

Can one query attend to many source positions?

Yes. A weighted combination of multiple values is the standard operation.

Is cross-attention the same as retrieval-augmented generation?

No. Retrieval selects external information; cross-attention is one possible internal mechanism for conditioning on supplied information.

AI Glossary / Generative AI and language / Intermediate

Cross-attention

Definition

Cross-attention is an attention mechanism in which queries come from one set of representations and keys and values come from another. It lets one stream of information selectively draw on a different stream, such as a decoder consulting an encoded input sentence.

What it means

An AI system often needs to combine information that begins in different places. A translator generating an output word may need relevant parts of the source sentence. An image generator may need text features that describe the requested scene. Cross-attention provides a learned way to make that connection.

The distinction from self-attention concerns the origin of the inputs. Self-attention forms queries, keys and values from the same representation stream. Cross-attention uses one stream to ask what matters and another to supply the information. The streams can contain the same type of data, such as two text sequences, or different types, such as image and text features.

“Cross” does not mean that information necessarily flows both ways in one operation. A cross-attention block usually updates the query-side representation using the source-side values. A reverse direction requires its own computation.

How it works

The query stream is projected into query vectors. The source stream is projected into key and value vectors. Each query is compared with the available keys, often using scaled dot products. A softmax turns the resulting scores into weights over permitted source positions, and a weighted combination of the values produces a context vector.

With T query positions and S source positions, each attention head commonly forms a T × S score matrix. T and S need not match. Learned projections make the relevant dimensions compatible, and multiple heads can form different combinations before their outputs are mixed.

Masks can exclude source padding or other unavailable positions. In an encoder - decoder Transformer, the decoder's causal self-attention controls access to earlier output tokens, while cross-attention can consult the encoded source. These are different masks and information boundaries.

Cross-attention is a mechanism for combining supplied representations. It does not itself search the web, retrieve a document database or verify a fact. External retrieval may provide material that a model subsequently processes through attention.

Three practical examples

A machine-translation decoder attends to source-language representations while producing a target-language sequence.
A text-conditioned image model uses text representations to influence visual denoising features.
A multimodal system uses question representations to attend to encoded image regions when preparing an answer.

Think of it this way

Think of a writer consulting an indexed reference file. The current writing need acts like a query, the index suggests relevance, and selected material contributes to the next draft.

The analogy is limited because neural keys and values are learned vectors, not verified reference cards. Attention weights reflect a computation, not a guarantee that the consulted information supports the final claim or a complete explanation of why the model answered.

Worked example

Combining information from a separate source

Consider one invented query and three source positions. After scoring and normalization, suppose the attention weights are 0.2, 0.5 and 0.3. Let the corresponding two-dimensional value vectors be (10, 0), (0, 4) and (2, 2).

The resulting context vector is 0.2 × (10, 0) + 0.5 × (0, 4) + 0.3 × (2, 2) = (2.6, 2.6). It is a weighted combination, not a selection of only the largest-weight position.

If there are two query positions, the model calculates a separate weight row for each. Both can consult the same three source positions differently. These small numbers illustrate the operation; actual systems use learned projections, larger vectors and further transformations before producing an output.

Illustrative case study

A fictional translation service, Cedar Languages, batches source sentences of different lengths. Its decoder has a correct causal mask for generated text, but the source padding mask is missing from cross-attention. Shorter sentences therefore expose padding representations as possible attention targets.

The team checks the query and source dimensions separately, supplies the correct source mask and compares outputs across different batch compositions. It retains tests that place the same sentence beside both short and long neighbors.

The case illustrates an integration issue: correctly masking one attention block does not establish that every information path in the model is masked appropriately.

Common misconceptions and mistakes

Cross-attention requires different modalities. Two text streams are sufficient.
Query and source sequences must have equal lengths. The score matrix can be rectangular.
It is ordinary self-attention with a different name. The origins of the query and source representations differ.
A high attention weight proves causal importance or factual support. Such interpretation requires additional evidence.
Every modern language model has a separate cross-attention layer. Some architectures condition on concatenated context through self-attention instead.

Business use

The mechanism matters when evaluating how a product combines documents, images, audio or other context. Ask where the conditioning information enters, what can be omitted through truncation, and whether the model was trained to use that information effectively.

For long source and query sequences, attention computation and memory can become substantial. Architecture choices, efficient implementations and source selection influence cost. Simply increasing the amount of supplied context does not ensure better answers if important material is absent, poorly represented or ignored.

Check your understanding

A layer receives 12 query positions and 30 source positions. What is the usual attention-score shape for one head before batching?

Answer: 12 × 30. Each query has a score for each source position; the output retains one resulting context vector per query position.

Related terms

Learn next

Draw two representation streams and label which supplies queries and which supplies keys and values. Trace one weighted sum by hand, then identify the masks needed for each stream. Compare this with a model that concatenates both inputs and uses self-attention.

Sources

Sources: Vaswani and colleagues, Attention Is All You Need, encoder–decoder attention: https://arxiv.org/abs/1706.03762 Keras, MultiHeadAttention query and source interfaces: https://keras.io/api/layers/attention_layers/multi_head_attention/ Rombach and colleagues, High-Resolution Image Synthesis with Latent Diffusion Models: https://arxiv.org/abs/2112.10752 Jain and Wallace, Attention is not Explanation: https://aclanthology.org/N19-1357/

Previous: Cromwell's rule Next: Cross-brackets