How does GQA relate to multi-query attention?

Multi-query attention is the one-group endpoint, where all query heads share one key and one value head.

How does it relate to conventional multi-head attention?

If every query head has its own key-value group, the sharing pattern becomes conventional multi-head attention.

Will grouped sharing always preserve quality?

No. It changes representational capacity, so the number of groups and training procedure need evaluation rather than an unconditional assumption of equivalence.

AI Glossary / Generative AI and language / Intermediate

Grouped Query Attention

Definition

Grouped Query Attention, usually written grouped-query attention or GQA, is an attention design in which several query heads share a smaller set of key and value heads. It sits between conventional multi-head attention, where each query head has its own key and value heads, and multi-query attention, where all query heads share one pair. The design can reduce key-value cache storage and memory traffic during language-model decoding while retaining multiple query representations.

What it means

Attention heads provide different learned ways to compare a current token with available context. Storing separate keys and values for every head can become expensive when a model generates long sequences or serves many requests. GQA shares those stored representations within groups. The queries remain distinct, so heads in the same group can still assign different attention weights. Sharing is an architectural tradeoff, and its effect on model quality and runtime must be evaluated for the actual model and workload.

How it works

Choose a number of query heads and a smaller number of key-value groups, with a defined mapping from each query head to a group. Each query head compares its query with that group's keys and combines the corresponding values. The resulting head outputs are combined through the model's output projection. Efficient implementations store the shared keys and values once rather than unnecessarily duplicating them. Ainslie and colleagues also describe converting existing multi-head checkpoints by pooling key-value projections within groups and continuing training; conversion is not automatically behavior-preserving.

Three practical examples

Language-model serving: reduce the key-value storage associated with active sequences, potentially supporting more concurrent requests within a memory budget.
Architecture selection: compare models with different numbers of key-value groups to evaluate the balance between task quality and decoding efficiency.
Model adaptation research: investigate a documented conversion and further-training procedure instead of assuming that deleting key-value heads leaves a pretrained model unchanged.

Think of it this way

Think of several researchers sharing a reference collection while each asks a different question. The collection need not be duplicated for every researcher, but their queries can lead to different selections from it. The analogy captures sharing, not the full mathematics: keys and values are learned numerical representations, and changing their sharing pattern can affect what the model can represent.

Worked example

Consider a simplified decoder layer with 16 query heads, head dimension 64, and a cached sequence length of 1,024. Assume both keys and values store two bytes per element, with one sequence and no extra overhead. Conventional multi-head attention with 16 key-value heads uses 2 times 1,024 times 16 times 64 times 2 = 4,194,304 bytes, or 4 MiB, for that layer's key-value cache. The first factor of two counts keys and values; the last counts bytes per element.
With four key-value groups, the same calculation becomes 2 times 1,024 times 4 times 64 times 2 = 1,048,576 bytes, or 1 MiB. Four query heads share each group's key and value representations. The cache is one quarter the original size under these assumptions. Multi-query attention with one group would use 0.25 MiB. These are logical storage calculations for the specified layer, not total-model memory measurements. Model weights, other activations, batch size, number of layers, cache layout, and implementation overhead remain relevant. A fourfold cache reduction does not imply a fourfold end-to-end speedup, because other operations and bottlenecks still contribute.

Illustrative case study

A fictional inference team compares two otherwise carefully controlled model configurations, one using separate key-value heads and one using grouped sharing. It measures quality on its actual tasks and profiles both prompt processing and token-by-token generation. The team finds that the cache estimate helps explain memory usage, but real latency also depends on batch size and kernel selection. It records those conditions with the comparison and avoids presenting the arithmetic reduction as a universal performance guarantee.

Common misconceptions and mistakes

GQA does not mean averaging query heads or making every head in a group attend identically. The shared objects are keys and values. It does not remove the need for causal masking or reduce every attention operation in proportion to the number of groups. GQA and FlashAttention address different aspects of attention and can be compatible. A software implementation that physically repeats shared tensors may lose some expected memory benefit. Model architecture, checkpoint layout, and kernel support must agree.

Business use

Use GQA as one factor in capacity planning and model evaluation, especially where key-value cache memory is significant. Compare achieved task quality, latency, throughput, and supported concurrency under realistic sequence lengths. Check the number of query heads and key-value heads separately when estimating resources. Include the exact inference implementation in testing, because theoretical sharing and actual memory allocation can differ. Treat architectural changes to an existing checkpoint as model changes requiring appropriate validation.

Check your understanding

Four query heads share one key-value group. Must they produce identical outputs?

Answer: No. Their query representations can differ, producing different attention scores over the same keys and therefore different weighted combinations of the shared values.

Related terms

Multi-head attention
P-value
Attention head
Decoder
multi-query attention
query
key
value
key-value cache
memory bandwidth
FlashAttention

Learn next

Sketch 16 query heads mapped to four key-value groups and calculate the cache size under a stated precision and sequence length. Compare the one-group and 16-group endpoints. Study scaled dot-product attention next, then inspect implementation support and benchmark complete decoding workloads rather than relying only on the theoretical cache ratio.

Dot-product attention

Sources

Ainslie and colleagues, GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints: https://aclanthology.org/2023.emnlp-main.298.pdf PyTorch, scaled_dot_product_attention: https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html PyTorch development discussion, Adding GQA to SDPA: https://dev-discuss.pytorch.org/t/added-grouped-query-attention-to-scaled-dot-product-attention-api/2340

Previous: Grouped Dirichlet distribution Next: Groupthink Bias