Is attention coverage the same as test coverage?

No. Here coverage concerns accumulated attention over source positions, not which software paths have been tested.

Must every source position end with coverage one?

No. The cited summarization mechanism penalizes repeated overlap rather than requiring a one-to-one translation of the entire source.

Is coverage identical to a pointer-generator network?

No. Copying and coverage are separate mechanisms; the cited paper combines them but also describes coverage as an addition to its baseline model.

AI Glossary / Natural language and speech / Advanced

Attention coverage mechanism

Definition

An attention coverage mechanism keeps a record of how much attention source positions have already received and uses that record during later decoding. In the cited summarization model, cumulative attention and an overlap penalty help discourage repeatedly focusing on the same source material.

What it means

A sequence-to-sequence summarizer can repeatedly attend to a passage and produce redundant sentences. Ordinary attention at the current step does not necessarily contain an explicit summary of all previous attention choices.

Coverage adds that history. In the mechanism described by See and colleagues, every source position has a cumulative value formed from earlier attention weights. The next attention calculation receives this information, and training penalizes overlap with previously attended positions.

Coverage is an internal computational measure, not a checklist proving that all important facts have been included. A position can receive attention without its meaning appearing correctly in the output. Conversely, good summarization deliberately omits much of the source, so complete uniform coverage is not the objective.

How it works

Let a_t be the attention distribution at decoding step t. Its entries are nonnegative and sum to one across source positions. Before that step, the coverage vector c_t is the sum of attention distributions from earlier steps. It begins as zeros.

The attention network uses c_t as an additional input when calculating the next distribution. A learned parameter controls how that history influences its scores; the mechanism is not a hard rule that forbids returning to a position.

The cited model also adds coverage loss: sum, over positions, of min(a_t[i], c_t[i]). This measures overlap between the current attention and prior coverage. It lies between zero and one for a normalized attention distribution.

The loss is weighted and combined with the token prediction loss. Training therefore balances producing the target text with avoiding redundant attention. Other coverage mechanisms can use different state updates or objectives.

Three practical examples

A summarization experiment tracks whether repeated output sentences coincide with repeated attention to the same passage.
A researcher compares token prediction loss alone with a version that also penalizes attention overlap.
A model review separates source-position coverage statistics from a human assessment of factual completeness.

Think of it this way

Think of a reader shading passages they have already consulted while drafting a summary. The shading reminds them where they have spent time and may help them avoid writing the same point twice.

The analogy is limited: attention weights are not human understanding, and a shaded passage does not prove its facts were accurately captured. Returning to a passage may also be necessary to express a name or relationship correctly.

Worked example

Suppose an invented source has three positions. At the first decoding step, attention is (0.6, 0.3, 0.1). Initial coverage is (0, 0, 0), so the overlap loss is zero.

Before the second step, coverage becomes (0.6, 0.3, 0.1). If second-step attention is (0.5, 0.2, 0.3), overlap loss is min(0.5, 0.6) + min(0.2, 0.3) + min(0.3, 0.1) = 0.8.

Afterward, cumulative coverage is (1.1, 0.5, 0.4). Its entries sum to two because two attention distributions have been accumulated. A value above one is valid; coverage is not a normalized probability distribution.

For comparison, second-step attention (0, 0, 1) would have overlap loss 0.1. That lower penalty does not prove it would generate a better word. The prediction loss and learned attention behavior must also be considered.

Illustrative case study

The fictional CedarSummary team builds summaries of internal project reports. A prototype often states the same milestone twice, and the team investigates an attention coverage mechanism.

Reviewers compare repeated passages, omitted facts and incorrect relationships separately. They discover that suppressing repetition alone does not reveal whether a crucial exception was left out. They also keep examples where a project name must legitimately recur.

The team treats coverage as one modeling intervention and evaluates the complete summaries against their sources. This invented case illustrates the difference between an internal anti-repetition signal and an editorial quality assessment; it reports no measured improvement or real deployment.

Common misconceptions and mistakes

Coverage guarantees factual completeness. It records attention history, not a verified inventory of included facts.
Coverage values must sum to one. The cumulative vector generally sums to the number of prior attention steps.
Previously attended positions are banned. The cited mechanism provides a learned signal and training penalty, allowing useful revisits.

Business use

Attention coverage is useful for understanding and experimenting with repetition control in source-conditioned generation. It gives engineers a concrete state variable and loss term to inspect rather than relying only on the final text.

Its value must be assessed alongside fidelity, omissions and necessary repetition. Record which coverage formulation is used, how its penalty is weighted and where attention is measured. Do not interpret a visually broad attention map as a quality certificate. A business summarization workflow still needs evaluation that checks whether decisions, figures and exceptions are represented correctly.

Check your understanding

After three normalized attention steps, what is the sum of the cumulative coverage vector, assuming no reset or masking changes the accounting?

Answer: Three. Each step contributes an attention distribution summing to one.

Related terms

Attention
Abstractive Summarization
Pointer-generator network
Decoder
sequence-to-sequence model
coverage loss
repetition
source fidelity

Learn next

Read pointer-generator networks and attention distributions. Calculate overlap penalties for several alternative attention vectors, then explain why the smallest overlap is not automatically the best next-token decision.

Attention

Sources

Abigail See, Peter J. Liu and Christopher D. Manning, Get To The Point: Summarization with Pointer-Generator Networks, ACL 2017: https://aclanthology.org/P17-1099/

Authoritative paper, Section 2.3: https://aclanthology.org/P17-1099.pdf

The cumulative-coverage definition, attention input and overlap-loss equations were reviewed in the original paper. The three-position calculations and fictional case are original. This entry scopes its detailed formula to that summarization mechanism.

Previous: Attention Next: Attention head