AI Glossary / Natural language and speech / Advanced
Pointer-generator network
Definition
A pointer-generator network produces output by combining a vocabulary-based generation distribution with a distribution that copies tokens from the input. A learned mixing weight controls their contributions at each decoding step, allowing the model to both reuse source wording and generate other words.
What it means
A summarizer may need to reproduce an unusual person or product name exactly while using ordinary language to connect the facts. A fixed-vocabulary generator can struggle when the required source word is outside its vocabulary.
The pointer component assigns probability to source positions through attention. The generator component assigns probability to items in its predefined vocabulary. Combining them creates a distribution over both vocabulary words and source words available for copying.
In the cited model, this is a soft mixture, not necessarily a hard decision that labels each output word as either copied or generated. A word present in both places can receive probability mass from both components. Copying also does not guarantee that the surrounding statement is true.
How it works
The encoder represents the source, and the decoder calculates attention over source positions. A context vector summarizes the attended source information and helps produce a vocabulary distribution.
A learned generation probability p_gen lies between zero and one. For a word w, the final probability is p_gen × P_vocab(w) + (1 − p_gen) × the sum of attention weights at source positions containing w.
If w is outside the fixed vocabulary, its vocabulary contribution is zero. If it is absent from the source, its copy contribution is zero. If it appears several times in the source, those positions contribute to the same output word's probability.
Training rewards the target word under this combined distribution. The model can be paired with a separate coverage mechanism to discourage repetition, but copying and coverage solve different parts of the generation problem.
Three practical examples
- A summary model copies an uncommon organization name that appears in the source but not in its fixed vocabulary.
- A report generator produces common linking words through its vocabulary component while retaining a source-specific label through copying.
- An evaluation checks whether repeated occurrences of a source word are correctly aggregated into one output probability.
Think of it this way
Think of a writer with both a phrasebook and the source document open. They can select familiar wording from the phrasebook or reuse a name directly from the document.
The analogy shows two routes to a word. The network combines numerical probabilities rather than making a conscious editorial choice, and a copied word can still be inserted into an incorrect relationship or unsupported claim.
Worked example
Suppose the source tokens are Mira, joined, Mira and Labs. At one decoding step, attention weights are 0.20, 0.10, 0.30 and 0.40, which sum to one. Let p_gen = 0.25.
Assume the vocabulary distribution assigns probability 0.10 to Mira and 0.20 to Labs. Mira appears at two source positions, so its copy probability before mixing is 0.20 + 0.30 = 0.50. Its final probability is 0.25 × 0.10 + 0.75 × 0.50 = 0.40.
Labs has final probability 0.25 × 0.20 + 0.75 × 0.40 = 0.35. A vocabulary-only word with probability 0.08 receives 0.25 × 0.08 = 0.02 if absent from the source.
If Mira were outside the fixed vocabulary, its generation term would be zero, but it could still receive 0.75 × 0.50 = 0.375 through copying. These invented numbers demonstrate the mixture; they are not a trained model's output.
Illustrative case study
The fictional RowanBrief team builds summaries containing uncommon project names. Its fixed-vocabulary baseline sometimes replaces those names with an unknown-word marker.
The team tests a pointer-generator design and inspects whether source-specific names become available in the output distribution. Reviewers then find a separate problem: a correctly copied name can be associated with the wrong action when several people are discussed together.
They therefore evaluate exact copying and relational accuracy separately. They also examine repeated sentences using a distinct repetition analysis. This invented case illustrates why solving vocabulary access is useful but does not settle factual fidelity or completeness.
Common misconceptions and mistakes
- A copied word is automatically used correctly. Copying preserves a token's form, not the truth of the statement surrounding it.
- The pointer can copy a word that never appears in the input. Its distribution is tied to source occurrences.
- Copying and generating are always mutually exclusive. A word can receive probability from both components in the soft mixture.
Business use
Pointer-generator networks are relevant to source-grounded text generation, especially when source-specific words matter. They provide a concrete mechanism for making input tokens available beyond a fixed output vocabulary.
Evaluate names, numbers and relationships separately rather than using one overall fluency score. Check how repeated source words are aggregated and how tokenization defines an out-of-vocabulary item. Modern subword vocabularies change the practical vocabulary problem, so benefits should be tested in the actual system. Keep access to source tokens distinct from permission to disclose them: the architecture itself does not determine which source content belongs in a user's output.
People also ask
Can the model produce words not found in the source?
Yes, through its vocabulary generation component.
Can it output a word absent from both the source and fixed vocabulary?
Not as a single word in the cited word-level formulation; another tokenization or vocabulary design would change the available units.
Is a pointer-generator summary necessarily extractive?
No. It can combine copied words with generated wording, so the output need not be a sequence of unchanged source passages.
Check your understanding
A source-only word has total attention mass 0.6, and p_gen is 0.2. What final probability does copying contribute?
Answer: (1 − 0.2) × 0.6 = 0.48. The word has no vocabulary contribution under the stated assumption.
Related terms
- Attention
- Abstractive Summarization
- Copy mechanism
- pointer network
- sequence-to-sequence model
- out-of-vocabulary token
- extended vocabulary
- coverage mechanism
Learn next
Read attention coverage and tokenization. Rework the example with a word appearing three times in the source, then explain why summing its attention mass differs from selecting only its largest occurrence.
More in Natural language and speech
- Pitch extraction - Pitch extraction is the computational estimation of pitch-related information from a sound recording, usually by estimating fundamental frequency, F0, over...
- Pitch track - A pitch track is a time-indexed sequence of estimated pitch-related values, usually fundamental-frequency estimates from a sound recording.
- Place of articulation - Place of articulation describes where the main constriction or contact occurs in the vocal tract when a speech sound is produced.
- Possessive pronoun - A possessive pronoun is a pronominal expression that indicates a possession-like relationship without repeating a full possessor expression.
- Post-editing - Post-editing is the revision of machine-generated output after it has been produced.
- Post-training - Post-training is additional training applied to an already pretrained model to shape its behavior or capabilities.
Sources
Abigail See, Peter J. Liu and Christopher D. Manning, Get To The Point: Summarization with Pointer-Generator Networks, ACL 2017: https://aclanthology.org/P17-1099/
Original paper, Section 2.2 and Equation 9: https://aclanthology.org/P17-1099.pdf
The extended vocabulary and probability-mixture definition were reviewed in the primary paper. The repeated-name calculation and fictional case are original. The detailed vocabulary discussion is explicitly scoped to the cited word-level formulation.
Previous: Pointer-based model Next: Pointer jumping