AI Glossary / Causal inference and experimentation / Intermediate
Citation recall
Definition
Citation recall measures how much of an answer's evaluated content is supported by the citations provided, under a defined claim or statement-level protocol. It focuses on evidence coverage: whether statements that require support actually have it. In ALCE-style evaluation, a statement receives credit when its cited passages jointly support it. Citation recall is different from retrieval recall, which asks whether a search process found relevant documents, and from answer completeness, which asks whether the user's question was fully addressed.
What it means
An answer may contain many facts but attach references to only a few. Citation recall makes that gap visible. Merely placing a citation after every sentence is not enough: the cited material must support the associated content. The unit of evaluation matters. A sentence can contain several claims, so sentence-level and atomic-claim-level scoring can produce different results. A useful report states what requires evidence, how content is segmented, and how partial or jointly supported statements are treated.
How it works
Split the answer into the units specified by the evaluation policy, such as factual statements. For each unit, identify its citations and inspect the cited passages. Mark whether those passages provide the required support, potentially in combination. Divide the number of supported units by the total number of evaluated units, using any declared weighting. ALCE's statement-level approach requires a nonempty citation set and joint entailment for full credit. Handle inaccessible sources, uncited claims, and nonfactual text consistently, and preserve the underlying decisions for review.
Three practical examples
- Explanatory answers: check whether every factual statement about a process is supported by the references attached to it.
- Research synthesis: identify conclusions that combine several sources but lack citations covering all parts of the conclusion.
- Internal assistants: measure whether answers based on company documents expose evidence for the policy statements they make.
Think of it this way
Think of checking a bridge's support points: each required point needs adequate support, even if some other points have several strong supports. The analogy illustrates coverage across claims. Its limit is that textual evidence is interpretive, and several documents do not automatically make a statement true or establish that the answer includes everything the reader needs.
Worked example
Suppose a fictional answer is divided into five factual statements under a sentence-level protocol. Statements 1 and 2 each have one citation that fully supports them. Statement 3 has two cited passages that jointly support it. Statement 4 has a citation discussing the topic but not its specific assertion. Statement 5 has no citation. Three of the five evaluated statements are fully supported, so citation recall is 3/5 = 60%.
Adding another supporting citation to statement 1 does not increase recall: that statement already had credit. Adding a valid supporting citation to statement 5 raises recall to 4/5 = 80%. If statement 4 is rewritten into two separate factual units, the denominator and support judgments may change, so the segmentation policy must remain consistent. Finally, deleting statements 4 and 5 would produce 3/3 = 100% for the shorter answer, but could remove information needed to answer the question. Citation recall therefore needs a separate measure of answer completeness; maximizing it by omitting difficult content is not necessarily an improvement.
Illustrative case study
A fictional report-writing team discovers that citations consistently cover introductory facts but miss the final recommendations and numerical comparisons. Its initial audit counted reference markers, so the gap remained hidden. The team begins reviewing each claim against the cited passages and distinguishes source-supported facts from clearly labeled analysis. It tracks citation recall while also reviewing whether reports answer the requested question. The new process exposes unsupported conclusions without encouraging writers to strip out all substantive reasoning merely to improve a coverage score.
Common misconceptions and mistakes
Citation recall does not count the proportion of all relevant documents retrieved. It does not measure how many bibliography entries are present or whether every link works. A cited source may support an inaccurate claim, so source reliability and external correctness remain separate concerns. An answer can achieve high recall while being unhelpfully brief or answering the wrong question. Automated entailment judgments also have errors, and a single sentence containing several propositions can hide partial support unless the evaluation unit is carefully defined.
Business use
Use citation recall to identify where readers cannot trace assertions to evidence. Establish a policy for which claims require citations, including numbers, comparisons, and time-sensitive statements. Keep the cited passage and claim boundaries available so disagreements can be adjudicated. Pair coverage with citation precision, source suitability, and answer completeness. Monitor whether improvements arise from better evidence or simply fewer claims. For synthesis, make the inferential step visible and distinguish what sources explicitly say from what the answer derives from them.
People also ask
Can one citation support several statements?
Yes, if the mapping is clear and the cited material supports each statement being evaluated.
Can several citations jointly support one statement?
Yes, under a joint-support protocol, though the evaluator must check the combined evidence rather than assume that more references mean better support.
Does a citation after every sentence imply perfect recall?
No. Markers without adequate supporting content do not satisfy the evidence requirement.
Check your understanding
Why does adding a second valid citation to an already supported statement leave citation recall unchanged in the worked example?
Answer: Recall counts supported statements, not the number of references. That statement already contributes one successful unit; the uncovered statements remain uncovered.
Related terms
- Citation precision
- Entailment
- evidence coverage
- statement segmentation
- atomic claim
- grounded answering
- retrieval recall
- answer completeness
- source attribution
- factual support
Learn next
Choose a short answer, define its factual units, and mark each as supported, partially supported, unsupported, or uncited. Calculate recall under an explicit full-support rule, then examine how splitting compound statements changes the result. Study citation precision and answer completeness next so coverage is assessed together with reference quality and usefulness.
More in Causal inference and experimentation
- Causality - Causality concerns relationships in which changes in one part of a system can produce changes in another.
- Causation - Causation is a cause-and-effect relationship in which one event, condition or intervention contributes to producing another.
- Character N-gram F-score - Character N-gram F-score evaluates generated text by combining precision and recall of matching character sequences against reference text.
- Cohen's kappa - Cohen's kappa is a statistic measuring agreement between two raters who assign categorical labels to the same items, adjusted for agreement expected under a...
- Collider - A collider is a variable on a causal path where two arrowheads meet, as in X→Z←Y.
- Computer experiment - A computer experiment is a planned study that runs a computational model at selected input settings and analyses its outputs.
Work in finance? Get the free CFO AI Prompt Pack: 7 prompts for finance leaders.
Sources
Gao and colleagues, Enabling Large Language Models to Generate Text with Citations: https://aclanthology.org/2023.emnlp-main.398/ ALCE original citation evaluation implementation: https://github.com/princeton-nlp/ALCE/blob/main/eval.py ALCE project documentation: https://github.com/princeton-nlp/ALCE
Previous: Citation precision Next: Clarification