Does adding a wrong constituent lower labeled recall?

Not directly if existing matches remain unchanged. It adds no recovered reference item, but it can lower labeled precision.

Why require the category label as well as boundaries?

The labeled metric evaluates both grouping and structural type. Recovering only the span leaves part of the prediction task unresolved.

Can a parser have perfect recall and imperfect precision?

In a set-based evaluation, yes: it can contain all evaluated reference structures plus extra incorrect ones. Structural constraints may limit which combinations a particular parser can produce.

AI Glossary / Causal inference and experimentation / Intermediate

Labeled recall

Definition

Labeled recall is the proportion of reference structures that a system recovers with the correct structural identity and label. In constituency parsing, a reference constituent is recovered only when the prediction has the same word span and phrase-category label. The number of exact labeled matches is divided by the number of evaluated reference constituents. This entry uses constituency parsing as its main context; other structured tasks must define their matching units explicitly.

What it means

Recall focuses on coverage of what should have been found. If a sentence contains several reference phrases, the question is how many appear correctly in the proposed parse. Finding a phrase boundary without assigning its correct category is insufficient for a labeled match.

High recall does not establish that every proposed structure is reliable. A system could recover many reference items while adding incorrect ones. Labeled precision evaluates those proposals from the other direction, using the predicted count as its denominator. The two measures share a match count but describe different error patterns, making them useful together when comparing parsers or other structured predictors.

How it works

Align the candidate and reference to the same token sequence, then convert the included constituents into labeled spans. Follow the evaluation rules about which nodes and labels are counted. Find exact matches in boundary positions and category.

Divide the match count by the reference constituent count. A missing reference span lowers recall, as does predicting that span with an incorrect label. For a corpus-level pooled score, add matches and reference counts across all evaluated sentences before dividing.

Record exclusions and any handling of zero-reference cases. Review results by relevant categories or sentence types when the aggregate may hide important gaps. The number only reflects coverage of the reference analysis and counting convention actually used.

Three practical examples

1. Phrase-structure evaluation: a research team checks how much of a treebank's labeled constituent structure a parser successfully recovers.

2. Document-domain testing: developers examine whether long instructions or unusual sentence forms cause the parser to omit structures needed by a later extraction stage.

3. Annotation comparison: a team measures recovered reference structures while reviewing disagreements to distinguish missing spans from correct spans assigned the wrong category.

Think of it this way

Think of checking a packing list against what was placed in a box, requiring both the correct item and the correct type. Recall asks how much of the required list was satisfied. The analogy is limited because a parse is a nested structural analysis, not a collection of independent objects. Reference conventions and shared boundaries can connect errors, and an alternative analysis may be plausible without matching the benchmark.

Worked example

Suppose a reference analysis contains five evaluated labeled spans: S[1-8], NP[1-2], VP[3-8], NP[4-5], and PP[6-8]. A candidate matches S[1-8], NP[1-2], and NP[4-5], but uses VP[3-5] and ADVP[6-8] for the remaining regions.

There are three exact labeled matches among five reference constituents, so labeled recall is 3/5 = 60%. VP[3-5] misses the reference boundary, and ADVP[6-8] uses the wrong category for the reference PP.

If a revised candidate also recovers VP[3-8], recall rises to 4/5 = 80%, provided the other matches remain. Adding unrelated incorrect predictions would not reduce recall directly, although it would lower precision. These abstract span sets illustrate counting and do not specify a complete linguistic tree.

Illustrative case study

Fictional case: Hazel Parse supplies phrase structures to a document-analysis pipeline. Aggregate labeled recall looks stable after a model change, but reviewers notice missing prepositional phrases in long sentences. Gains on common short noun phrases compensate for those losses in the pooled count.

The team breaks down results by phrase category and sentence length, then traces which missed structures matter to the extraction task. It also checks whether reference and prediction tokenization agree. The revised evaluation shows where coverage changed instead of relying on one overall score. The case illustrates diagnostic use of recall and makes no claim about a real parser's performance.

Common misconceptions and mistakes

Do not use the predicted constituent count as the recall denominator; that produces precision instead. The reference count represents the structures to be recovered.

Do not give labeled credit for the right boundaries with the wrong category. Unlabeled recall deliberately asks a less restrictive question.

Avoid interpreting high constituent recall as complete sentence understanding or complete-tree correctness. Also avoid inflating apparent coverage by changing exclusions, dropping difficult sentences, or using a different reference convention without disclosure. These changes alter the evaluated task rather than simply improving the parser.

Business use

Labeled recall helps organizations identify what a language-processing component fails to supply. This is relevant when downstream rules depend on particular phrases or relations being present. Missing a useful structure can prevent later processing even if the structures that do appear are mostly accurate.

Interpret coverage together with precision, relevant error categories, and application outcomes. Decide whether the reference annotation scheme suits the documents and decisions involved. When comparing vendors or model versions, request identical datasets and evaluation rules. A headline recall value from a different domain may not describe the missing-information risks in the intended workflow.

Check your understanding

A reference has 12 evaluated constituents. A system predicts 15, of which nine match reference spans and labels. What is labeled recall?

Answer: 9/12 = 75%. The 15 predicted constituents belong in the precision calculation, which would be 9/15 = 60% for the same match count.

Related terms

Labeled precision
Constituency parsing
Treebank
PARSEVAL
labeled F1
constituent
unlabeled recall
coverage
reference annotation

Learn next

Create a reference list of labeled spans and a candidate list with one omission, one category error, and one extra prediction. Compute recall and precision separately. Then correct each error in turn and explain which denominator or match count changes, keeping the evaluation protocol fixed.

Sources

Primary teaching source: Manning and Schütze, Foundations of Statistical Natural Language Processing course slides, parsing evaluation, https://nlp.stanford.edu/fsnlp/fsnlp-land-slides.pdf . Supporting material: Mohit Bansal, Syntactic Parsing lecture, https://www.cs.unc.edu/~mbansal/teaching/slidesFall20/nlp_comp786_fall2020_lec4_sep2.pdf . The labeled matching rule and reference-based denominator were consulted. The entry uses the constituency-parsing meaning and retains that scope explicitly. All counts, span lists, and the fictional project are original illustrations.

Previous: Labeled precision Next: Labiodental consonants