Is labeled precision just ordinary classification precision?

It follows the same correct-predictions-over-predictions idea, but matching a structured object requires the task's structural and label rules.

Does a correct phrase label earn credit if its boundaries are wrong?

Not under exact labeled-span evaluation. Both the boundaries and category must match.

Can two evaluation tools give different scores for the same trees?

Yes, if their inclusion, normalization, alignment, or aggregation rules differ. The protocol must be specified for meaningful comparison.

AI Glossary / Causal inference and experimentation / Intermediate

Labeled precision

Definition

Labeled precision is the proportion of predicted structures that match reference structures in both their structural identity and their label. In constituency parsing, a predicted constituent is correct only when its word span and phrase-category label match a reference constituent. Labeled precision divides the number of such matches by the number of evaluated predicted constituents. This entry uses that parsing context, while recognizing that other structured-prediction tasks define their own matching units.

What it means

A parser does more than group words. It also names the groups, for example identifying a noun phrase or a verb phrase. Predicting the right span with the wrong category is not fully correct under a labeled evaluation.

Precision asks how dependable the structures proposed by the system are. It does not ask how many reference structures the system found; that is recall. A parser can propose relatively few structures and make most of them correct while missing important parts of the reference analysis. Reporting labeled precision with labeled recall makes that distinction visible, and their F1 score provides a combined summary.

How it works

Represent each evaluated constituent by its beginning, ending, and category. Align the prediction and reference to the same token sequence. Apply the evaluation protocol's rules about punctuation, root nodes, preterminal tags, and label normalization before counting matches.

Count predicted constituents that have an exact matching reference span and label. Divide by the number of predicted constituents included under the protocol. For corpus-level micro evaluation, sum the matching and predicted counts across sentences before dividing.

Do not assume an average of sentence-level percentages equals the pooled result. Also state what happens for a sentence with no evaluated predictions, since its individual precision fraction has a zero denominator. The matching protocol is part of the metric, not incidental formatting.

Three practical examples

1. Parser comparison: researchers compare proposed phrase structures against a treebank to identify whether a new model reduces unsupported constituent predictions.

2. Domain adaptation: a team checks whether a parser trained on general prose assigns reliable phrase categories to product instructions in a new domain.

3. Structured annotation review: engineers compare a parser's labeled spans with expert annotations, separating category mistakes from boundary mistakes to guide error analysis.

Think of it this way

Think of marking boxes around groups of words and attaching a name to each box. A proposed box earns credit only when both its boundaries and its name agree with the reference. The analogy is limited because real parse trees contain nested structures and evaluation tools may exclude some nodes. A visually convincing tree can still differ from the formal counting rules.

Worked example

Consider tokens numbered 1 through 5: “The small robot moves quickly.” For a simplified exercise, suppose the evaluated reference set is S[1-5], NP[1-3], and VP[4-5]. Ignore individual word tags and other nodes by explicit convention.

The predicted set is S[1-5], NP[1-3], ADJP[4-5], and NP[2-3]. Only S[1-5] and NP[1-3] match exactly, so there are two labeled matches among four predictions. Labeled precision is 2/4 = 50%.

ADJP[4-5] has the correct span for the reference VP but the wrong label, so it receives no labeled credit. Labeled recall is 2/3, about 66.7%. These sets are a metric illustration rather than a claim that the candidate is a linguistically appropriate full parse.

Illustrative case study

Fictional case: Fern Text adapts a parser for assembly instructions. Its team initially celebrates a higher labeled-precision score, but the revised system produces fewer evaluated phrase structures. Reviewing labeled recall reveals that some previously found structures are now missing.

The team reports both measures and compares counts by phrase category. It also discovers that one evaluation script strips a label suffix while another retains it, so the two reports were not applying identical matching rules. After aligning the protocol, the team examines whether the remaining differences help its downstream extraction task. The case illustrates why a precision improvement needs context before it supports a product decision.

Common misconceptions and mistakes

Do not count a correct span with an incorrect category as a labeled match. That would mix labeled and unlabeled evaluation.

Do not interpret 90% labeled precision as 90% completely correct sentences. Constituent-level partial credit differs from complete-tree accuracy.

Avoid changing which nodes are counted between system versions or silently averaging sentence scores when the comparison expects corpus-level counts. Tokenization differences can also invalidate span matching before any genuine parsing difference is considered. Diagnose alignment and protocol issues separately from model errors.

Business use

Labeled precision can help evaluate language-processing components used in document analysis or extraction. It reveals whether the structures supplied to later stages are supported by the chosen reference convention.

Business teams should ask how false structural predictions affect the application and whether omitted structures matter equally. Pair precision with recall and inspect the errors in categories the workflow actually uses. A general treebank score may not predict performance on specialized documents. Keep the parser metric as component evidence and add tests of the information extraction or user task when judging practical value.

Check your understanding

A parser proposes ten evaluated constituents. Eight have reference-matching spans, but two of those eight have incorrect phrase labels. What is labeled precision?

Answer: Six predictions match both span and label, so labeled precision is 6/10 = 60%. Matching boundaries alone would overstate the labeled result.

Related terms

Learn next

List the labeled spans in a short reference tree and a candidate tree. Mark exact matches, category-only errors, and boundary errors. Calculate labeled precision and recall, then repeat after removing labels. Explain why the two evaluations answer different questions and record the nodes excluded from counting.

Sources

Primary teaching source: Manning and Schütze, Foundations of Statistical Natural Language Processing course slides, parsing evaluation examples, https://nlp.stanford.edu/fsnlp/fsnlp-land-slides.pdf . Supporting teaching material: Mohit Bansal's parsing lecture, https://www.cs.unc.edu/~mbansal/teaching/slidesFall20/nlp_comp786_fall2020_lec4_sep2.pdf . The labeled-span matching and denominators were consulted. This entry fixes the constituency-parsing scope; other structured tasks require their own unit definition. Sentence examples and the fictional case are original.

Previous: Labeled example Next: Labeled recall