Is LAS normally no greater than UAS?

With the same aligned tokens and scoring scope, yes, because correct labeled attachment also requires the correct head.

Can I compare constituency F1 directly with dependency LAS?

No. They evaluate different structural units and correctness conditions.

Should evaluation include punctuation?

Follow and report the chosen benchmark or application protocol. Different protocols make different choices, so the convention cannot be assumed.

AI Glossary / Natural language and speech / Intermediate

Evaluating parsers

Definition

Evaluating parsers means measuring how accurately a system analyzes grammatical structure, usually by comparing its predicted parses with reference annotations. The appropriate measures depend on the representation: dependency parsers are commonly assessed through attachment scores, while constituency parsers are commonly assessed through matching labeled phrase spans.

What it means

A parser can get some relationships right while getting others wrong. An evaluation should show those partial successes and failures rather than treating every imperfect sentence as equally incorrect.
In dependency parsing, the central question is whether each word has the correct head and, when required, the correct relation label. In constituency parsing, the question concerns which spans form constituents and what grammatical labels they receive.
Scores are meaningful only in context. Tokenization, annotation conventions, punctuation rules, input quality, and the test domain affect what is being measured. Two reported percentages should not be compared as though they represented identical tasks unless their evaluation conditions are compatible.

How it works

Choose a held-out reference dataset with the intended annotation scheme. Run the parser under a stated input condition, such as raw text or correctly segmented words, and align its predictions with the reference.
With fixed, aligned tokens, unlabeled attachment score, UAS, measures correct heads. Labeled attachment score, LAS, additionally requires the correct dependency label. Constituency evaluation commonly computes precision, recall, and F1 over matching labeled spans, with exclusions determined by the scorer’s rules.
Some shared tasks incorporate segmentation errors directly into attachment evaluation. Their definitions may use precision and recall over aligned nodes rather than assuming identical token lists.
Inspect errors by construction, relation, language, and document type. Aggregate scores summarize performance but do not explain why a parser fails.

Three practical examples

A language-processing team compares two dependency parsers on the same held-out customer-support messages.
A researcher checks constituent-span F1 to evaluate phrase boundaries and labels in a constituency parser.
An extraction team examines subject and object attachment errors that could change who performed an action in downstream records.

Think of it this way

Think of checking a map against a surveyed reference. You can count correct road connections, correct road labels, or completely correct map regions. These answer different questions. The analogy is limited because grammatical annotations can reflect conventions and disagreements, so the reference is not an infallible physical measurement.

Worked example

Suppose an invented evaluation uses five fixed, aligned words, excludes punctuation, and scores each word’s head attachment, including its link to an artificial root where applicable.
For words 1 through 5, a parser gets the heads correct for words 1, 2, 3, and 4. It assigns the wrong head to word 5. Its UAS is therefore 4 / 5 = 80%.
Among those four correct-head predictions, word 3 has the wrong dependency label. Only words 1, 2, and 4 have both the correct head and correct label. LAS is 3 / 5 = 60%.
Even if word 5 carries the right label text, its wrong head prevents it from counting as a correct labeled attachment. The complete sentence is also not an exact match.
This toy calculation assumes identical tokenization. If the parser split or merged words differently, a suitable alignment and scoring policy would be necessary before applying the metric. It would be misleading to reuse these fractions without addressing the mismatched units.

Illustrative case study

In a fictional project, Finch Analytics uses parses to organize statements in maintenance reports. A candidate parser has a slightly better overall attachment score but makes more errors on short imperative instructions common in those reports.
The team reviews a held-out sample of that construction and tests the downstream extraction results. It documents the input segmentation and punctuation policy alongside the scores.
The imagined decision process illustrates why a benchmark improvement is evidence to investigate, rather than an automatic reason to replace an operational parser. No performance improvement or business outcome is claimed for this fictional project.

Common misconceptions and mistakes

UAS does not check dependency labels. LAS does not give full credit for a correct label attached to the wrong head.
A high attachment score does not mean the same percentage of whole sentences is parsed perfectly. One error is enough to make a sentence fail an exact-match test.
Scores obtained with reference tokenization are not directly interchangeable with scores from raw-text pipelines.
Finally, a reference treebank encodes an annotation scheme. Disagreements with it should be examined carefully rather than automatically treated as proof that a competing linguistic analysis is impossible.

Business use

Parser evaluation helps teams choose components, diagnose regressions, and judge whether structural analysis is reliable enough for an application. The most useful test material resembles the language, genres, and sentence patterns the system will actually encounter.
Keep the test data separate from model selection, and record the parser version, preprocessing, scoring implementation, and annotation version. These details make later comparisons interpretable.
When parsing supports another task, measure that task as well. A small overall attachment change may matter greatly for a critical relation or may have little effect on the final user-facing result.

Check your understanding

A parser has the correct head for a word but gives that relation the wrong label. How is this counted under fixed-token UAS and LAS?

Answer: It counts as correct for UAS and incorrect for LAS. Head selection and relation labeling are distinct parts of the prediction.

Related terms

Constituency parsing
Treebank
Precision
Recall
F-measure
Tokenization
Dependency parsing
labeled attachment score
unlabeled attachment score
held-out evaluation

Learn next

Learn the difference between dependency trees and constituency trees. Then manually score a tiny prediction against a reference, recording each correctness decision. Finally, read the actual evaluation script or protocol for the benchmark you intend to use, especially its segmentation and punctuation rules.

Sources

Jurafsky and Martin, Speech and Language Processing, archived dependency-parsing evaluation section: https://web.stanford.edu/~jurafsky/slp3/old_jan23/18.pdf. Official CoNLL 2018 shared-task evaluation protocol: https://universaldependencies.org/conll18/evaluation.html. Stanford CoreNLP Evalb implementation documentation: https://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/parser/metrics/Evalb.html. Independent review consulted the official CoNLL alignment and LAS protocol and Stanford Evalb normalization documentation.

Previous: Eulerian video magnification Next: Evaluation