AI Glossary / Statistics and probability / Intermediate

Compact letter display

Definition

A compact letter display is a concise way to encode the outcomes of a set of pairwise statistical comparisons using letters beside group estimates. In the usual difference-testing convention, groups sharing at least one letter have not been found significantly different at the stated threshold and adjustment. Groups with no shared letter have a significant comparison. Shared letters do not establish equality or practical equivalence.

What it means

Comparing many group means produces a matrix of pairwise results that can be cumbersome to show beside a chart. A letter display compresses that matrix into symbols such as a, ab and b. A group can carry several letters because nonsignificance is not transitive: A can be nonsignificantly different from B, and B from C, while A and C differ significantly. The letters summarize decisions from a specified analysis; they do not contain effect sizes, confidence intervals or the full evidence.

How it works

Fit an appropriate model and define the estimates and comparison family. Calculate the relevant pairwise tests with the selected multiplicity adjustment and threshold. A letter-assignment algorithm encodes the resulting significant/nonsignificant relationships so each pair can be interpreted by whether its labels overlap. Verify the mapping against the comparison matrix. State the model, outcome scale, adjustment and threshold in the caption. Some software also supports displays based on significant differences or formal equivalence tests; those variants reverse or change the usual interpretation and require explicit labels. The display cannot repair inappropriate comparisons or a misspecified model.

Three practical examples

• Experiment reporting: annotate treatment estimates with a concise summary of adjusted pairwise difference tests.
• AI benchmarking: summarize comparisons among several systems when the underlying analysis accounts for the evaluation design.
• Research review: decode a published letter-labeled table while checking the test method and avoiding claims of equality from shared symbols.

Think of it this way

Think of letters as a compact index into a comparison table. Two rows sharing a symbol point to a particular test decision, not membership in a scientifically proven class of identical performance. Reading the index without the table's rules can turn a useful shorthand into a misleading ranking.

Worked example

Suppose a hypothetical adjusted comparison table has p-values A - B = 0.20, A - C = 0.01, A - D = 0.001, B - C = 0.30, B - D = 0.01 and C - D = 0.02. Using alpha = 0.05 and the usual nonsignificance-sharing convention, a valid display is A: a; B: ab; C: b; D: c. A and B share a, and B and C share b, matching their nonsignificant comparisons. A and C share nothing, matching their significant comparison. D shares nothing with any other group, also matching the table. All six pair relationships can be checked directly. The example does not provide mean differences or prove that A and B are close enough for practical interchangeability.

Illustrative case study

Fictional case: Sequoia Evaluation presents results from four assistants with letters beside estimated task scores. Reviewers initially interpret the shared letters as proof that two systems are equally good. The analyst revises the caption, adds differences and uncertainty, and explains that the tests merely failed to detect some differences under the stated method. For a decision about interchangeability, the team separately defines a meaningful equivalence margin and plans an appropriate analysis. This invented case illustrates responsible reporting, not a conclusion about actual assistant performance.

Common misconceptions and mistakes

Nonsignificance does not establish equality, and two estimates with the same letter can differ by an operationally important amount when precision is poor. Letters are arbitrary symbols, not grades: a is not inherently better than b. Different valid letter assignments can encode the same comparison matrix. A small change around the threshold may change letters despite little change in evidence. The display also hides whether a significant difference is large or trivial. Overlapping confidence intervals are not a reliable substitute for the actual pairwise test results used to construct the display.

Business use

Businesses should use compact letter displays only when their audience understands the convention and the underlying comparisons are appropriate. They can save space, but decisions about AI systems usually need effect sizes, uncertainty, cost and performance on important task groups. A pairwise p-value matrix or plot of differences may communicate the evidence more directly. If the business question is whether systems are sufficiently similar, define a practical margin and use an equivalence framework rather than relabeling nonsignificant difference tests as evidence of sameness.

People also ask

Why can one group have two letters?

It may be nonsignificantly different from two groups that are significantly different from each other.

Does sharing a letter mean the means are equal?

No. Under the usual convention, it means the specified difference test did not reject at the chosen threshold.

Are the letters always based on Tukey comparisons?

No. They can encode other valid pairwise procedures, so the adjustment and comparison family must be reported.

Check your understanding

Why cannot B's label in the example be only a?

Answer: Then B and C would share no letter, falsely encoding their comparison as significant even though its adjusted p-value is 0.30. The label ab preserves both B's nonsignificant relationship with A and its nonsignificant relationship with C.

Related terms

• Multiple comparisons
• Statistical significance
• Equivalence test
• Confidence interval
• Effect size
• pairwise contrast
• adjusted p-value
• Tukey procedure
• estimated marginal means
• comparison family

Learn next

construct a letter display from a small pairwise matrix and verify every pair. Compare it with a plot of estimated differences and intervals to see what the letters omit. Study the distinction between difference testing and equivalence testing before using any grouping display to support a claim that AI systems are interchangeable.

More in Statistics and probability

• Common cause - A common cause is a source of variation that forms part of the usual operation of a process.
• Common source bias - Common source bias is distortion in a measured relationship that arises because the variables are obtained from the same information source.
• Commonality analysis - Commonality analysis decomposes the variance accounted for by a regression model into components associated uniquely with individual predictors and...
• Compact model - A compact model is a small machine-learning model designed to operate on devices with limited computational resources, such as phones, tablets or embedded...
• Comparative trial - A comparative trial is a study that compares two or more treatment or intervention groups to evaluate differences in outcomes.
• Complement Naive Bayes - Complement Naive Bayes, or CNB, is a classification method derived from multinomial naive Bayes that estimates each class's weights using examples outside...

Sources

Russell V. Lenth, emmeans: Compact letter displays: https://rvlenth.github.io/emmeans/reference/CLD.emmGrid.html Russell V. Lenth, emmeans comparisons vignette: https://github.com/rvlenth/emmeans/blob/main/vignettes/comparisons.Rmd Piepho, An algorithm for a letter-based representation of all pairwise comparisons, Journal of Computational and Graphical Statistics, 13(2), 456–466 (2004), as referenced in the emmeans method documentation.

Previous: Commutativity Next: Compact model