Must best mean most positive?

No; it means highest on the specified property.

Can the simple score be negative?

Yes, when worst selections exceed best selections.

AI Glossary / Natural language and speech / Advanced

Best-worst scaling

Definition

Best-worst scaling is a comparative annotation method in which a person selects the highest and lowest items on a specified property from a small group. Repeated choices across overlapping groups support scores or rankings for the items. In natural-language processing, it can be used to annotate properties such as sentiment intensity. Best and worst refer to the chosen property, rather than necessarily meaning good and bad overall.

What it means

Instead of assigning each item a number on a rating scale, an annotator compares the items presented together. For sentiment positivity, the questions could ask which expression is most positive and which is least positive. For clarity, they would ask which text is clearest and least clear.
The comparison gives information about relative position within the displayed group. It does not directly measure an absolute quantity or establish a preference. The property, items and instructions define the score. Combining different properties obscures its meaning.

How it works

Arrange items into overlapping groups, often groups of four, so that each item appears repeatedly with different companions. Ask annotators to choose one item at each extreme. Record which items were shown as well as the selections, because scoring requires exposure counts.
A simple counting score is the proportion of appearances in which an item was selected best minus the proportion in which it was selected worst. With B best selections, W worst selections and N evaluated appearances, the score is (B − W)/N. It ranges from −1 to 1.
Group composition and coverage matter. Balanced appearances help comparison, but equal counts alone do not remove every design effect or disagreement. Reliability should be assessed using additional evidence, such as independent annotation subsets, rather than inferred from the presence of numerical scores.

Three practical examples

A lexicon project compares short expressions for their degree of positivity. Overlapping groups provide relative judgments that can be summarized into sentiment-association scores.
A writing team compares alternative instructions for clarity. It asks about clarity specifically, keeping factual correctness and completeness as separate evaluation questions.
A language-generation study compares candidate responses on a defined quality dimension. The resulting preference scores describe the annotation task and respondent population, rather than automatically measuring every aspect of response quality.

Think of it this way

Think of examining a small selection of objects and picking the heaviest and lightest by feel. Those choices reveal some relative information without requiring a numerical weight for every object. Changing the comparison group can change which object is selected as heaviest. Repeating comparisons with overlap helps build a broader ordering, but does not turn the judgments into exact physical measurements.

Worked example

Consider five invented text items, A through E, being compared for clarity. Use all five possible groups containing four of these items, and collect one judgment per group:
For A, B, C, D: best A; worst D.
For A, B, C, E: best A; worst E.
For A, B, D, E: best B; worst E.
For A, C, D, E: best A; worst D.
For B, C, D, E: best B; worst E.
Every item appears in four evaluated groups. A is selected best three times and worst zero times, giving (3 − 0)/4 = 0.75. B is selected best twice and worst zero times, giving 0.50. C is never selected at either extreme, giving zero. D is selected worst twice and never best, giving −0.50. E is selected worst three times and never best, giving −0.75.
The ranking is A, B, C, D, E. However, A's score does not mean it is 75% correct or that exactly 75% of a broader population would prefer it. It summarizes its best and worst selection frequencies in these particular evaluated appearances.
C's zero also does not establish absolute neutrality or perfect average clarity. It was simply never chosen at either extreme in the displayed comparisons. In other data, a zero could instead arise from equal numbers of best and worst selections, which is a different response pattern.
This example contains only five judgments and demonstrates arithmetic, not a reliable population estimate. A practical study would need an appropriate comparison design, sufficient responses and a way to examine uncertainty and disagreement.

Illustrative case study

In a fictional editing exercise using the worked data, a manager initially interprets the scores as percentages of readers who find each text clear. The analyst explains that the denominator is evaluated appearances, and the numerator is best selections minus worst selections. The scores have a different meaning from a yes-or-no clarity survey.
The team then inspects C. Its zero score hides the fact that no one selected it as either extreme in this small example. Reporting both selection counts alongside the score makes that pattern visible.
Before using the ordering for a larger decision, the team collects more judgments with documented instructions and examines whether independent sets of responses produce similar rankings. It also reviews factual correctness separately. A response can be judged clearer than its companions while still containing an error; the clarity task did not ask annotators to verify every claim.

Common misconceptions and mistakes

Selecting the best and worst of four items does not reveal a complete ranking of all four. The ordering of the two unselected middle items remains unspecified by that judgment alone.
A best - worst score is not a calibrated probability. Nor does the method eliminate subjectivity, position effects or differences between annotators. Evidence that it performed reliably in a particular study does not establish that it always outperforms every rating method for every task.

Business use

Define the property to compare, select representative items and plan overlapping comparison groups. Record exposures and individual choices so that scores can be checked. Report the scoring method and evaluate reliability before treating small differences as meaningful. Keep conclusions specific to the property assessed and the population providing judgments.

Check your understanding

What is A's score?

Answer: Three best selections minus zero worst selections, divided by four appearances: 0.75.

Why does C score zero here?

Answer: It was never selected best or worst.

What would equal best and worst counts produce?

Answer: Zero, despite a different pattern of judgments.

Related terms

Preference learning
Rating scale
Ranking
Comparative annotation
pairwise comparison
sentiment intensity
annotation reliability
exposure count

Learn next

Compare counting scores with choice-based statistical models, then examine how group design and repeated annotations affect reliability.

Sources

Kiritchenko and Mohammad, Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation, 2017, especially sections 3.3 and 5. https://aclanthology.org/P17-2074.pdf The five-item clarity exercise and fictional editing case are original teaching illustrations, not results from the cited study.

Previous: Best node search Next: Beta-binomial model