AI Glossary / Retrieval and knowledge systems / Intermediate

Rating scale

Definition

A Rating scale is a defined set of response options used to record a judgment about an item, experience, or characteristic. Options may be numbers, stars, words, or positions along a visual range. In AI and retrieval systems, rating scales can collect user preferences or evaluator judgments, but their meaning depends on the question, anchors, and conditions under which ratings are given.

What it means

A rating records how someone evaluates a specified target. A five-star recommendation score, a relevance judgment, and an agreement response can all use ordered categories, yet they measure different things. The same numerical label does not make their meanings interchangeable.
Many familiar rating scales are ordinal: higher categories indicate more of the rated quality, but equal numerical gaps do not automatically represent equal perceived differences. Treating the difference between 1 and 2 as equivalent to the difference between 4 and 5 is an additional analytical assumption, not a fact established by printing numbers beside the options.

How it works

Define the attribute to be rated and the object or experience being evaluated. Write a clear question and choose response options that respondents can interpret consistently. Labels at the endpoints, and sometimes at every category, explain what the choices mean.
Keep ordered categories in a meaningful sequence. A neutral middle category, an uncertainty option, and not applicable serve different purposes and should not be silently combined. The suitable number of categories depends on the judgment and audience; adding more numbers does not necessarily add useful information.
Collect ratings with the question wording, scale version, and relevant context. When comparing results over time, changes in anchors, presentation, or respondent population can complicate interpretation. Pretesting helps reveal misunderstandings before large-scale collection.
Analysis should match the intended interpretation. Category frequencies, medians, and defined threshold proportions describe ordered responses without assuming equal spacing. Means are often reported in practice, but their interpretation should acknowledge the numerical coding and measurement assumptions. A multi-item attitude scale also requires more than simply averaging unrelated questions.

Three practical examples

• Search evaluation: ask assessors to rate how relevant a result is to a stated information need.
• Recommendation systems: collect explicit item preferences through a clearly defined star scale.
• AI-output review: use anchored categories to assess a specific quality such as completeness, while evaluating factual correctness separately when needed.

Think of it this way

Think of a rating scale as a labeled measuring guide for judgments. The labels help people place their response, but they do not guarantee that everyone uses the guide identically. Clear anchors and consistent questions make comparisons more meaningful than an unexplained row of numbers.

Worked example

Consider a fictional evaluation of an AI search result using five relevance categories coded 1 through 5. Twenty reviewers provide the following counts: two choose 1, three choose 2, five choose 3, six choose 4, and four choose 5. The counts sum to 20.
The proportion choosing either 4 or 5 is (6 + 4)/20 = 50%. The cumulative count through category 2 is 5, through category 3 is 10, and through category 4 is 16. The two central observations are therefore in categories 3 and 4. For an ordinal report, stating that the middle pair spans those categories avoids implying a physically meaningful category 3.5.
If the analyst uses the numeric coding to calculate a mean, the weighted sum is 2 times 1 + 3 times 2 + 5 times 3 + 6 times 4 + 4 times 5 = 67. The coded mean is 67/20 = 3.35. This is a summary under the chosen coding, not proof that perceived relevance has equal intervals.
Now recode the same ordered categories as 0, 1, 2, 4, and 10. Their order is unchanged, but the coded mean becomes (2 times 0 + 3 times 1 + 5 times 2 + 6 times 4 + 4 times 10)/20 = 77/20 = 3.85. The top-two-category proportion remains 50%. This demonstrates why ordering alone does not determine a unique arithmetic mean.
Finally, compare two separate ten-response profiles under the original 1-to-5 coding. One has ten ratings of 3. The other has five ratings of 1 and five ratings of 5. Both means are 3, but the first shows agreement around the middle and the second shows polarization. Retaining the distribution reveals a difference the mean hides. None of these invented ratings establishes population representativeness or agreement beyond the observed sample.

Illustrative case study

Fictional case: RowanSearch asks reviewers to score answers from poor to excellent without defining the criteria. Some reviewers emphasize readability while others emphasize relevance. The team replaces the single vague question with separately anchored judgments for the qualities it needs to evaluate.
It tests the revised instructions on shared examples and examines disagreements before comparing model versions. This invented case illustrates measurement design; it does not claim that more rating questions automatically produce valid or reliable evaluation.

Common misconceptions and mistakes

Common mistakes include confusing a rating with a ranking, treating missing responses as zero, and assuming a five-point scale is always a Likert scale. A rating evaluates an item on stated categories; a ranking orders items relative to one another. Another error is interpreting a small change in an average without checking the distribution, sample, uncertainty, and consistency of the question.

Business use

Businesses can use rating scales to organize feedback, compare experiences, and provide training or evaluation labels. The useful work is defining what the rating means and how it informs a decision. Store the original categories and question version so later analysts can distinguish genuine changes in judgments from changes in measurement or participation.

People also ask

Are stars objective measurements?

They encode judgments under a particular interface and question.

Can ratings be averaged?

They often are, but equal-spacing assumptions and the full distribution deserve attention.

Is a rating the same as a preference between two items?

No. Pairwise preference is a different response format.

Check your understanding

What percentage selected categories 4 or 5?

Answer: Fifty percent.

Why does recoding change the mean?

Answer: Arithmetic depends on numeric spacing, which ordering alone does not fix.

Why retain category counts?

Answer: Identical means can conceal very different patterns of response.

Related terms

• Likert scale
• Ranking
• Response Bias
• Inter-rater agreement
• Ordinal data
• preference feedback
• relevance judgment
• measurement validity
• survey design
• explicit feedback

Learn next

Next, write anchors for one evaluation question and ask several reviewers to explain how they interpret them. Compare category distributions before deciding which numerical summaries are appropriate for the intended decision.

More in Retrieval and knowledge systems

• Query likelihood model - The query likelihood model is an information-retrieval approach that ranks documents by how likely each document's language model would be to generate the...
• R-precision - R-precision is a ranked-retrieval evaluation measure calculated at a cutoff equal to the number of relevant documents for the query.
• Ranked retrieval - Ranked retrieval returns search results in an order intended to place the most useful or relevant items first.
• Retrieval-augmented generation - Retrieval-augmented generation, usually shortened to RAG, is a way of giving an AI model relevant information before it generates an answer.
• Robust collaborative filtering - Robust collaborative filtering is the design and evaluation of recommendation methods that remain useful when some user feedback is noisy, unreliable, or...
• Schema heterogeneity - Schema heterogeneity is the presence of different ways of naming, organizing or representing data structures across a collection or its sources.

Sources

Pew Research Center, Writing Survey Questions: https://www.pewresearch.org/writing-survey-questions/ Pew Research Center, On a Scale from 1 to 10, How Much Do the Numbers Used in Survey Scales Really Matter?: https://www.pewresearch.org/decoded/2019/03/06/on-a-scale-from-1-to-10-how-much-do-the-numbers-used-in-survey-scales-really-matter/

Previous: Rate - distortion theory Next: Ratio estimation