Must MOS be based on human ratings?

A direct subjective MOS is, but standardized terminology also distinguishes objective or estimated MOS-related results. State which type is reported.

Can two systems have the same MOS but different user reactions?

Yes; their rating distributions can differ substantially.

AI Glossary / Statistics and probability / Intermediate

Mean opinion score

Definition

A mean opinion score, or MOS, is an average of ratings assigned on a predefined opinion scale in a quality-evaluation study. It is widely used for speech, audio, video, and audiovisual systems. A reported MOS should identify what was rated, the scale and test procedure, and whether it came directly from human judgments or from a model intended to estimate such judgments.

What it means

MOS summarizes how a defined group of assessors rated a defined set of experiences under specified conditions. For synthetic speech, a study might ask listeners about overall quality or naturalness. Other studies ask different questions, so their averages are not automatically interchangeable.
The mean is a compact summary of opinions, not a physical unit of quality. A score of four on a five-point scale is not equivalent to eighty-percent correctness or a guarantee that every listener found the output good.

How it works

Define the attribute being evaluated. Overall listening quality, intelligibility, naturalness, speaker similarity, and conversational experience are distinct targets. Use a question and rating scale that match the intended attribute rather than assuming one MOS covers them all.
Specify the scale and its verbal anchors. A commonly used five-point listening-quality scale runs from bad at one to excellent at five. Other scales and assessment procedures exist, so include the actual scale whenever a result is reported.
Choose representative stimuli and assessors. Language familiarity, hearing conditions, task expectations, and the range of sample quality can affect ratings. A study of a few easy sentences should not be treated as evidence about every language, accent, or conversational situation.
Control the presentation procedure. Listening equipment, background noise, volume, sample order, and whether identities are revealed can influence responses. Use an appropriate study protocol and record deviations that matter for interpretation.
Compute the mean for the defined evaluation unit. This might be a particular stimulus, system, or condition. If different stimuli receive different numbers of ratings, averaging all ratings and averaging stimulus-level means give different weights.
Inspect rating distributions as well as means. Two systems can share the same MOS while one receives consistently middle ratings and the other strongly divided opinions. The variation can reveal audience differences or unstable performance that the average conceals.
Quantify uncertainty according to the design. Ratings from the same listener or on the same stimulus are not automatically independent. Repeated-measures or hierarchical structures can require methods that reflect both listener and stimulus variability.
Distinguish human-derived MOS from objective or estimated MOS-like scores. A model may predict an expected human score from signal features, but that output is not a new panel of human opinions. Validate such a predictor on the relevant domain and label its origin clearly.
Interpret differences in context. Rating categories are ordered, and their numerical averaging is a conventional summary; equal numerical gaps need not represent equal perceptual differences. Cross-study comparisons require comparable populations, materials, questions, scales, and procedures.
Finally, connect perceived quality with other evaluation targets. An AI voice can sound natural while mispronouncing a name or speaking incorrect content. MOS should complement measures that address the actual task and its important failure modes.

Three practical examples

A speech-synthesis team collects listener ratings of naturalness for a defined set of generated utterances.
A communication-system study compares perceived listening quality under specified transmission conditions.
An AI product team reports human MOS separately from a learned quality predictor's estimated score.

Think of it this way

Think of asking several people to rate the same experience using the same labeled scale, then summarizing their responses. The average is meaningful only when the experience, question, and scale are clear.

Worked example

An original example collects five ratings for one audio sample on a one-to-five scale: 4, 5, 4, 3, and 4. The total is 20, so MOS = 20/5 = 4.0. The sample variance is [(0)^2 + 1^2 + 0^2 + (-1)^2 + 0^2]/4 = 0.5.
Suppose a second sample receives one rating of 2. Pooling all six ratings gives (20+2)/6 = 11/3, approximately 3.6667. Giving the two samples equal weight instead gives (4+2)/2 = 3.0. The difference comes from the chosen evaluation unit, not an arithmetic error.
As a separate distribution check, ratings 3, 3, 3, and 3 have the same mean of 3 as ratings 1, 1, 5, and 5. The latter show sharply divided opinions that the common mean does not reveal.
These small examples explain aggregation. They are not a recommended listener count and do not establish a reliable difference between real systems. An uncertainty analysis would need the actual listener-and-stimulus design.

Illustrative case study

In a fictional case, Elm Voice compares two synthetic voices. One receives a slightly higher overall MOS in a small test, but the team examines the materials and listener groups before selecting it.
A review finds that one voice performs inconsistently on unfamiliar names and that the study contains few such examples. Analysts expand the evaluation, report pronunciation findings separately, and retain the MOS question and rating distribution in the report. The case illustrates careful interpretation, not a measured advantage for a real voice system.

Common misconceptions and mistakes

Do not treat MOS as percent accuracy, assume every study uses the same five-point scale, or compare numbers from different protocols as if they shared a common calibration.
Do not present an automatically predicted MOS as directly collected human ratings. A higher mean does not establish that every listener prefers the system or that the system's content is correct.

Business use

MOS can make subjective experience visible during product evaluation, especially when signal-based metrics do not capture what users hear or see. Its value depends on the study design and the audience represented. Use it with task-specific checks and uncertainty reporting so a small average difference does not become an unsupported product claim.

Check your understanding

Why does pooling the example's ratings give a different result from averaging the two sample means?

Answer: Pooling gives the first sample five times the rating weight, while the mean of sample means weights both samples equally.

Related terms

Rating scale
Subjective evaluation
Listening quality
Naturalness
Inter-rater variability
Quality of experience
Objective quality prediction

Learn next

Next, design a small blinded listening study with a defined question and representative materials. Then examine rating distributions and uncertainty methods that reflect repeated listeners and stimuli.

Sources

ITU-T P.800.1, Mean opinion score terminology, July 2016 edition: https://www.itu.int/rec/dologin_pub.asp?id=T-REC-P.800.1-201607-I!!PDF-E&lang=e&type=items ITU-T P.800.2, Mean opinion score interpretation and reporting, May 2013 edition: https://www.itu.int/rec/dologin_pub.asp?id=T-REC-P.800.2-201305-S!!PDF-E&lang=e&type=items ITU-T P.800, Methods for subjective determination of transmission quality: https://www.itu.int/rec/dologin_pub.asp?id=T-REC-P.800-199608-I!!PDF-E&lang=e&type=items

Previous: Mean field approximation Next: Mean Percentage Error