Does ABX identify which version is better?

No. It tests matching or discrimination, not preference.

Is chance performance one third because there are three labels?

No. X matches one of two references, so the usual binary guessing probability is one half.

Does a nonsignificant result prove no audible difference?

No. It supports only a limited conclusion under the study's conditions and sensitivity.

AI Glossary / Statistics and probability / Intermediate

ABX test

Definition

An ABX test is a discrimination test in which a participant compares known references A and B with an unidentified sample X and decides whether X matches A or B. X is assigned according to a controlled randomization procedure. The method is widely associated with audio comparison. It tests the ability to distinguish alternatives under specified conditions, rather than asking which alternative is preferred or objectively better.

What it means

A listener might compare an original audio clip with a processed version. The listener can examine the labeled references, then identify which one corresponds to X. Repeated trials provide evidence about discrimination beyond what random guessing would achieve.
Blinding helps prevent labels, expectations, or an experimenter's knowledge from revealing the answer. The physical and software setup also matters: differences in loudness, timing, switching noise, or visible information can provide unintended clues. A valid result should reflect the intended comparison rather than a side effect of presentation.

How it works

Define the alternatives and the question. Prepare stimuli that differ in the intended way while controlling irrelevant differences. In an audio test, verify playback, alignment, and level conditions appropriate to the comparison.
Choose the number of trials, randomization, analysis, and stopping rule in advance where possible. Give participants clear instructions and distinguish familiarization from scored trials. Conceal X's assignment from the participant and prevent the person administering the test from providing cues.
Collect identification responses across trials. Under a suitable no-discrimination model with independent, equally likely assignments, the probability of a correct response is one half. A binomial tail can then quantify how unusual the observed number of correct responses would be under that model.
Interpret the result within the tested conditions. Multiple listeners, clips, repeated attempts, and dependence between trials may require a more careful analysis. Failure to demonstrate discrimination does not establish universal identity, while successful discrimination does not reveal which alternative listeners prefer or whether the difference matters operationally.

Three practical examples

An audio team compares an original recording with a compressed version to investigate whether listeners can identify the difference.
A speech-processing team tests whether a change in generated or enhanced audio is perceptually distinguishable under controlled conditions.
A product researcher separates a discrimination question from a later preference study, avoiding the assumption that noticeable means better.

Think of it this way

Think of being shown two labeled fabric samples and then receiving one of them without its label. Your task is to identify the match, not choose your favorite fabric. Repetition can reveal discrimination ability, but the result depends on the samples, presentation, and absence of unintended clues.

Worked example

Consider an invented audio ABX session with 16 independently randomized trials, each assigning X to A or B with probability one half. The listener answers 13 correctly. Under the no-discrimination guessing model, the number of correct answers K follows a binomial distribution with n = 16 and p = 0.5.
For a prespecified one-sided test of above-chance identification, calculate P(K ≥ 13). The relevant binomial coefficients are 560 for 13 correct, 120 for 14, 16 for 15, and 1 for 16. Their sum is 697, so the tail probability is 697 / 65,536, approximately 0.01064.
The observed accuracy is 13 / 16 = 81.25%. That descriptive percentage and the null-model tail probability answer different questions. The latter is not the probability that the listener guessed or that the two audio signals are identical.
If the listener had answered 12 correctly, the tail would also include the coefficient 1,820, giving 2,517 / 65,536, approximately 0.03841. Repeating sessions until one looks favorable would require accounting for that procedure. The example assumes a fixed trial count and valid randomization; it does not establish a general listening ability across all material.

Illustrative case study

In this fictional case, Cedar Audio evaluates a processing change for an AI speech product. A casual comparison suggests that one version sounds different, but the team notices that the clips have mismatched playback levels.
It corrects the unintended presentation difference, specifies an ABX protocol, and records all scored trials. It also keeps the discrimination study separate from a preference assessment, because users might hear a difference without preferring either version.
The final report states which clips, listeners, equipment, and conditions were tested. It does not convert a limited result into a claim that the processing is always audible, inaudible, better, or worse.

Common misconceptions and mistakes

An ABX test is not an ordinary website A/B experiment. The letters describe reference stimuli and a matching task, not two independently assigned business-treatment groups.
Another mistake is equating discrimination with quality. A participant may reliably detect an artifact but still prefer the processed version for another reason. Conversely, a failure to reject chance performance may reflect limited trials, difficult material, fatigue, or weak sensitivity.
Uncontrolled cues and selective reporting can also mislead. Repeating trials or sessions until a favorable p-value appears changes the analysis, and pooling dependent observations as if they were independent can exaggerate evidence.

Business use

ABX testing can help teams evaluate whether a technical change creates a detectable perceptual difference. It is useful when claims about transparency or perceptual effects need evidence beyond informal demonstrations.
Match the test to the intended users and content. Record the full protocol and distinguish detection, preference, and task performance. Consider uncertainty and sensitivity before interpreting a null result, and retain unsuccessful as well as successful sessions in the evidence record.

Check your understanding

What is the example's observed accuracy?

Answer: 81.25%.

Why is the null probability one half per trial?

Answer: X has two equally likely reference identities under the stated procedure.

What does the 0.01064 value describe?

Answer: The probability of at least 13 correct responses under the specified guessing model.

Related terms

Binomial test
P-value
discrimination testing
double-blind experiment
randomization
statistical power
perceptual evaluation
preference testing

Learn next

Learn next how blinding and stimulus control support perceptual experiments. Study binomial testing, multiple comparisons, and power, then compare ABX discrimination with preference and quality-rating methods.

Multiple comparisons

Sources

Clark, High-Resolution Subjective Testing Using a Double-Blind Comparator, AES publication record: https://aes2.org/publications/elibrary-page/?id=3839 foobar2000, official ABX Comparator component description: https://www.foobar2000.org/components/view/foo_abx

Previous: Abstractive Summarization Next: Accelerated failure time model