Must the same physical sample visit every site?

No; comparable distributed materials may be used when suitable.

Is reproducibility the same as repeatability?

No; reproducibility concerns specified changed conditions, whereas repeatability concerns closely repeated conditions.

Does the study identify every cause of variation?

No; the design determines which causes can be separated.

AI Glossary / Statistics and probability / Intermediate

Round-robin test

Definition

A round-robin test is an interlaboratory study in which several participants independently perform measurements, analyses, or a defined process under an agreed protocol. Their results are compared to investigate consistency and variation across participants. It can assess reproducibility or method performance, but its conclusions depend on the materials, participants, conditions, and statistical design actually studied.

What it means

Repeating a test within one laboratory shows only part of the picture. Another laboratory may use different operators, equipment, calibration practices, or interpretations of the same procedure. A round-robin test reveals variation that a single-site exercise can miss.
The aim is not automatically to rank laboratories or certify a product. A study designed to estimate a method's reproducibility differs from a proficiency exercise designed to evaluate participant performance against a suitable assigned value.

How it works

Define the objective before recruiting participants. Decide whether the study concerns a measurement method, a manufacturing procedure, an analysis workflow, or agreement with a reference. Different objectives require different controls and reporting.
Select materials and operating conditions that support the intended scope. Homogeneity and stability matter: differences between distributed specimens or changes during transport can masquerade as participant differences. A sequentially circulated artifact needs checks for drift or damage.
Write a clear protocol covering preparation, equipment requirements, repetitions, calibration, reporting units, and treatment of departures. Too little specification can make results uninterpretable; excessively idealized conditions may fail to represent routine use.
Arrange genuinely independent execution where appropriate. Sharing preliminary results can encourage participants to adjust toward one another, reducing the study's ability to reveal natural differences. Blinding or concealed replicate materials may help answer particular questions.
Include replication sufficient for the quantities of interest. Repeated results within a laboratory help assess within-laboratory variation, while multiple laboratories provide information about between-laboratory differences. Many repetitions from one laboratory do not replace broad participant coverage.
Analyze data according to the design. A variance-components model may separate within- and between-laboratory contributions under its assumptions. Check anomalies and protocol departures using documented rules rather than deleting inconvenient observations to obtain a pleasing result.
Distinguish precision from accuracy. Laboratories can agree closely while sharing the same calibration error. Assessing closeness to a true or reference value requires suitable reference information and attention to its uncertainty.
Report limitations, uncertainty, participation coverage, and relevant operating details. A round-robin result describes the investigated process under studied conditions. It does not automatically establish reliability for different materials, future software versions, unrepresented laboratories, or every production environment.

Three practical examples

Material testing: laboratories measure the same property on comparable specimens to study reproducibility.
Environmental analysis: participants analyze prepared samples under a defined analytical protocol.
AI-assisted measurement: independent sites evaluate a fixed image-analysis workflow on controlled reference material, documenting software and preprocessing differences.

Think of it this way

Think of giving several kitchens the same recipe and comparable ingredients, then comparing the results. Repeated dishes from each kitchen reveal consistency within a kitchen; differences between kitchen averages reveal another layer. Agreement alone does not establish that the recipe produces the intended dish.

Worked example

Consider an original illustrative dataset from three fictional laboratories, each reporting two measurements: A gives 99 and 101, B gives 103 and 105, and C gives 97 and 99 units. Their means are 100, 104, and 98; the grand mean is 302/3, approximately 100.667.
Within each laboratory the squared deviations from its mean total 2. The pooled within-laboratory sum of squares is 6, with 3 degrees of freedom, giving a mean square of 2.
The between-laboratory sum of squares is 2 times the sum of squared laboratory-mean deviations, or 112/3. Dividing by 2 degrees of freedom gives 56/3. Under a simple balanced random-intercept model, the method-of-moments between-laboratory variance estimate is [(56/3)−2]/2 = 25/3, approximately 8.333 squared units.
This tiny example illustrates the decomposition, not a reliable qualification study. The assumptions, representativeness of three laboratories, and uncertainty would need scrutiny before broader conclusions.

Illustrative case study

Fictional case: Alder Materials asks several testing sites to evaluate an AI-assisted microscope measurement procedure. It supplies stable reference specimens, fixes the software version, and records calibration and preprocessing settings.
One site's results are consistently higher. Investigators find a scale-setting difference and document it rather than silently adjusting the published data. A follow-up study examines whether the revised instructions improve agreement and whether the method remains accurate against the reference measurements.

Common misconceptions and mistakes

A round-robin test is not a round-robin sports schedule or a computer scheduling algorithm.
Independent sites agreeing does not prove absence of common bias.
More results from a few sites do not necessarily represent the diversity of all intended users.
Observed site differences can include specimen or protocol effects, so they should not automatically be attributed to operator competence.
Successful participation in one study is not blanket certification for unrelated methods or conditions.

Business use

Use round-robin studies when a process must transfer reliably across sites, partners, or equipment. They can expose ambiguous procedures, inconsistent calibration, and hidden implementation differences before wider adoption. Budget for coordination, specimen quality, and analysis rather than treating the exercise as simply collecting many measurements.

Check your understanding

Answer: All sites give tightly clustered values that are 5 units above a reliable reference.

What does this suggest?

Answer: Good agreement can coexist with shared bias.

Why include repeat measurements within sites?

Answer: They help distinguish within-site variation from differences between site averages.

Related terms

Repeatability
Variance components
Interlaboratory comparison
reproducibility
proficiency testing
measurement uncertainty
reference material

Learn next

Sketch a study protocol specifying participants, specimens, repetitions, and controls. Identify which comparisons estimate within-site variation and which provide information about between-site variation.

Sources

Moylan, Brown, and Slotwinski, Recommended Protocol for Round-Robin Studies in Additive Manufacturing: https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=917433 NIST, Interlaboratory Comparisons: https://www.nist.gov/publications/interlaboratory-comparisons The measurement dataset, variance decomposition, and case are original illustrations.

Previous: Rough fuzzy hybridization Next: Round-trip translation