Does replication always require different people or physical objects?

No; the appropriate unit depends on the experiment, but independence and the assignment structure must be justified.

Can a design have replication without randomization?

Yes, although systematic assignment or run order can undermine causal interpretation.

Is replication the same as computational reproducibility?

Terminology varies, but a useful explicit distinction is new data or independent runs versus recomputation with the same data and methods.

AI Glossary / Statistics and probability / Intermediate

Replication (statistics)

Definition

Replication in statistics is the repetition of experimental conditions across appropriate units or runs to obtain information about variability and effects. In broader research practice, it also refers to a new study that collects new data to address the same scientific question as an earlier study. These meanings are connected but operate at different scales. This entry focuses on replication within experimental design while distinguishing it from repeating measurements, copying records and recomputing an existing analysis.

What it means

One response at each treatment setting cannot show how much outcomes vary when that setting is repeated. Replication provides multiple observations at the relevant experimental-unit level, allowing variation to be estimated and treatment comparisons to be interpreted with uncertainty. It can also reveal lack of fit when a model is compared with repeated responses at identical settings. More replication can improve precision under suitable assumptions, but it does not remove confounding, measurement bias or an inappropriate choice of units.

How it works

Identify the unit to which each treatment combination is independently applied. Allocate enough units to repeated treatment conditions, using randomization and blocking where appropriate. Record treatment settings, run order, blocks and nested measurements. Estimate within-condition variability from the true replicates, and use the experimental design to choose the correct error term for comparisons. Where treatment is assigned to clusters, replication for that treatment occurs across clusters; many observations inside one cluster do not substitute for additional independently assigned clusters.

Three practical examples

Example 1: A factory independently processes several parts at each temperature setting to compare average strength. Example 2: An AI-assisted training study randomly assigns several teams to each program and measures workers within those teams. Example 3: A software experiment repeats stochastic training runs at each configuration to quantify run-to-run variability on fixed data. The last example addresses one source of variation only. Claims about new datasets or user populations require evidence at those additional levels.

Think of it this way

Think of repeating a controlled recipe under each of several settings. Multiple preparations show how much the result naturally varies before you judge whether one setting changes the outcome. Tasting a single preparation many times may improve measurement of that preparation, but it does not reveal how the recipe varies across independent preparations. The experimental unit determines which repetition supports the treatment comparison.

Worked example

Suppose six independent experimental units are randomly assigned, three to setting A and three to setting B. Responses under A are 8,10 and 12, while responses under B are 12,14 and 16. The means are 10 and 14, giving an estimated difference B minus A of four. Each group has sample variance four: its squared deviations sum to eight and are divided by two. Under independent errors with a common variance, the pooled variance estimate is 16/4 = 4, using four residual degrees of freedom. The estimated standard error of the mean difference is the square root of 4/3 + 4/3, approximately 1.633. Replication makes this variability estimate possible. These small groups still provide limited precision and weak information about model assumptions; the calculation is not a guarantee that the treatment difference is practically important or generalizable.

Illustrative case study

In this fictional case study, Birch Packaging tests an AI-recommended machine setting. The pilot initially applies each setting to one production batch and measures many packages from that batch. The analyst explains that batch differences remain confounded with settings. The team redesigns the experiment with several independently assigned batches per setting and records packages within batches. Its analysis separates the treatment comparison from within-batch measurement variation. The report explains the number of batches, the uncertainty and the range of operating conditions actually tested.

Common misconceptions and mistakes

Common mistakes include calling repeated measurements independent replications, assuming randomization can compensate for no replication, and treating more data rows as automatic additional degrees of freedom. Replication is also different from blocking: replication supplies repeated conditions, while blocking organizes known nuisance variation. Repeated runs performed in a fixed sequence can be confounded with drift. Finally, replication across studies should compare estimates and uncertainty, rather than declaring success or failure solely because both p-values fall on the same side of a threshold.

Business use

Businesses use replication to make experimental conclusions less dependent on one unusually favorable or unfavorable run. It supports uncertainty estimates for process changes, model configurations and service interventions. Planning should balance independent units, treatment combinations and measurement cost. In AI evaluation, teams should identify whether variability comes from training seeds, sampled test items, human raters or deployment sites. Reporting these levels separately makes the strength and limits of the evidence easier to judge.

Check your understanding

What does replication allow us to estimate in the worked example?

Answer: Within-setting variability and the standard error of the estimated treatment difference.

What is the pooled variance estimate?

Answer: Four.

If each setting were applied to only one batch with many package readings, would there be multiple independent batch replicates?

Answer: No.

Does more replication remove a consistent measurement bias?

Answer: No; it can narrow uncertainty around a biased result.

Related terms

Blocking
Lack of fit
Pseudoreplication
Related terms include replicate
experimental unit
randomisation
pure error
power and reproducibility

Learn next

Learn next how experimental units determine the correct error term in nested and split-plot designs. Study the roles of randomization, replication and blocking together, then examine power calculations for a practically meaningful effect. Before increasing measurements, identify which source of variation the additional observation can actually inform.

Blocking

Sources

Sources: NIST/SEMATECH e-Handbook, Completely randomized designs, https://www.itl.nist.gov/div898/handbook/pri/section3/pri331.htm; A Glossary of DOE Terminology, https://www.itl.nist.gov/div898/handbook/pri/section7/pri7.htm; National Academies, Reproducibility and Replicability in Science, https://www.nationalacademies.org/read/25303/chapter/6. The treatment comparison and fictional packaging case are original illustrations of the stated experimental-unit assumptions.

Previous: Replicate Next: Reporting bias