AI Glossary / Statistics and probability / Intermediate
Spillover (experiment)
Definition
A spillover in an experiment occurs when a unit's outcome is affected by treatment assigned to another unit. It is a form of interference between units and can be positive or negative. For example, a trained employee may share knowledge with an untreated colleague, or a promotion may divert demand from nearby untreated sellers. Spillovers complicate an analysis that assumes each person's outcome depends only on their own treatment assignment.
What it means
The usual treatment-versus-control comparison often imagines that the control group remains unaffected by treatment elsewhere. With spillovers, that comparison can answer a different question from the effect of treating nobody versus treating everybody. Outcomes depend on a treatment configuration or exposure pattern, not just a single personal label. Randomization still provides valuable information, but the estimand and analysis must account for the design and the relevant connections. Spillovers do not automatically make all experimental learning impossible.
How it works
Specify how one unit's treatment could reach another, such as through a team, neighborhood, communication network or market. Define the effect of interest: a direct effect at a given peer exposure, an effect on untreated units, or an overall policy effect. Choose a design that provides informative variation in those exposures, potentially including cluster randomization or two-stage randomized treatment saturation. Record the relevant connections and use an analysis consistent with the randomization. Assumptions about unobserved connections and exposure definitions need to be explicit.
Three practical examples
Example 1: Employees assigned an AI training course share techniques with colleagues assigned to control. Example 2: A marketplace experiment changes one seller's visibility and affects other sellers' demand. Example 3: A neighborhood information campaign reaches nearby residents who were not directly assigned the message. Spillovers can improve or worsen others' outcomes, and may operate across the chosen cluster boundaries. A simple geographic or organisational separation reduces some pathways only if it matches how influence actually travels.
Think of it this way
Think of turning on a lamp in one part of a shared room. The intended recipient receives light, but nearby people can also benefit. Comparing the illuminated person with someone sitting beside them does not measure the difference between a fully dark room and a fully lit room. The analogy highlights shared exposure; actual experiments need a more precise account of who can affect whom.
Worked example
Consider a hypothetical pair of colleagues with outcome Y = 50 + 10 times own-treatment status + 4 times the colleague's treatment status. Status is zero or one. With neither treated, each outcome is 50. If only the first is treated, the first's outcome is 60 and the untreated colleague's is 54. Their observed difference is six, even though the direct effect of own treatment at fixed peer status is ten. If both are treated, each outcome is 64, giving a fourteen-point change from the nobody-treated condition. The pair's average improvement when one is treated is (60 + 54)/2 minus 50 = seven. These distinct numbers answer different questions. They follow from the stipulated potential-outcome model and are not estimates from a real experiment or proof that spillovers always reduce treatment-control contrasts.
Illustrative case study
In this fictional case study, Cedar Teams tests an AI coaching program. Staff members work closely together and exchange the coaching tips, so researchers anticipate spillovers before allocation. They randomize teams to different treatment proportions and then randomize eligible employees within teams. The analysis distinguishes effects on assigned employees from changes among untreated colleagues under different team exposures. Managers receive estimates aligned with possible rollout strategies, with explicit caveats about contact between teams and the precision available from the number of teams.
Common misconceptions and mistakes
Common mistakes include assuming individual randomization eliminates interference, treating cluster-robust standard errors as a complete solution, and calling every departure from protocol a spillover. Adjusting standard errors does not by itself redefine an inappropriate causal comparison. Noncompliance concerns whether assigned treatment is received; spillover concerns effects of others' treatment, although both can occur. Another error is assuming the bias must be towards zero. Competition, displacement and different exposure patterns can change either the direction or magnitude of a naive comparison.
Business use
Businesses should consider spillovers when users share information, compete for resources or interact through a platform. The question of interest may be the effect of a partial rollout, universal rollout or treatment of a specific subgroup. Those policies can produce different outcomes. A useful plan identifies likely pathways before testing, chooses an allocation design that can support the intended comparison and documents what exposure data are available. Scaling an experiment should not assume that a small isolated treatment contrast equals the effect of changing the entire system.
People also ask
Are spillovers always beneficial?
No; competition can create negative effects.
Does cluster randomization guarantee no spillovers?
No; effects can cross cluster boundaries.
Can randomized experiments study spillovers?
Yes, with suitable designs, exposure assumptions and analyses.
Is a spillover identical to noncompliance?
No.
Does a standard-error correction alone recover the effect of treating everyone?
No; identification of that policy effect needs appropriate information and assumptions.
Check your understanding
What outcome does the untreated colleague have when their peer is treated in the example?
Answer: 54.
Why is the observed treated-minus-untreated difference six?
Answer: The untreated colleague receives a four-point spillover.
What is the change when both are treated compared with neither?
Answer: Fourteen points per person.
Are these three contrasts interchangeable?
Answer: No; they describe different treatment and exposure comparisons.
Related terms
- Related terms include interference
- SUTVA
- exposure mapping
- cluster randomisation
- treatment saturation
- direct effect
- indirect effect and policy effect
- rather than the entire concept
Learn next
Learn next how potential outcomes can be indexed by both own treatment and peer exposure. Compare designs that minimize spillovers with designs that deliberately estimate them. Then examine the assumptions needed to extrapolate from observed allocation patterns to a full rollout, including possible market-wide or network-wide changes.
More in Statistics and probability
- Spectrum bias - Spectrum bias occurs when an evaluation's mix of cases does not adequately represent the range of cases relevant to its intended use, leading to a...
- Speed prior - The Speed Prior is an algorithmic probability proposal introduced by Jürgen Schmidhuber that incorporates computational resources into a preference for...
- Sphericity - Sphericity is a covariance condition used in the conventional univariate analysis of repeated measures.
- Squared ranks test - The squared ranks test, often called the Conover squared ranks test, is a rank-based procedure for comparing dispersion across independent groups.
- Stable model - A stable model is a formally defined interpretation of a logic program with default negation.
- Staged tree (mathematics) - A staged tree is a probabilistic event tree whose nonterminal situations are grouped into stages when they share the same conditional distribution over...
Sources
Sources: Aronow, Eckles, Samii and Zonszein, Spillover Effects in Experimental Data, https://arxiv.org/abs/2001.05444; Aronow and Samii, Estimating Average Causal Effects Under General Interference, https://arxiv.org/abs/1305.6156; Large sample randomization inference of causal effects in the presence of interference, https://pmc.ncbi.nlm.nih.gov/articles/PMC3960089/. These primary methodological sources develop interference-aware inference. The paired-colleague calculation is original and independently checked; the case is fictional.
Previous: Spike-and-slab regression Next: Spindling