Does field mean outdoors?

No. It means a relevant real-world context, which can include digital or indoor workplaces.

Is every field experiment randomized?

Randomization is common and especially valuable, but the actual assignment design must be stated.

Does random assignment guarantee generalization?

No. It supports the internal comparison under assumptions, while external applicability needs separate assessment.

AI Glossary / Statistics and probability / Intermediate

Field experiment

Definition

A field experiment evaluates an intervention in a setting that reflects real-world activity rather than only an artificial laboratory task. Researchers deliberately vary a treatment, often through random assignment, and compare outcomes under a defined design. Field context can concern participants, tasks, stakes, information and environment, so the term covers a range of designs. A study is not an experiment merely because it collects data outside a laboratory; observing naturally occurring differences without controlled intervention is a different approach.

What it means

The attraction is learning how an intervention works where people or systems normally operate. Realistic behavior can reveal effects that a simplified task misses, while the experimental design can strengthen causal interpretation. Yet realism alone does not remove confounding, implementation failures or uncertainty about generalization. A well-designed field experiment identifies an effect for a particular population, treatment, time and setting. Extending that result to other contexts requires substantive reasoning and often additional evidence, rather than assuming real-world automatically means universally applicable.

How it works

Define the intervention, comparison condition, outcome and population before implementation. Choose an assignment unit, such as an individual, team or location, that fits the intervention and potential spillovers. Random assignment creates a basis for comparing potential outcomes under stated assumptions, but researchers must track treatment delivery, noncompliance, missing outcomes and contamination. Analyze outcomes according to the planned design, often beginning with the effect of assignment. Match uncertainty calculations to clustering or blocking. Document implementation and evaluate whether the treatment contrast was actually delivered. Appropriate participant protections and permissions are part of conducting the study, not a substitute for sound design.

Three practical examples

Service communication: randomly assign eligible users to alternative reminder formats and compare a prespecified response outcome.
Workplace process evaluation: assign teams to two workflow procedures while accounting for team-level clustering.
Retail operations: test a store-level intervention across locations with a defined comparison period and allocation plan.

Think of it this way

Think of testing a change on the working stage where it will be used, while deliberately arranging a fair comparison. The stage makes behavior realistic; the comparison design makes the evidence interpretable. Neither feature alone supplies everything needed for a reliable conclusion.

Worked example

Suppose a hypothetical field experiment independently randomizes 800 eligible users equally between a new reminder and the existing reminder. The outcome is completion within seven days, with complete follow-up and no interference. There are 120 completions among 400 assigned to the new reminder and 100 among 400 controls. Rates are 30% and 25%, giving an estimated assignment effect of five percentage points. An illustrative unpooled large-sample standard error is sqrt[0.30 x 0.70/400 + 0.25 x 0.75/400], approximately 0.03152. A normal 95% interval is about -1.18 to 11.18 percentage points. The observed difference is compatible with substantial uncertainty and does not establish a positive effect at that conventional level. If assignment occurred by team or follow-up were incomplete, this individual-level calculation would require revision.

Illustrative case study

Fictional case: Pine Service Design tests two reminder formats in its normal workflow. The team records assignment separately from actual message delivery and preserves users in their assigned groups for its primary analysis. Investigators find that some messages failed to send, so they report implementation quality alongside the estimated effect of assignment. A follow-up study is considered before broad rollout. This invented case illustrates the difference between an intervention as assigned and as received, without claiming that excluding undelivered messages would recover an unbiased treatment effect.

Common misconceptions and mistakes

Do not call a before-and-after comparison randomized when no random assignment occurred. Random assignment does not imply random sampling from a wider population. Spillovers can make one unit's outcome depend on another unit's assignment, undermining a simple isolated-unit interpretation. Attrition can bias comparisons even after valid initial randomization. Selecting outcomes or stopping rules after seeing results can distort inference. A natural field experiment in an experimental taxonomy should not be confused with a natural experiment whose assignment arises outside the researcher's control.

Business use

Businesses can use field experiments to reduce uncertainty about changes before committing substantial resources. Choose outcomes that reflect the decision, including implementation costs and relevant adverse consequences, rather than only convenient engagement metrics. Preserve the assignment record, analysis plan and data-quality checks. Interpret estimates together with uncertainty and the scale of the tested intervention. A small pilot may differ from a full deployment because of congestion, learning or spillovers. The strongest practical lesson may sometimes concern feasibility and delivery rather than a precise outcome effect.

Check your understanding

Why is the worked example analyzed by assigned reminder rather than only by messages successfully delivered?

Answer: Assignment preserves the randomized comparison. Delivery can depend on other factors, so restricting to recipients can introduce selection.

Related terms

Spillover (experiment)
External validity
randomized experiment
treatment assignment
intention-to-treat effect
spillover
attrition
natural experiment

Learn next

distinguish random assignment from random sampling, then compare individual and cluster assignment and examine how noncompliance changes the causal question being estimated.

Sources

Sources: Harrison and List, Field Experiments, author-distributed paper: https://s3.amazonaws.com/fieldexperiments-papers2/papers/00058.pdf ; List, field-experiment methods discussion, NBER working paper: https://www.nber.org/papers/w16062.pdf ; J-PAL, research resources on experimental design and implementation: https://www.povertyactionlab.org/resource/randomization ; J-PAL, Data analysis: https://www.povertyactionlab.org/resource/data-analysis .

Previous: Fibonacci search technique Next: FIFO queue