AI Glossary / Causal inference and experimentation / Intermediate
Experiment
Definition
An experiment is a planned investigation in which a researcher deliberately changes one or more conditions and observes the resulting outcomes. A well-designed experiment specifies the question, comparison, measurements and analysis so that the evidence can support a clear conclusion about the tested conditions.
What it means
Experiments differ from purely observational studies because the investigator controls or assigns some aspect of the conditions. That could be a website variant, a machine setting, a training offer or an input to a computer simulation.
Not every experiment is randomized, and not every trial-and-error activity provides strong evidence. Randomization, replication, control conditions and careful measurement help separate the effect of the planned change from other influences.
The result applies to the particular intervention, population or system and setting studied. A successful laboratory or simulation experiment does not automatically establish the same effect in a different operating environment.
How it works
Define the question and the experimental unit: the entity receiving a condition, such as a user, a production batch or a site. Choose factors and levels, identify outcomes and set a comparison strategy. Use random assignment where appropriate, and blocking or other design features to address known nuisance variation.
Plan sample size, replication, stopping rules and analysis before collecting results. Execute the conditions consistently, record deviations and analyze according to the design. Quantify uncertainty and distinguish exploratory findings from preplanned tests. Consider interference between units, nonadherence, missing outcomes and measurement changes that can complicate interpretation even when assignment was randomized.
Three practical examples
- A product team randomly assigns eligible visitors to two sign-up interfaces and compares completion rates.
- A manufacturer varies temperature and pressure across planned runs to study a quality response.
- A simulation researcher changes input scenarios to investigate the behavior of a computational model.
Think of it this way
Think of arranging a fair comparison between alternatives while keeping track of what else could influence the result. The experiment is the comparison plan, not simply the act of trying something new. The analogy is limited because complex studies may vary several factors together, and making every condition identical except one is not always feasible or even the most informative design.
Worked example
Suppose a randomized website experiment assigns 1,000 eligible users to version A and 1,000 to version B. Completion occurs for 200 A users and 240 B users. The observed rates are 20% and 24%, an increase of four percentage points or a relative increase of 20%.
The different numerical descriptions are both correct but should not be confused. Statistical uncertainty still needs calculation, and the experiment must be checked for assignment or measurement problems.
If the same users can appear repeatedly, the analysis must respect the actual randomization unit and dependence structure. If version B changes who reaches the measured page, restricting analysis to page visitors may alter the question. The invented numbers demonstrate basic reporting, not a complete test or proof that B is ready for universal deployment.
Illustrative case study
Fictional case: Willow Products launches a small interface experiment. Midway through, a team member notices an encouraging result and proposes stopping immediately. Another team also changes the completion event definition during the run.
The experiment owner checks the planned stopping rule, records the measurement change and assesses whether the results remain interpretable. The team distinguishes exploratory monitoring from the preplanned analysis and prepares a corrected follow-up if necessary. This case illustrates why design and execution matter as much as the final calculation. It does not imply that every early stop is invalid or that experiments must follow one universal monitoring rule.
Common misconceptions and mistakes
A before-and-after comparison is not automatically a controlled experiment. Random assignment is different from random sampling: one supports internal comparison, while the other concerns representation of a population.
More observations do not repair systematic measurement errors or a broken assignment process. Repeatedly testing and stopping when a desired result appears can distort conventional uncertainty calculations unless the design accounts for it. Statistical significance does not establish practical importance. Finally, an experiment can answer the wrong question if its intervention or outcome differs from what the organization plans to deploy.
Business use
Experiments help organizations learn which changes improve outcomes and how factors interact. They are especially valuable when plausible stories or predictive associations point in different directions.
Start with a decision that the result can inform. Define meaningful outcomes, guard against unintended consequences and coordinate simultaneous changes. Maintain an experiment record containing eligibility, assignments, versions, dates, metrics and deviations. Report absolute effects, uncertainty and the tested context. Use follow-up studies when deployment conditions differ materially. Good experimentation supports cumulative learning: an inconclusive or negative result can still be useful when the design and question are clear.
People also ask
Must every experiment be randomized?
No, although randomization is an important tool for credible comparisons.
Is an A/B test an experiment?
Usually, when it deliberately assigns alternatives and measures outcomes under a defined design.
Does a significant result mean a large benefit?
No. Statistical evidence and practical effect size are different considerations.
Check your understanding
A study randomly assigns customers to two designs but recruits only existing premium customers. Does randomization alone make the result representative of every potential customer?
Answer: No. It supports the comparison within the study, while generalization to other customers requires additional evidence or assumptions about how effects differ across populations.
Related terms
- Control group
- Replication (statistics)
- Blocking
- Factorial design
- A/B testing
- Estimand
- External validity
- Randomisation
- replication
- experimental unit
- statistical power
Learn next
Learn experimental units and randomization first. Draft a short experiment plan naming the decision, alternatives, outcome, assignment method and stopping rule. Then identify how repeated measurements, spillovers or missing outcomes could affect the analysis before the first run begins.
More in Causal inference and experimentation
- Evaluation function - An evaluation function is a rule or program that assigns a value to something being assessed.
- Evaluation metric - An evaluation metric is a defined measure used to summarize an aspect of a model's or system's performance.
- Exact match - Exact match is an evaluation rule that gives credit only when a prediction equals an accepted reference answer under the specified comparison procedure.
- Extrinsic evaluation - Extrinsic evaluation assesses an AI component or output by measuring its effect on a downstream task or wider application.
- F 1 - F1, sometimes written F 1 or F1 score, combines precision and recall through their harmonic mean.
- False negative rate - False negative rate is the proportion of actual positive cases that a binary classifier incorrectly labels negative.
Sources
NIST/SEMATECH e-Handbook, Choosing an experimental design: https://www.itl.nist.gov/div898/handbook/pri/section3/pri3.htm Miguel A. Hernán and James M. Robins, Causal Inference: What If: https://miguelhernan.org/whatifbook Official and author-source excerpts consulted. Examples are original; full experimental-design and editorial review remain pending.
Previous: Experience replay Next: Experimental benchmarking