AI Glossary / Causal inference and experimentation / Intermediate
A/B testing
Definition
A/B testing is a randomized experiment that compares two versions of a product, process, or experience by assigning eligible units to different conditions and measuring predefined outcomes. Version A is often the existing experience and version B a proposed change. With a sound design and analysis, the comparison can estimate the change's effect for the tested population and period. It is more than comparing two historical averages.
What it means
The purpose of random assignment is to make the groups comparable on average before the treatment, including characteristics that were not measured. Differences observed afterward can then be interpreted through the experimental design rather than through users' self-selection into versions. The unit assigned might be a user, account, device, or another suitable entity. Choosing that unit matters: repeated sessions from one assigned user are not automatically independent experimental subjects, and interactions between treated and untreated units can complicate interpretation.
How it works
Define the hypothesis, eligible population, assignment unit, outcome window, primary metric, and important guardrail metrics before running the comparison. Randomly assign units and keep treatment exposure and logging consistent with the design. Plan sample size or duration around a practically meaningful effect and the intended analysis. After checking data integrity, estimate the treatment difference with uncertainty. Use a stopping and monitoring procedure that matches the statistical method. A fixed-horizon test and a valid sequential test have different rules; repeatedly checking ordinary significance and stopping at a favorable moment can distort results.
Three practical examples
- Website experience: compare two search-result layouts using successful task completion and loading-time guardrails.
- AI assistance: compare a revised response interface with an existing one using user outcomes, error reports, and response latency.
- Business workflow: randomize eligible teams or accounts to different support processes when that assignment level fits how the process is delivered.
Think of it this way
Think of dealing shuffled cards to place comparable participants into two groups before showing them different experiences. The analogy captures random assignment. Its limit is that real experiments must handle repeated exposure, missing data, and interactions; a fair initial deal does not automatically ensure a trustworthy final measurement.
Worked example
Suppose an illustrative user-level experiment assigns 10,000 users to A and 10,000 to B. Each user's outcome is whether they complete a defined task within the same observation window. A has 1,000 completions, or 10%; B has 1,100, or 11%. The observed absolute difference is 1 percentage point. The relative increase is (11% minus 10%)/10% = 10%.
Assuming independent users, no interference, and a simple large-sample comparison of these two binary proportions, an approximate standard error for the difference is the square root of [0.10 times 0.90/10,000 + 0.11 times 0.89/10,000], or about 0.004335. A rough 95% normal interval is 0.01 plus or minus 1.96 times that value, approximately 0.0015 to 0.0185, or 0.15 to 1.85 percentage points. This interval depends on the assumptions and analysis plan. It does not address logging errors, repeated unplanned peeking, multiple comparisons, or whether the effect is large enough to justify costs. The values are fictional, not a reported project result.
Illustrative case study
A fictional software team tests a new AI-assisted search experience. Completion improves, but the team also finds longer response times and more repeated queries for one task category. It checks whether assignment and logging worked as intended, then weighs the estimated benefit against those guardrails. The team examines predefined task categories and plans further investigation where needed. It avoids announcing success solely because one favorable metric crosses a threshold while other important outcomes deteriorate.
Common misconceptions and mistakes
A before-and-after comparison is not automatically an A/B test because seasonality, audience changes, or other releases may explain differences. Statistical significance is not the probability that B is better, nor does it establish practical importance. A non-significant result does not prove equivalence. Randomization does not guarantee exact balance in every observed feature. Excluding users based on behavior affected by treatment can undermine the comparison, and a surprising allocation imbalance may indicate an assignment or data-processing problem.
Business use
A/B testing can inform decisions when a change can be assigned and outcomes measured meaningfully. Define what improvement would matter and which harms would outweigh it. Include the relevant costs, latency, reliability, and user-experience measures, rather than optimizing an easy proxy in isolation. Ensure the analysis matches the randomization unit and account for the experimental design's dependencies. Report the effect estimate, uncertainty, tested population, duration, and any limitations that restrict how far the result can be generalized.
People also ask
Must A be the old version?
Often it is, but the labels themselves have no statistical significance; the conditions must simply be clearly defined.
Does a 10% relative lift mean ten percentage points?
No. Moving from 10% to 11% is one percentage point and a 10% relative increase.
Can an experiment prove a permanent benefit?
It estimates effects under the tested conditions. Longer-term behavior, new populations, or changes in the product may differ.
Check your understanding
Why is the worked example's improvement described as one percentage point rather than ten percentage points?
Answer: The rates are 10% and 11%, whose direct difference is one percentage point. Ten percent is the relative increase compared with the original 10% rate.
Related terms
- Control group
- Effect size
- Confidence interval
- Sample ratio mismatch
- Randomized experiment
- treatment
- randomization unit
- statistical power
- guardrail metric
- sequential testing
Learn next
Learn absolute versus relative changes and how random assignment differs from observational comparison. Then design a small hypothetical experiment with a clear assignment unit and outcome window. Study uncertainty, sample-size planning, and stopping rules before interpreting experiment scorecards or subgroup results.
More in Causal inference and experimentation
- 10-fold cross-validation - 10-fold cross-validation is a model-evaluation procedure that divides a dataset into ten parts, trains a fresh model on nine parts, and evaluates it on the...
- A-optimal design - A-optimal design is an experimental-design criterion that chooses observations to minimize the sum, or equivalently the average for a fixed number of...
- Accuracy - Accuracy is the proportion of evaluated predictions that match the correct labels under a stated scoring rule.
- Area under the PR curve - Area under the PR curve summarizes a model's precision-recall tradeoff as its decision threshold changes.
- AUC (Area under the ROC curve) - AUC, when specified as area under the ROC curve, summarizes how well prediction scores separate positive from negative examples across decision thresholds.
- Augmented inverse probability weighting - Augmented inverse probability weighting, or AIPW, is a causal estimation approach that combines outcome predictions with corrections weighted by treatment...
Sources
Microsoft Experimentation Platform, A/B Testing Across Products: https://www.microsoft.com/en-us/research/articles/a-b-testing-across-products/ Microsoft Experimentation Platform, Data Quality: Fundamental Building Blocks for Trustworthy A/B Testing Analysis: https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/data-quality-fundamental-building-blocks-for-trustworthy-a-b-testing-analysis Microsoft Research, Patterns of Trustworthy Experimentation: Pre-Experiment Stage: https://www.microsoft.com/en-us/research/?p=680556
Previous: A* algorithm Next: A-D conversion