Is SRM an outcome-performance test?

No; it checks allocation-related counts.

Can a treatment effect cause a mismatch in a filtered subset?

Yes, if treatment changes the condition used to select that subset.

Should every mismatch trigger automatic data deletion?

No; investigate its cause and consequences.

AI Glossary / Statistics and probability / Intermediate

Sample ratio mismatch

Definition

Sample ratio mismatch, abbreviated SRM, is a statistically significant discrepancy between the observed numbers of experimental units in treatment groups and the allocation proportions expected under the experiment's design. It is a diagnostic for possible assignment, logging, filtering, or analysis problems. It is not simply any unequal group size, because random variation and deliberately unequal allocation are both possible.

What it means

An A/B experiment may be configured to assign half of eligible users to each variant. If the analyzed data contain substantially different proportions, the team must understand why before trusting an outcome comparison.
SRM asks whether the counts are compatible with the expected allocation mechanism. It does not test whether the new feature improves conversion, revenue, or accuracy. A favorable outcome result cannot substitute for checking whether the groups being compared were assembled and recorded correctly.

How it works

Identify the randomization unit and intended allocation. Users, devices, accounts, and sessions are different units. If assignment occurs by user, counting page views can produce a different ratio for legitimate reasons when treatment changes activity.
Calculate expected counts using the configured allocation and the relevant observed total, where appropriate for the design. A planned 70/30 allocation should be checked against 70/30, not against equal group sizes. Time-varying allocation may require period-specific expectations.
For a simple fixed-probability allocation with adequate expected counts, a Pearson chi-square goodness-of-fit test is commonly used. Sum (observed−expected)²/expected over groups and compare with the appropriate reference distribution. Small counts or constrained assignment mechanisms may require another exact or design-specific test.
Choose a monitoring rule in advance. Repeatedly testing many times or many segments can generate false alarms. Thresholds and sequential monitoring methods should reflect how the diagnostic is actually used, rather than treating every p-value as an isolated test.
Localize the mismatch. Compare assignment records with exposure logs, metric records, eligibility filters, and joins. A problem introduced after assignment can leave original allocation balanced while the analyzed sample becomes distorted.
Examine filtering conditions carefully. Conditioning on an action affected by treatment can select different types of users in the two groups. A downstream subset may therefore fail the expected ratio without a broken random-number generator.
Do not repair the symptom by randomly discarding users, inventing missing records, or reweighting until counts look equal. Those actions do not necessarily restore the missing outcome distribution or the intended estimand. Diagnose the mechanism and determine whether valid reconstruction, reanalysis, or a new experiment is needed.
Treat a passed check as limited evidence. Balanced counts do not prove correct randomization, absence of contamination, complete outcomes, or an unbiased metric. SRM is one useful diagnostic in a broader assessment of experimental validity.

Three practical examples

Website testing: an experiment platform detects that one variant loses users during a logging redirect.
AI assistant evaluation: analysts check whether a post-assignment eligibility filter excludes treatment users differently.
Mobile experimentation: a team investigates duplicated device identifiers that distort participant counts across variants.

Think of it this way

Think of a planned lottery that should distribute tickets in a known proportion. Before comparing the winners' experiences, check whether the recorded ticket counts fit the lottery design. An unexpected count imbalance points to a process worth investigating, but it does not reveal the cause by itself.

Worked example

A fictional experiment independently assigns users with probability 0.5 to each of two variants. The analyzed sample contains 10,000 unique assigned users: 5,200 in A and 4,800 in B. Under the stated null model, expected counts are 5,000 each.
The Pearson statistic is (5,200−5,000)²/5,000 + (4,800−5,000)²/5,000 = 8+8 = 16. With one degree of freedom, the approximate chi-square p-value is 0.0000633. A prespecified single-check threshold of 0.001 would flag the result.
The observed split is 52/48. The same percentage split among only 100 users, 52 versus 48, yields a statistic of 0.16 instead. Thus a visual percentage gap is not enough to judge compatibility with chance; sample size matters.
This calculation detects a discrepancy under the assumed allocation model. It does not establish which pipeline stage failed or quantify bias in the outcome estimate.

Illustrative case study

Fictional case: Spruce Commerce tests an AI search ranking feature. The overall assignment table is balanced, but the dataset used for conversion analysis contains fewer treatment users.
Investigators trace the difference to a filtering rule applied after exposure. They assess whether complete records can be reconstructed and whether the intended comparison remains valid before reporting an effect. They do not declare the feature a winner merely because the filtered conversion rate appears higher.

Common misconceptions and mistakes

Unequal counts are not automatically an SRM when unequal allocation was planned.
A statistically significant mismatch is not proof of a specific technical defect; the expected model and analysis subset also need checking.
A large p-value does not certify the entire experiment.
An SRM can arise after successful random assignment through differential missingness or selection.
Forcing counts to match does not by itself repair the causal comparison.

Business use

Build SRM checks into experiment quality review before release decisions depend on treatment effects. Preserve assignment histories and enough pipeline metadata to identify when counts diverge. For AI product experiments, distinguish participant allocation from treatment-driven activity so the diagnostic targets the correct units and expected ratios.

Check your understanding

Why is 52/48 among 10,000 users more concerning than 52/48 among 100 under independent equal allocation?

Answer: The larger sample allows much less proportional random variation.

If user assignment is balanced but page-view counts differ, is that automatically SRM?

Answer: No; page views may be affected by treatment and are not the assignment units.

Related terms

A/B testing
Selection bias
Missing data
randomization
chi-square goodness-of-fit test
experimental unit
sequential monitoring

Learn next

Trace the counts in an experiment from assignment through exposure, logging, filtering, and analysis. Specify the expected ratio and unit at each stage before interpreting a detected discrepancy.

Sources

Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing: https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/ Microsoft Research, Alerting in Microsoft's Experimentation Platform: https://www.microsoft.com/en-us/research/articles/alerting-in-microsofts-experimentation-platform-exp/ The count example, chosen threshold, and case are original illustrations.

Previous: Sample mean Next: Sample rotation