AI Glossary / Statistics and probability / Intermediate

Between-group design experiment

Definition

A between-group design experiment compares outcomes from separate groups exposed to different experimental conditions. Each participant receives one condition for the factor being compared. Often called a between-subjects experiment, it can evaluate an AI intervention against a control or alternative while avoiding exposure to both conditions by the same participant.

What it means

Imagine testing an AI-assisted onboarding page against an existing page. One group uses the assisted page, while another uses the existing version. Their outcomes are compared across people rather than by asking every person to try both versions.

The design describes who experiences each condition. It does not, by itself, guarantee comparable groups or a trustworthy causal conclusion. Random assignment, reliable measurement, appropriate controls, and handling of missing outcomes determine what the comparison can support.

This term substantially overlaps with between-subjects designs. Here the emphasis is on conducting an experiment; the broader design label can also describe comparisons involving naturally existing groups, where causal interpretation requires additional care.

How it works

First define the intervention, comparison, target population, primary outcome, and period of observation. Choose the assignment unit: a person, account, classroom, or another unit that can meaningfully receive a condition.

Use a documented random assignment procedure when feasible. Keep other relevant procedures comparable and prevent people from switching conditions inadvertently. Measure outcomes consistently, recording nonparticipation, exclusions, and missing data.

Estimate the prespecified contrast, such as a difference in completion rates or average task time. Report its uncertainty and practical importance. The analysis must reflect the assignment structure: assigning whole teams is not equivalent to independently assigning every team member. A causal claim also needs attention to spillovers, adherence, and outcome availability.

Three practical examples

• A website randomly assigns new accounts to ordinary search or an AI search assistant and compares successful task completion over a fixed week.
• A training study assigns staff to two instructional formats and assesses them using the same independently scored exercise.
• A product team evaluates three explanation styles, with each recruited tester seeing only one style before rating clarity and completing a comprehension task.

Think of it this way

Think of comparing two recipes by giving different diners one version each. Their ratings provide a comparison between groups. Random allocation helps avoid giving one recipe only to enthusiasts, but differences in diners' tastes still create uncertainty that the analysis must acknowledge.

Worked example

Consider an original randomized onboarding experiment with 400 independent users, 200 assigned to each version. Assume every assigned user has a recorded outcome and there are no cross-user spillovers.

In the AI-assisted group, 60 complete the defined task: 60/200 = 30%. In the control group, 40 complete it: 40/200 = 20%. The observed difference is 10 percentage points. The relative increase is (0.30 - 0.20)/0.20 = 50%; these are different summaries of the same counts.

For an illustrative large-sample, unpooled normal confidence interval, the estimated standard error is sqrt[(0.30 × 0.70)/200 + (0.20 × 0.80)/200] = 0.0430.

Using 1.96 standard errors gives 0.10 ± 0.0843, approximately 1.57 to 18.43 percentage points. This interval is approximate and tied to the stated independent-user assumptions. Other interval methods can differ.

The result estimates an average assignment effect in this study. It does not show that every user benefits, identify which feature caused the change, or guarantee the same improvement after a wider launch. Costs and unintended effects still matter.

Illustrative case study

Fictional case: Orbit Learning tests an AI hint feature. Its first proposal compares volunteers requesting hints with learners who decline them. The analyst notes that motivation and difficulty may differ before the feature is used.

The revised pilot randomly assigns eligible learners to access or no access, defines the main learning outcome before analysis, and logs whether the feature is actually used. The primary report compares users according to assignment, with usage patterns described separately.

The company also records frustration and task abandonment. It avoids declaring success solely because average completion improves. A feature can help one measured outcome while creating burdens elsewhere, and the experimental report should make that tradeoff visible.

Common misconceptions and mistakes

• Random assignment and random sampling are different: one allocates conditions, while the other selects people from a population.
• Equal group sizes do not prove that groups are comparable on every characteristic.
• Comparing customers who chose different products is not automatically a randomized experiment.
• Giving one participant both versions creates a within-subjects element, even if the outcomes are later placed in separate spreadsheet columns.
• Thousands of observations from a few assigned teams do not necessarily provide thousands of independent experimental units.
• Repeatedly checking results and stopping when a favorable threshold is crossed requires a suitable sequential analysis plan.

Business use

This design supports decisions about AI interfaces, employee assistance, and customer workflows. Its value depends on asking a concrete question: what changes for which users when access to a defined intervention is assigned?

Before launch, specify eligibility, allocation, exposure logging, outcome windows, and the rule for reaching a decision. Retain the original assignment even when people ignore their assigned feature, so the analysis can distinguish assignment from self-selected usage.

Consider the size of improvement needed to justify implementation and support costs. A statistically detectable change can be commercially trivial, while an inconclusive study may simply be too imprecise to resolve the business question.

People also ask

Must there be exactly two groups?

No. Multiple conditions and factorial arrangements are possible.

Does a control group receive nothing?

Not necessarily; it may receive the current product or another active alternative.

Can randomization guarantee identical groups?

No. It provides a principled allocation mechanism and supports uncertainty assessment, but chance differences remain possible.

Check your understanding

Answer: 1.

What makes the example between-group?

Answer: Different users receive the two versions. 2.

Is a rise from 20% to 30% a 10% relative rise?

Answer: No; it is 10 percentage points and a 50% relative rise. 3.

Why record assignment separately from usage?

Answer: Choosing to use a feature may be related to outcomes, so the two comparisons answer different questions.

Related terms

• A/B testing
• Control group
• Confidence interval
• Between-subjects designs
• randomized experiment
• experimental unit
• intention-to-treat analysis

Learn next

Study randomization and experimental units, then learn how outcome type determines analysis. Compare parallel-group and repeated-measures designs, and practice writing a short analysis plan before viewing results.

More in Statistics and probability

• Berk-Jones test - The Berk-Jones test is a goodness-of-fit approach that assesses discrepancies between an empirical distribution and a specified reference distribution using...
• Berlin procedure - The Berlin procedure is a method for decomposing economic time series into estimated underlying and recurring components.
• Bernoulli Naive Bayes - Bernoulli Naive Bayes is a supervised classification method for binary features, such as whether a word appears in a document.
• Between­subjects designs - Between-subjects designs are study arrangements in which different participants experience the different levels of a factor.
• Bimodal - Bimodal describes a distribution with two distinct peaks, or modes. Values tend to concentrate around two regions rather than one.
• Binary cross-entropy - Binary cross-entropy, often abbreviated BCE, is a loss function for assessing predicted probabilities of a binary outcome.

Sources

Price, Jhangiani, and Chiang, Research Methods in Psychology, Experimental Design: https://opentextbc.ca/researchmethods/chapter/experimental-design/

NIST/SEMATECH, Completely randomized designs: https://www.itl.nist.gov/div898/handbook/pri/section3/pri331.htm

Primary educational accounts were read. The onboarding counts, interval calculation, and fictional case are original; the interval is an explicitly stated teaching approximation.

Previous: Beta-binomial model Next: Between­subjects designs