Does a baseline measurement make the groups equivalent?

No. It reveals some starting information but does not eliminate all differences or justify every causal assumption.

Must the groups have different starting means?

No. Nonrandom formation can leave important differences even when observed outcome means are similar.

Is difference-in-differences always appropriate?

No. Its assumptions must fit the assignment process, outcome dynamics, and available comparison group.

AI Glossary / Statistics and probability / Intermediate

Non-equivalent groups design

Definition

A non-equivalent groups design compares intervention and comparison groups that were not formed by random assignment and may differ in important ways before the intervention. It is commonly used in quasi-experimental research when existing classes, sites, teams, or customer groups receive different conditions. A common version measures both groups before and after the intervention. Those measurements can strengthen interpretation, but they do not automatically remove selection differences or establish a causal effect.

What it means

The central problem is that the groups may have had different outcomes even if neither had received the intervention. Differences in experience, resources, motivation, customer mix, or prior trends can become entangled with the effect being investigated.

The supplied glossary emphasizes differences on important variables at the start. In broader design terminology, the absence of random assignment is the reason equivalence cannot be assumed. The term does not require a statistically significant baseline difference to have been demonstrated. A nonsignificant baseline test cannot certify that the groups are causally comparable.

How it works

Define the intervention, comparison condition, assignment process, outcome, and timing. Choose a comparison group whose context provides a credible basis for estimating what would otherwise have happened. Collect relevant pre-intervention outcomes and characteristics, ideally across enough time to understand prior patterns.

Analyze the data using a method justified by the design, such as an appropriate adjusted comparison or a difference-in-differences analysis. These methods rely on different assumptions and do not become interchangeable merely because both use baseline information. For difference-in-differences, the key issue is whether untreated outcome trends would have been comparable, not whether starting levels were identical.

Investigate alternative explanations, including group-specific events, changing membership, differential attrition, maturation, regression to the mean, and measurement changes. Report the basis for assignment and limitations of the comparison. A clear design argument is more informative than labeling the study quasi-experimental without explaining why its counterfactual comparison is credible.

Three practical examples

A business introduces an AI support tool in one existing office and compares its outcomes with another office.
A school evaluates a new learning program using an existing class and a comparison class without randomly assigning students.
An organization compares departments adopting a training program at different times, checking whether their prior trends and concurrent changes support the intended analysis.

Think of it this way

Think of comparing two gardens when one receives a new fertilizer. If the gardens already differ in sunlight, soil, and watering, the final difference cannot be attributed to fertilizer alone. Measuring both gardens beforehand helps, but you also need to consider whether their growing conditions changed differently during the study.

Worked example

Consider an original example measuring weekly tasks completed per worker. An intervention office rises from 40 before rollout to 52 afterward. A comparison office rises from 50 to 56. The intervention office's change is 12, while the comparison office's change is six.

The post-intervention comparison alone is 52 − 56 = −4 tasks, apparently favoring the comparison office. The difference in changes is (52 − 40) − (56 − 50) = 6 tasks, apparently favoring the intervention under a difference-in-differences interpretation.

Now suppose the intervention office would have gained nine tasks without the tool because of a separately scheduled staffing improvement, while the comparison office would have gained six. The true intervention effect in this hypothetical construction is 12 − 9 = 3 tasks, not six. The difference-in-differences calculation includes a three-task differential untreated trend.

The arithmetic illustrates both the value of baseline measurements and their limitation. It does not show which counterfactual is correct in a real study. Two time points alone cannot establish the parallel-trends assumption or rule out a group-specific concurrent event.

Illustrative case study

Fictional case: ServiceMaple rolls out an AI assistant in a willing, experienced support team and compares it with a team that has not volunteered. Initial results show faster resolution in the adopting team.

The analyst documents that adoption was voluntary and that the teams handled different requests before rollout. The evaluation adds earlier outcome periods, measures case mix, and investigates staffing changes. Management treats the findings as conditional evidence, with the remaining selection concerns stated explicitly. The case demonstrates why an operationally convenient comparison needs a substantive account of how the groups were formed.

Common misconceptions and mistakes

Non-equivalent does not mean the study has no value. It means that comparability requires investigation and cannot be presumed from random assignment. Conversely, similar baseline means do not establish similarity on all relevant measured or unmeasured factors.

A pretest does not automatically control every source of bias. Matching or regression adjustment can address specified measured differences but does not guarantee removal of unmeasured confounding. A statistically significant difference in changes is also not proof of parallel untreated trends. If each condition contains only one office, many worker observations do not supply many independent office-level assignments.

Business use

Businesses often encounter this design during phased implementation, voluntary adoption, or evaluation of existing programs. It can provide useful evidence when a randomized rollout is unavailable, especially if the comparison is planned before outcomes are examined.

Record why each group received its condition, preserve pre-intervention history, and track simultaneous operational changes. Distinguish the descriptive result from the causal interpretation and its assumptions. When feasible, improve the design through additional comparison groups, more pre-intervention periods, or a prospective assignment rule that supports a stronger analysis. More data alone do not repair a systematically unsuitable comparison.

Check your understanding

The intervention group improves by 12 units and the comparison group by six. Is the intervention's causal effect necessarily six?

Answer: No. Six is the difference in observed changes. Its causal interpretation requires a justified account of what the intervention group's change would have been without treatment.

Related terms

Selection bias
Confounding
Quasi-experiment
comparison group
baseline measurement
difference-in-differences
parallel trends
randomized experiment

Learn next

Study threats to internal validity, then compare baseline-adjusted regression with difference-in-differences. Practice writing the counterfactual assumption needed for a particular operational rollout rather than relying on the design's name.

Internal validity

Sources

Howell, psychology glossary, non-equivalent groups design: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html

CausalPy, quasi-experimental design notation: https://causalpy.readthedocs.io/en/stable/knowledgebase/design_notation.html

The office calculations are original and independently checked. ServiceMaple is fictional.

Previous: Non-destructive editing Next: Non-existent value