Does every group receive a posttest?

Yes, in the standard four-group arrangement.

Does every group receive a pretest?

No; its absence is an intentional design feature.

Is the interaction the same as the average intervention effect?

No; it measures whether the intervention contrast differs by pretest exposure.

Does randomization guarantee identical baseline groups?

No; it supports balance in expectation, while chance differences remain possible.

AI Glossary / Statistics and probability / Intermediate

Solomon four-group design

Definition

The Solomon four-group design is a randomized experimental design that crosses an intervention with the presence or absence of a pretest. Participants enter four conditions: pretest plus intervention, pretest plus control, intervention without pretest, and control without pretest. All four groups receive a posttest. The arrangement allows researchers to examine the intervention and whether taking the pretest changes later outcomes or changes the apparent intervention effect.

What it means

Measuring people can influence them. A knowledge quiz may prompt further study; a survey can draw attention to a behavior; an initial task can provide practice. A conventional pretest-posttest experiment may therefore evaluate an intervention in people who have already been affected by measurement. The Solomon design includes participants who were not pretested, making that difference visible. It is particularly useful when the evaluation setting may differ from ordinary deployment, where users would not normally take a baseline assessment.

How it works

Randomly allocate eligible units to the four combinations of intervention status and pretest status, following a prespecified allocation plan. Keep the intervention and posttest comparable across the relevant groups. Analyze posttest outcomes with terms for intervention, pretest exposure and their interaction, using methods suitable for the outcome and randomization unit. Pretest scores are available only in two groups, so methods that require them for every participant cannot simply be applied across all four. Plan sample size, missing-data handling and the interaction analysis before collecting results.

Three practical examples

Example 1: A training evaluation checks whether an initial knowledge test itself encourages employees to study. Example 2: A digital wellbeing study examines whether a baseline behavior questionnaire changes engagement with an intervention. Example 3: An AI onboarding trial tests whether an initial practice task makes the new interface more effective. Each application distinguishes the intervention from the experience of being measured. The design still requires careful implementation: merely labeling four convenience groups does not create random assignment or eliminate selection differences.

Think of it this way

Think of testing a new recipe while wondering whether a starter changes how diners judge the main course. Offer the recipe and comparison meal both with and without the starter. The extra conditions reveal whether the starter changes the meal comparison. Here, the pretest is the starter: it may prepare participants, influence their expectations or have an effect of its own.

Worked example

Consider four equally sized randomized groups with posttest means as follows: pretested intervention 80, pretested control 65, intervention without pretest 72, and control without pretest 62. The intervention contrast among pretested participants is 80 minus 65 = 15 points. Without a pretest it is 72 minus 62 = 10 points. Their difference, five points, is the estimated interaction on this additive scale. The equal-weight average intervention contrast is (15 + 10)/2 = 12.5 points. The equal-weight pretest contrast is (80 + 65)/2 minus (72 + 62)/2 = 5.5 points. These are descriptive estimates from the displayed means. Without within-group variability and sample sizes, no standard error or significance claim follows. If ordinary deployment has no pretest, the ten-point contrast may answer the most relevant intervention question.

Illustrative case study

In this fictional case study, Willow Services evaluates an AI training course. Its team worries that the baseline quiz reveals the exact skills that the course later teaches. Employees are randomly allocated to four conditions, and everyone takes the same final assessment. The report presents intervention contrasts separately for pretested and non-pretested employees, alongside uncertainty estimates and participation rates. Rather than treating the most favorable contrast as the headline, managers focus on the condition that matches their planned rollout and investigate whether testing altered engagement.

Common misconceptions and mistakes

Common mistakes include comparing the two intervention groups and calling that the treatment effect, interpreting a nonsignificant interaction as proof of no sensitization, and analyzing all participants as if baseline scores existed for everyone. Four groups also do not automatically control every threat to validity. Attrition, contamination, inconsistent delivery and outcome measurement problems can remain. If randomization occurs by classroom or team, the analysis must respect that clustering. Interaction findings depend on the outcome scale and can require substantial sample size.

Business use

Businesses can use this design when evaluation activities may change the very behavior being evaluated. It helps distinguish an intervention's performance under study conditions from its performance without a baseline test. The extra groups increase recruitment and coordination needs, so the design should address a meaningful concern rather than be added routinely. Preserve the randomized comparisons, monitor delivery and report effect estimates with intervals. A decision to pool across pretest conditions should be justified by the target question and evidence, not solely by a significance threshold.

Check your understanding

What is the intervention contrast without pretesting in the example?

Answer: Ten points.

What is the difference between the two intervention contrasts?

Answer: Five points.

Can the four means establish statistical significance?

Answer: No; variability, sample size and the design are also needed.

Why might a pretest affect external relevance?

Answer: Real users may receive the intervention without the assessment experience that influenced trial participants.

Related terms

Factorial design
Related terms include pretest sensitisation
random assignment
interaction
posttest-only design
internal validity and external validity

Learn next

Learn next how a two-by-two factorial model represents main effects and interactions. Then compare analyses for continuous and binary outcomes, examine clustering and attrition, and plan power for the interaction if it is central. Specify which pretest condition best represents the intended deployment environment.

Sources

Sources: Assessment Effects in Educational and Psychosocial Intervention Trials: An Important but Often-Overlooked Problem, https://pmc.ncbi.nlm.nih.gov/articles/PMC4439294/; Instructional demand and pretesting in the assessment of social anxiety, https://www.sciencedirect.com/science/article/pii/S0005789483801221. These methodological and primary experimental sources describe assessment effects and the crossed design. The four means and business case are original illustrations.

Previous: Solid compression Next: Solomonoff's theory of inductive inference