Is changing one factor at a time always sufficient?

No. It can miss interactions and may use resources inefficiently for questions involving several factors.

Does replication mean asking the same model the same question repeatedly?

That can measure one source of variability, but it is not equivalent to independent replication across the experimental units of interest.

Must every experiment be a full factorial design?

No. The design should match the question, constraints and effects that need to be estimated.

AI Glossary / Statistics and probability / Intermediate

Design of experiments

Definition

Design of experiments is the planned organization of interventions, experimental units and measurements to answer a defined question efficiently and credibly. It determines which factors are varied, how treatments are assigned, what outcomes are recorded and how comparisons will be analyzed. Randomization, replication and blocking are central tools. For AI projects, a sound experimental design connects a product or model change to evidence about its effects under specified conditions.

What it means

An experiment changes something deliberately and observes the response. Design decisions made before data collection determine which effects can be distinguished and how much uncertainty remains. Factors might include prompt format, retrieval settings or interface features, while outcomes might include task success, latency or user effort. The experimental unit is the entity independently assigned to a condition. Repeating measurements on one unit does not automatically create additional independent experimental units. Clear identification of that unit is essential to avoid exaggerated precision.

How it works

State the objective and select meaningful outcomes and factor levels. Identify experimental units, nuisance variables and practical restrictions. Choose a design that supports the desired comparisons, such as a randomized parallel comparison, blocked design or factorial experiment. Randomize allocation where appropriate, replicate at the correct level and standardize measurement. A factorial design varies several factors together and can reveal interactions, where one factor's effect depends on another. Plan analysis and stopping rules before inspecting outcomes. Record deviations and evaluate whether the executed experiment still answers the intended question.

Three practical examples

Prompt research: compare prompt format and retrieval settings jointly to assess both main effects and their interaction.
Product evaluation: randomly assign eligible users to interface conditions and measure prespecified outcomes.
System tuning: use controlled runs and blocking to compare configurations while accounting for differences between machines or workload periods.

Think of it this way

Think of testing a recipe by changing ingredients according to a deliberate plan. If every ingredient changes at once without a comparison structure, you cannot tell what contributed to the result. If some ingredients work differently together, testing only one at a time can miss that interaction.

Worked example

Consider a hypothetical two-factor experiment with factors A and B each off or on. Mean scores are 60 with both off, 70 with only A on, 65 with only B on and 85 with both on. A's effect is 10 when B is off and 20 when B is on. With equal weighting of B's levels, A's average effect is 15. B's effects are 5 when A is off and 15 when A is on, averaging 10. The difference of A's effects is 20 - 10 = 10, demonstrating interaction on this score scale. A one-factor-at-a-time plan starting at both off would not observe the score of 85 and could miss the combined benefit. These cell means alone do not provide uncertainty; replication and a suitable analysis are still required.

Illustrative case study

Fictional case: Birch Assistant Lab wants to improve task completion using a revised prompt and a retrieval option. Rather than deploy both simultaneously and attribute every change to the prompt, analysts plan a factorial comparison. They define the task population, randomize suitable assignments and block on known task categories. The report examines interaction before summarizing average effects and includes uncertainty from the actual experimental units. A later evaluation checks whether findings extend to new tasks. This invented case illustrates experimental planning, not measured evidence that either modification improves performance.

Common misconceptions and mistakes

More runs do not compensate for systematic confounding or an incorrect experimental unit. Randomization reduces systematic allocation bias but does not guarantee perfect balance in every realized sample. Repeated model calls on the same task may be dependent and should not be mistaken for independent tasks. Blocking and randomization serve different roles: blocking organizes comparable units, while randomization governs assignment. Fractional factorial designs can save runs but may alias effects. Statistical significance alone does not establish operational value, and an effect measured in a narrow test setting may not generalize to production.

Business use

Businesses can use designed experiments to compare alternatives before committing resources and to understand which combinations work under relevant conditions. A useful plan includes the decision to be informed, the smallest meaningful effect, the required precision and the cost of running the experiment. For AI systems, preserve model versions, prompts, data access and evaluation rules so conditions remain interpretable. Monitor harms and implementation failures as well as the primary outcome. An experiment should support a concrete decision rather than produce a collection of unplanned favorable comparisons.

Check your understanding

Why is A's average effect of 15 incomplete without the interaction information?

Answer: A improves the score by 10 with B off and 20 with B on. The average depends on how B's conditions are weighted and can hide a practically important difference between settings.

Related terms

Replication (statistics)
Blocking
Factorial design
Confounding
Treatment effect
Randomization
replication
experimental unit
interaction
response surface
statistical power

Learn next

identify the experimental unit in several AI evaluation scenarios. Draw a two-factor table and calculate conditional effects before their averages. Then study blocking, replication and restricted randomization, followed by fractional designs and aliasing when resources cannot support every combination. Connect the design to its intended analysis before collecting results.

Sources

NIST/SEMATECH, Process Improvement and experimental design: https://www.itl.nist.gov/div898/handbook/pri/pri_d.htm NIST/SEMATECH, Choosing an experimental design: https://www.itl.nist.gov/div898/handbook/pri/section3/pri3.htm NIST/SEMATECH, Randomized block designs: https://www.itl.nist.gov/div898/handbook/pri/section3/pri332.htm

Previous: Design effect Next: Detailed balance