Is this identical to a sequential hypothesis test?

No; the broad phased-study approach does not automatically specify stopping boundaries or error control.

Can the process stop early?

Yes, when further work is not justified or the question has been adequately addressed.

Must every phase use the same method?

No; methods can change to answer the next question.

AI Glossary / Statistics and probability / Advanced

Sequential studies

Definition

Sequential studies divide an investigation into distinct phases and allow findings from one phase to influence the design or purpose of later phases. In experimental work, an early phase may identify promising factors, a later phase may refine their settings, and another may check whether the resulting prediction holds in new observations.

What it means

The investigation develops as evidence accumulates. Keep a clear purpose, decision record and distinction between exploratory and confirmatory conclusions.

The defining feature is that earlier results inform later work. Simply running three unrelated surveys on successive dates does not create this relationship. A phase can inform by ruling out an idea or exposing measurement problems.

How it works

Begin with a question, feasible experimental region and suitable outcome measures. Design the first phase to resolve an uncertainty that affects what should happen next. Analyze its results, record the implications and use them to plan the following phase.

A common engineering progression moves from screening factors to investigating a promising region and then checking a proposed setting. Some investigations need further screening; others stop when the original idea loses support.

Keep exploration and confirmation distinguishable. If the same observations both select an option and evaluate it, the apparent advantage may be optimistic. Fresh confirmation helps assess a selected result, but its interpretation still depends on design, sample size and relevant operating conditions. Preserve unsuccessful phases as part of the evidence record.

Three practical examples

1. AI configuration research: screen several settings, investigate the most promising neighborhood, then evaluate the selected configuration on reserved tasks.

2. Process improvement: identify influential production inputs before running a more focused experiment on their interactions and practical settings.

3. Survey development: use early interviews to identify confusing wording, revise the instrument and test the revised version with new respondents before a larger survey.

Think of it this way

Think of exploring unfamiliar ground with a map that becomes more detailed as you travel. An early observation helps choose the next route.

Worked example

Consider a fictional AI document-processing experiment. The team wants a useful quality score while controlling processing time. It begins with a small exploratory comparison of a baseline and two candidate configurations, using the same scoring definition throughout.

In phase one, the baseline produces scores of 70, 72 and 71 on three illustrative evaluation batches, averaging 71. Candidate A produces 73, 75 and 74, averaging 74. Candidate B produces 76, 78 and 77, averaging 77. These are invented batch scores, not three individual documents or sufficient evidence for a deployment decision.

The observed differences from the baseline are 3 points for A and 6 points for B. The team chooses to investigate settings near B. That choice responds to earlier evidence without establishing an optimum.

In phase two, the team evaluates three related settings on a new exploratory batch set. The settings are B-low, B-middle and B-high. Their recorded mean quality scores are 78, 80 and 79, with corresponding mean processing times of 9, 11 and 15 seconds per document.

Suppose the team had specified a 12-second processing-time ceiling for this phase. B-high exceeds it. B-low and B-middle satisfy it, and B-middle has the higher observed quality score among those two. The team selects B-middle for confirmation.

For phase three, freeze the selected configuration and scoring procedure before examining the confirmation batches. On four fresh matched batches, baseline scores are 70, 72, 71 and 73. The selected configuration scores 76, 77, 75 and 78.

The baseline mean is 286/4 = 71.5. The selected configuration mean is 306/4 = 76.5. Batch-by-batch differences are 6, 5, 4 and 5, averaging 5 points. The confirmation mean of 76.5 is lower than the exploratory phase-two mean of 80. That change should be reported, not hidden by quoting only the largest earlier score.

The discrepancy does not by itself prove that selection optimism caused the entire decline. New batches, random variation, implementation changes or other differences could contribute.

Suppose confirmation processing times are 10, 12, 11 and 11 seconds. Their mean is 11 seconds, and all four observed values meet the ceiling. This does not guarantee that every future document will meet it; the measurements are only the stipulated confirmation observations.

The three phases answer different questions: which direction looks promising, which tested setting meets the stated tradeoff, and how the frozen selection performs on fresh batches. These small illustrative samples provide no sample-size recommendation, significance test or global-optimum guarantee.

Illustrative case study

Fictional Cedar Documents initially proposes announcing the phase-two score of 80 as its expected production performance. Its reviewer asks whether that score helped choose the configuration and whether it has been checked independently.

Cedar reports the confirmation mean of 76.5, the matched improvement of 5 points and the limited scope of the evaluation. It investigates performance across additional relevant document types before making a broader claim. This invented case illustrates testing the selected idea on fresh observations.

Common misconceptions and mistakes

"Sequential means the studies merely happen one after another." Earlier findings must inform subsequent phases in this sense of the term.

"The first winning setting is the optimum." It is a candidate selected from the tested possibilities.

"Changing the plan erases earlier evidence." Decisions and unsuccessful phases should remain documented.

"A confirmation average guarantees future outcomes." It describes the observations and design used, with uncertainty and scope limitations.

Business use

Reserve resources for follow-up and confirmation instead of spending the entire budget on the first exploration. Define what each phase must clarify and record why its findings lead to the next decision.

For AI projects, track configuration versions, evaluation sets and metrics across phases. Prevent exploratory tuning from repeatedly consuming the supposedly untouched confirmation data. If the selected system changes after confirmation results are reviewed, describe the change and assess whether new confirmation is needed.

Check your understanding

Why investigate B after phase one?

Answer: Its observed mean exceeded those of the other tested options.

Why was B-high excluded in phase two?

Answer: Its 15-second time exceeded the stated ceiling.

What was the fresh matched quality improvement?

Answer: Five points on average across the four batches.

Related terms

Selection bias
Validation set
Iterative experimentation
screening design
response surface
confirmation run
pilot study
experimental design

Learn next

Recalculate the phase-two choice if the processing-time ceiling were 10 seconds. Explain why changing that constraint after seeing outcomes should be disclosed.

Sources

NIST/SEMATECH e-Handbook, glossary, sequential studies: https://www.itl.nist.gov/div898/handbook/glossary.htm

NIST/SEMATECH e-Handbook, Iterative nature: https://www.itl.nist.gov/div898/handbook/pri/section2/pri223.htm

NIST/SEMATECH e-Handbook, How to confirm DOE results: https://www.itl.nist.gov/div898/handbook/pri/section4/pri46.htm

Previous: Sequential probability ratio test Next: Sequentially ordered organization