Must the study inspect one observation at a time?

No; group-sequential procedures inspect batches.

Can a study reach its maximum size without a clear result?

Yes; the plan should specify how to report that outcome.

Is this the same as modeling a time series?

No; sequential analysis here concerns accumulating evidence and decisions about continuation.

AI Glossary / Statistics and probability / Advanced

Sequential analysis

Definition

Sequential analysis is statistical analysis in which accumulating observations can determine whether to continue collecting data or stop according to a specified procedure. Decisions may be reviewed after individual observations or at planned group intervals. Valid inference accounts for the opportunities to stop, rather than treating the final sample size as if it had been fixed independently of the results.

What it means

A study need not always wait for one predetermined final sample before making a decision. Sample size can vary with the evidence.

The flexibility changes the statistical problem. Repeatedly applying an ordinary fixed-sample test and stopping when it becomes significant can increase the chance of a false positive. A sequential procedure evaluates the complete decision rule, including the timing and conditions of its checks.

How it works

Specify the question, observations, decision criteria and allowed analyses. At each permitted look, update the evidence and follow the rule: continue, stop for a specified conclusion, or stop without resolving the question.

Approaches include group-sequential designs, sequential probability ratio tests and methods providing inference valid across allowed stopping times. A maximum sample size, a futility rule or a changed analysis schedule must be handled consistently with the chosen method.

Evaluate both error probabilities and resource requirements. Average sample size is an average across possible study outcomes; an efficient design does not promise an early answer in every run. Record planned and actual analyses.

Three practical examples

1. Product experiments: review an interface comparison at planned stages using a procedure that accounts for those interim looks.

2. Quality inspection: inspect successive items and continue, accept or reject a lot according to a calibrated sampling plan.

3. AI evaluation: accumulate task outcomes until the prespecified evidence criterion or resource limit is reached, reporting an unresolved result when the plan does not support a stronger conclusion.

Think of it this way

Think of a journey with decision points marked in advance. At each point, the available evidence determines whether to continue. The rules must describe the whole journey, because offering several chances to stop is different from assessing only one destination.

Worked example

Consider a fictional success-rate experiment with independent binary outcomes. Under the null hypothesis, every outcome has success probability 0.5. The alternative of interest is a higher success rate.

An analyst proposes two looks, after 10 and 20 observations. At each look, the analyst uses a one-sided exact binomial test with nominal significance level 0.011, or 1.1%.

After 10 observations, rejection occurs with 9 or 10 successes. Under the null, that probability is [10 + 1]/1,024 = 11/1,024, approximately 0.0107422.

After 20 observations, rejection occurs with at least 16 successes. Its null probability is [4,845 + 1,140 + 190 + 20 + 1]/1,048,576 = 6,196/1,048,576, approximately 0.0059090. This single look also stays below 0.011. Discrete outcomes explain the differing actual probabilities.

Now combine the looks: stop and reject at the first look if its threshold is crossed; otherwise continue to the second look and apply its threshold. Overall rejection occurs if either threshold would be crossed.

The events overlap, so their probabilities cannot simply be added. Imagine all 20 outcomes available for the calculation, even though early stopping would leave some unobserved. If the first 10 contain exactly 9 successes, the last 10 need at least 7 successes to reach 16 overall. If the first 10 are all successes, the last 10 need at least 6.

There are 10 times 176 plus 1 times 386 = 2,146 equally likely 20-outcome strings meeting both thresholds. Their total probability is 2,146/1,048,576, approximately 0.0020466.

Subtract that overlap from the sum of the two rejection probabilities. The overall false-positive probability is:

11/1,024 + 6,196/1,048,576 - 2,146/1,048,576 = 15,314/1,048,576, approximately 0.0146046.

That is about 1.46%, exceeding the intended 1.1%. Each look individually meets its nominal threshold, but their combined stopping rule does not. The looks are dependent because the second includes the first ten outcomes; the calculation explicitly handles that dependence.

A valid redesign must calibrate the complete rule. One conservative option for exactly two looks is to allocate at most 0.0055 to each, so the union bound limits the overall probability to 0.011. This rule must be checked under its assumptions; simply renaming the original procedure “sequential” changes nothing.

These calculations describe a constructed experiment. They provide no evidence about a real product, and the chosen null, outcome independence and two-look schedule are essential to the arithmetic.

Illustrative case study

Fictional Alder Evaluation reports an early success after checking an AI feature repeatedly. Its reviewer asks for the full checking schedule and discovers that the report described only the final fixed-sample p-value.

The team reconstructs the decision procedure and distinguishes the exploratory finding from a properly calibrated sequential conclusion. For its next experiment, it specifies the monitoring plan before collecting outcomes. This invented case illustrates why analysis history matters.

Common misconceptions and mistakes

"Every individual test controls error, so the whole procedure does too." The worked example disproves that inference.

"Sequential analysis always uses fewer observations." Resource savings depend on the design and underlying process.

"Stopping without significance proves no effect." The data may remain inconclusive.

"Any confidence interval remains valid after optional stopping." Its construction must support the relevant monitoring rule.

Business use

Choose a monitoring procedure that matches how decisions will actually be made. State the target effect, permitted looks, stopping rules and resource limit, and preserve the analysis record.

For AI-generated reports, distinguish fixed-sample evidence from evidence calibrated for sequential use. Check whether additional metrics, unplanned subgroup searches or extra looks altered the claimed guarantee. A stopping threshold addresses statistical evidence; the practical value and consequences of acting still require assessment.

Check your understanding

Why subtract 2,146 outcome strings?

Answer: They satisfy both rejection conditions and would otherwise be counted twice.

What is the combined false-positive probability?

Answer: Approximately 0.0146046 under the stated null.

Why can 0.0055 per look be conservative?

Answer: The union bound does not require the two rejection events to be independent.

Related terms

Stopping rule
Sequential probability ratio test
Type I error
Interim analysis
optional stopping
group-sequential design
confidence sequence
average sample number

Learn next

Verify the overlap count by summing the binomial coefficients for at least seven and at least six successes in ten trials. Explain why the second look cannot be treated as an independent new experiment.

Sources

NIST/SEMATECH e-Handbook, What is a Sequential Sampling Plan?: https://itl.nist.gov/div898/handbook/pmc/section2/pmc26.htm

Ramesh Johari, Leo Pekelis and David J. Walsh, Always Valid Inference: Bringing Sequential Analysis to A/B Testing, author abstract: https://arxiv.org/abs/1512.04922

Previous: Sequence-to-sequence task Next: Sequential decision making