Must resamples have the original sample size?

The ordinary bootstrap described here does; other bootstrap variants can differ.

Can a record appear twice?

Yes. Replacement is essential to the ordinary method.

Is a narrow interval proof of reliable deployment?

No. It does not resolve omitted populations, poor measurements, or changing operating conditions.

AI Glossary / Statistics and probability / Advanced

Bootstrap

Definition

The bootstrap is a statistical resampling method used to approximate how an estimate would vary across repeated samples. In the ordinary nonparametric bootstrap, observations are repeatedly drawn with replacement from the observed dataset, and the statistic is recalculated for each resample. The resulting distribution can help estimate standard errors, bias, or uncertainty intervals, subject to appropriate assumptions.

What it means

A single AI evaluation provides one accuracy estimate, median response time, or performance difference. Collecting many genuinely new evaluation datasets may be expensive. Bootstrapping uses the available sample as a working representation of the population to explore the estimate's sensitivity to sampling variation.
It does not manufacture new independent evidence. Repeating a customer record in a resample is a computational device, not another customer. If the original dataset misses important users or conditions, resampling cannot automatically fill those gaps.

How it works

Choose the quantity to estimate and identify the units that can reasonably be resampled. For an ordinary independent-observation bootstrap with n observations, draw n records with replacement. Some records may appear several times and others not at all. Compute the statistic, repeat this process many times, and examine the resulting values.
The sample standard deviation across simulated statistic values estimates the bootstrap standard error. Confidence intervals require a further construction: percentile, basic, and bias-corrected and accelerated intervals are different choices, with different behavior.
Paired comparisons should preserve pairs during resampling. Repeated observations within customers may require resampling customers rather than individual rows. Time-dependent data may require blocks or another dependence-aware method. The resampling scheme must reflect the sampling structure and the inferential question.

Three practical examples

An evaluator compares two classifiers on the same documents. Resampling document-level pairs preserves which predictions came from the same case when calculating the performance difference.
A service team estimates uncertainty in median response time. It resamples suitable independent sessions and recalculates the median, while checking whether rare delays are adequately represented.
A modeling team assesses optimism in a prediction model. A bootstrap validation procedure repeats model development within resamples and compares appropriate performance measures, rather than treating training performance as independent validation.

Think of it this way

Imagine placing one card for every observed case in a bag, drawing a card, recording it, and returning it before the next draw. A new handful can contain repeated cards. Repeating the exercise shows how a summary changes under that empirical sampling model; it does not reveal cards that were never in the bag.

Worked example

Consider an invented dataset of three response times: 2, 4, and 6 seconds. The observed mean is 4 seconds. An ordinary bootstrap sample draws three times, independently and with replacement, from these three values.
There are 3 cubed, or 27, equally likely ordered resamples. For example, (2, 2, 6) has mean 10/3, while (6, 2, 2) has the same mean but is a different ordered resample. The sample (4, 4, 4) has mean 4.
Enumerating all possibilities gives bootstrap means 2, 8/3, 10/3, 4, 14/3, 16/3, and 6. Their respective frequencies are 1, 3, 6, 7, 6, 3, and 1, totaling 27. The exact mean of this bootstrap distribution is 4.
A single draw from the empirical distribution has variance ((2-4)^2 + (4-4)^2 + (6-4)^2)/3 = 8/3. The variance of the mean of three independent draws is therefore 8/9. Its standard deviation is approximately 0.943 seconds: the exact conditional bootstrap standard error in this illustration.
For comparison, the original sample variance calculated with denominator n-1 is 4, giving the familiar estimated standard error of the mean sqrt(4/3), approximately 1.155 seconds. The finite-sample difference arises because the ordinary bootstrap uses the empirical distribution with variance denominator n. It is not an arithmetic contradiction.
With a large dataset, enumerating every ordered resample is impractical, so random resampling approximates the bootstrap distribution. Increasing the number of resamples improves that computational approximation; it does not increase the number of original observations. Three observations are used here to expose the mechanics, not to support a trustworthy service-level conclusion.

Illustrative case study

Fictional company Cedar Support evaluates an assistant using several conversations from each customer. An initial analysis resamples individual messages, treating every message as independent. The analyst recognizes that messages from one customer share context and behavior.
The team revises the resampling unit to the customer, retaining that customer's associated records together. It documents whether its performance statistic weights customers equally or weights individual messages equally, because these answer different questions. The revised report describes sampling uncertainty alongside dataset coverage and measurement limitations. This scenario illustrates analysis design; it is not a real company result or a guarantee that cluster resampling solves every dependence problem.

Common misconceptions and mistakes

The bootstrap is not assumption-free. An inappropriate resampling scheme can give misleading uncertainty even after millions of repetitions.
More resamples do not repair a biased original sample or add unseen rare events.
A bootstrap confidence interval does not have guaranteed exact coverage in every finite sample. Small samples, extreme statistics, boundaries, and degenerate data can cause problems.
Repeatedly evaluating an already fitted model on resampled training rows is not, by itself, a complete correction for model-selection optimism.

Business use

Bootstrapping can support uncertainty reporting when an estimate is complicated or an analytic standard-error formula is inconvenient. Record the resampling unit, statistic, number of repetitions, interval method, and random seed. Also state which uncertainty is represented: sampling variation, model-development variation, and future distribution changes are different concerns.

Check your understanding

How many ordered resamples exist in the three-value example?

Answer: There are 27.

Why preserve paired predictions?

Answer: To retain their shared case-level relationship.

What does increasing repetitions primarily improve?

Answer: The numerical approximation to the chosen bootstrap distribution.

Related terms

Resampling
Empirical distribution
Standard error
Confidence interval
paired bootstrap
cluster bootstrap
bootstrap validation

Learn next

Compare percentile, basic, and bias-corrected and accelerated intervals. Then study how cluster and block resampling preserve important dependence structures.

Resampling

Sources

Frank Harrell, Glossary of Statistical Terms, bootstrap: https://hbiostat.org/glossary/ . SciPy, bootstrap documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html . Numerical examples and the company scenario are original illustrations.

Previous: Boosting Next: Bootstrap confidence interval