Must the interval be symmetric?

No. Percentile and BCa intervals can be asymmetric.

Does an interval containing zero prove no improvement?

No. It indicates uncertainty compatible with zero under that interval construction, not demonstrated equivalence.

Can BCa always be calculated?

No. Degenerate data or unsuitable statistics can prevent a meaningful calculation.

AI Glossary / Statistics and probability / Advanced

Bootstrap confidence interval

Definition

A bootstrap confidence interval is an uncertainty interval for a population quantity constructed using repeated resampling and recalculation of a statistic. Different bootstrap methods convert the resampled estimates into endpoints in different ways. Its stated confidence level describes the intended repeated-sampling coverage of the procedure, which is generally approximate and depends on the method and assumptions.

What it means

An AI evaluation might estimate that a new assistant improves a score by three points. An interval adds information about how uncertain that estimate is under the chosen sampling model. Bootstrapping can be useful when an analytic formula for the estimate's sampling distribution is inconvenient.
The interval is not simply the range of all observed outcomes. Nor is it automatically a range for an individual future customer's result. An interval for average improvement concerns an aggregate population quantity; a prediction interval answers a different question.

How it works

First identify the statistic and a suitable resampling scheme. In an ordinary nonparametric bootstrap, draw samples with replacement from the observed data and recalculate the statistic for each sample. Preserve paired observations, clusters, or other dependence when the design requires it.
For a nominal 95-percent percentile interval, use the bootstrap distribution's 2.5th and 97.5th percentiles. The basic interval reflects those quantiles around the original estimate. Bias-corrected and accelerated, or BCa, intervals adjust the percentile levels using information about bias and how the statistic responds to changes in the sample.
These methods are not interchangeable formulas that must give identical answers. Software also uses quantile conventions that can affect finite-resample endpoints. Record the method, confidence level, number of resamples, and resampling unit so the result can be interpreted and reproduced.

Three practical examples

A team reports a bootstrap interval for the difference in accuracy between two classifiers. It resamples documents together with both classifiers' predictions to preserve the paired comparison.
An operations analyst estimates an interval for median session duration. The resampling unit reflects independent sessions, and the report explains that the result concerns the population median rather than every session.
A customer-experience study estimates mean satisfaction differences using customer-level resampling. Keeping each customer's responses together avoids pretending that repeated responses are independent people.

Think of it this way

Imagine repeatedly reshuffling the representation of your observed cases and calculating the same summary. The interval uses that collection of summaries to describe uncertainty about a target quantity. It is a statistical lens built from the sample, not a window onto every possible population the sample failed to represent.

Worked example

Consider an invented original estimate of a performance improvement, theta-hat = 10 points. Suppose a completed bootstrap calculation has a 2.5th-percentile estimate of 7 and a 97.5th-percentile estimate of 16. These are hypothetical computed quantiles, supplied here to illustrate the endpoint arithmetic rather than claimed results from a real dataset.
The percentile interval is [7, 16]. Its distances from the original estimate are 3 points below and 6 points above. Equal tail probabilities do not require equal endpoint distances from the estimate or numerical symmetry around the bootstrap median.
The basic interval uses [2 times theta-hat minus the upper quantile, 2 times theta-hat minus the lower quantile]. Substituting the values gives [20 - 16, 20 - 7] = [4, 13]. Both intervals have width 9, but their locations differ. This demonstrates why naming the method matters.
The reflection has a rationale: the bootstrap errors relative to the original estimate have quantiles 7 - 10 = -3 and 16 - 10 = 6. Approximating the original estimation error by that distribution leads to subtracting its upper and lower endpoints from the original estimate, reversing their order.
Now imagine an original dataset in which all observed outcomes equal 10. Ordinary resampling always produces the same sample mean, so every bootstrap mean is 10. A percentile calculation collapses to [10, 10]. That numerical result does not prove that the wider population has no variability: the available empirical distribution contains none. Some more elaborate methods can instead become undefined or produce warnings with degenerate data. The right response is to examine the data and inferential assumptions, not declare perfect certainty.

Illustrative case study

Fictional company Maple Evaluation compares a document assistant with an existing process. It initially reports only the estimated average improvement. Reviewers request an uncertainty interval and details of how the evaluation documents were sampled.
The analyst constructs paired resamples and names the interval method in the report. When the endpoints change after accounting for documents grouped within clients, the team retains the analysis that reflects the study design. It also records that the evaluation excludes certain document types. The final interpretation separates uncertainty for represented clients from uncertainty about applying the assistant elsewhere. This is an illustrative scenario, not a documented company study.

Common misconceptions and mistakes

A nominal 95-percent bootstrap interval does not guarantee exact 95-percent coverage in the particular sample size and setting. Poor coverage can arise with small samples, problematic statistics, boundaries, or unsuitable resampling.
The interval should not be interpreted as a frequentist 95-percent probability that the fixed population quantity lies between the specific computed endpoints. Coverage describes how the procedure behaves across repeated datasets.
Increasing resampling repetitions reduces simulation instability; it does not remove sampling bias or add missing populations.
Choosing whichever interval looks most favorable after inspecting several methods undermines a clear analysis rationale. Select a defensible method and investigate important sensitivity transparently.

Business use

Report bootstrap intervals beside estimates when decisions depend on uncertainty as well as average performance. Specify the target population, statistic, resampling unit, and method. Compare the interval with practically meaningful performance requirements, while keeping measurement quality and applicability to future operations visible.

Check your understanding

Answer: With estimate 10 and bootstrap quantiles 7 and 16, what is the percentile interval? [7, 16]. What is the basic interval? [4, 13].

Why can they differ?

Answer: They use different mappings from the bootstrap distribution to uncertainty endpoints.

Related terms

Bootstrap
Standard error
Prediction interval
percentile interval
basic bootstrap interval
BCa interval
coverage probability

Learn next

Study interval coverage through simulation under known populations. Compare percentile, basic, studentized, and BCa methods while keeping the sampling design fixed.

Sampling design

Sources

SciPy, bootstrap documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html . R boot package, Nonparametric Bootstrap Confidence Intervals: https://stat.ethz.ch/R-manual/R-devel/library/boot/html/boot.ci.html . Worked values and the company scenario are original illustrations.

Previous: Bootstrap Next: Bootstrap error-adjusted single-sample technique