AI Glossary / Data science and analytics / Intermediate
Percentile Bootstrap Confidence Interval
Definition
A percentile bootstrap confidence interval estimates uncertainty by repeatedly resampling the observed data, recalculating a statistic and taking specified percentiles of the resulting bootstrap estimates as interval endpoints. A two-sided 95% version uses the 2.5th and 97.5th percentiles.
What it means
Suppose a sample gives an average response time, but the team wants to communicate sampling uncertainty around the population average. The bootstrap uses the observed sample as a stand-in for the unknown population and explores how the estimate varies across resamples.
The percentile method reads the endpoints directly from that distribution of recalculated estimates. It is different from taking percentiles of the original individual response times. The statistic being resampled could be a mean, median, difference or another quantity for which the method is appropriate.
The procedure is convenient, but not assumption-free. Its usefulness depends on the sample, the statistic and the resampling design. A large number of resamples cannot repair a biased original sample or guarantee accurate coverage for a difficult statistic.
How it works
For a simple independent sample of n observations, draw n observations with replacement from the original sample. Replacement means that an observation can appear several times in a resample while another is absent. Compute the statistic and repeat this process many times.
Sort the resulting estimates and obtain the required lower and upper percentiles. For a 90% two-sided interval, use the 5th and 95th percentiles; for 95%, use the 2.5th and 97.5th. Software may interpolate between ordered estimates, so record the percentile convention when exact reproduction matters.
Preserve the sampling structure. Paired measurements should remain paired. Repeated observations within customers or time-dependent measurements may require a different resampling scheme rather than treating every row as independent.
The percentile method is one bootstrap interval construction. Basic, bias-corrected and accelerated, and studentised intervals use different calculations and assumptions. They are not interchangeable names for the same output.
Three practical examples
- A service analyst estimates uncertainty around a median waiting time using a resampling procedure suited to the collected observations.
- An AI evaluation team resamples evaluation cases to examine uncertainty in an average score, while preserving paired model comparisons on the same cases.
- A researcher compares average outcomes in two independently sampled groups and recalculates their difference for each appropriately constructed bootstrap replicate.
Think of it this way
Think of using a bowl containing copies of the observed values as a small model of the population. Repeatedly drawing from the bowl with replacement shows how the chosen summary varies within that model.
The analogy has limits: the bowl contains only what was observed. It cannot reveal an entirely missing customer group or an unrecorded class of failures, and repeated draws do not create new independent evidence about the real population.
Worked example
Suppose an invented evaluation sample has an average score of 0.72. An analyst generates 10,000 bootstrap resamples and calculates the mean score for each. Assume the resulting 2.5th percentile is 0.68 and the 97.5th percentile is 0.76.
The percentile bootstrap interval is therefore [0.68, 0.76]. These endpoint values are stipulated teaching outputs, not results from a supplied dataset. The interval concerns the population mean represented by the sampling design; it does not claim that 95% of individual evaluation scores lie between those numbers.
Increasing the resample count may make the estimated percentiles more stable, but it does not increase the original number of evaluation cases. If all cases came from one unusually easy source, the interval would not automatically describe performance across every intended source.
Under appropriate conditions, confidence refers to the long-run coverage of the interval-producing procedure, not a guarantee about this particular pair of endpoints.
Illustrative case study
Illustrative fictional case: Birch Search compares two retrieval systems using the same 200 questions. Its first bootstrap script resamples each system's scores independently, breaking the link between scores on the same question.
The analyst changes the procedure to resample question identifiers and carry both systems' scores together, then recalculates the difference for each resample. The report describes the target quantity, pairing, interval method and limitations of the question set.
The team also notices that several questions came from each document and considers whether that dependence changes the appropriate sampling unit. This fictional case does not prescribe one universal bootstrap for all evaluations; it shows why the resampling scheme must reflect how the evidence was collected.
Common misconceptions and mistakes
Do not confuse a percentile bootstrap interval with a range containing 95% of individual observations. Do not assume that 10,000 resamples are equivalent to collecting 10,000 new cases.
Avoid treating every row as independent when observations are paired, clustered or serially related. The simple percentile method can have poor coverage in some settings, including biased or problematic sampling distributions. A narrow interval is not evidence that selection bias, measurement error or missing populations have disappeared.
Business use
Use the interval to communicate uncertainty about a clearly defined quantity, alongside the point estimate and sampling context. Record the sample size, resampling unit, method, resample count and reproducibility settings.
Before a consequential comparison, assess whether the method is suitable for the statistic and data structure. Inspect the bootstrap distribution for unexpected behavior and consider alternative interval methods when justified. An AI-generated report should retain the distinction between uncertainty in an estimate and variability among individual cases, rather than describing every interval as a performance guarantee.
People also ask
Why sample with replacement?
It allows resamples of the same size to differ from the original sample and from one another, producing variation in the recalculated statistic.
Does the percentile interval always contain the original estimate?
Not necessarily. Its endpoints are percentiles of the bootstrap distribution, and the method is not defined by forcing the original estimate into the interval.
Can a percentile bootstrap interval be asymmetric?
Yes. The lower and upper percentile endpoints need not be equally distant from the original estimate.
Check your understanding
A team bootstraps 50 observations one million times. Has it created the same evidence as collecting one million independent observations?
Answer: No. It has reduced simulation noise in its resampling calculation while still relying on the original 50 observations and the assumptions connecting them to the target population.
Related terms
- Bootstrap
- Confidence interval
- Resampling
- Sampling distribution
- Percentile
- Standard error
- Paired data
- BCa interval
Learn next
Read sampling distribution and bootstrap. Identify the statistic and independent sampling unit in one dataset, then describe a valid resample in words. Compare percentiles of the raw observations with percentiles of bootstrap estimates and explain why they answer different questions.
More in Data science and analytics
- Ordinal variable - An ordinal variable is a categorical variable whose categories have a meaningful order.
- Outlier - An outlier is an observation that is unusually different from the other observations or from the pattern expected under a specified model.
- p-hacking - P-hacking is the practice of selecting or changing data analyses in response to their results so that a desired finding appears statistically significant...
- Persistence barcode - A persistence barcode is a collection of intervals that records when topological features appear and disappear as a parameter changes.
- Persistence module - A persistence module is an indexed family of algebraic objects together with maps that describe how information passes from one index to later indices.
- Persistent Betti number - A persistent Betti number counts the independent homology classes present at an earlier filtration parameter that remain nonzero and independent at a later...
Sources
University of Chicago, data-science glossary: https://ds1.datascience.uchicago.edu/glossary.html
Carnegie Mellon, lecture on bootstrap interval constructions: https://www.stat.cmu.edu/~siva/teaching/700/lec14.pdf
SciPy, bootstrap procedure, pairing and interval methods: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html
Wasserman, All of Statistics, bootstrap confidence intervals: https://www.stat.cmu.edu/~brian/valerie/617-2022/0%20-%20books/2004%20-%20wasserman%20-%20all%20of%20statistics.pdf
The numerical endpoints and retrieval case are fictional illustrations.
Previous: Percentile Next: Percept