Percentile Bootstrap Confidence Intervals

From Bootstrap Replicates to an Interval

A bootstrap distribution can do more than describe the variability of a statistic: its quantiles can also be used to form a confidence interval. The percentile method is direct. Resample the observed data with replacement, calculate the statistic in each resample, and take the lower and upper quantiles of those bootstrap statistics. The method can be applied to a statistic such as a sample mean or sample median.

The resulting interval is an estimate of uncertainty about a population parameter, not a range containing a specified proportion of individual observations. Its quality depends on whether resampling the observed data provides a useful approximation to repeated sampling from the population.

Definition: Suppose $T$ is a statistic computed from a sample, and $T_1^*,\ldots,T_B^*$ are its values in $B$ bootstrap resamples. For a confidence level $100(1-\alpha)\%$, a percentile bootstrap confidence interval is $$ \left[q_{\alpha/2}^*,\ q_{1-\alpha/2}^*\right], $$ where $q_{\alpha/2}^*$ and $q_{1-\alpha/2}^*$ are the corresponding lower and upper quantiles of the bootstrap replicates. For a mean, $T=\bar{x}$ estimates the population mean $\mu$; for a median, the sample median estimates the population median.

For example, a 95% interval uses the 2.5th and 97.5th percentiles of the bootstrap statistics. With a finite number $B$, these quantiles are estimated from the sorted replicates. There are several software conventions for locating a quantile between observed values. To keep hand calculations clear, the examples below use the nearest-rank convention: the estimated $p$ quantile is the observation at rank $\lceil pB\rceil$, counting from the smallest value. Software may instead interpolate between neighboring values, so its reported endpoints can differ slightly.

Conditions for Using the Method

The bootstrap does not create new information about the population. It treats the observed sample as a stand-in for that population and resamples from it. Before interpreting an interval, consider how the observations were collected and what was resampled.

Representative sampling: The original observations should provide a reasonable basis for learning about the target population. A biased sample does not become representative when resampled.
Independent observations: For the ordinary one-sample procedure described here, observations should be independently sampled from the population. If the design involves clusters, pairs, or other dependence, the resampling scheme must preserve that structure rather than resample observations as if independent.
Correct resampling unit: Each bootstrap sample has the original sample size $n$, and is drawn with replacement from the $n$ observed values. Calculate the same statistic in every replicate as in the original sample.
A useful empirical approximation: The sample must contain enough information about the population distribution for its resamples to reflect relevant sampling variability. Small or highly unusual samples may not do this well.

In practice, the number $B$ of bootstrap replicates should be large enough that the estimated tail quantiles are reasonably stable. Increasing $B$ reduces simulation noise in the estimated endpoints, but does not correct a poor sample or a method that has inadequate coverage for the population and statistic of interest.

Worked Example: A Percentile Interval for a Mean

Suppose a hypothetical sample of six independent, representative observations records time spent on a task, in minutes: 12, 14, 15, 17, 18, and 24. Construct and interpret a 95% percentile bootstrap interval for the population mean time. Assume 2,000 bootstrap resamples have been generated.

First calculate the sample mean: $$ \bar{x}=\frac{12+14+15+17+18+24}{6} =\frac{100}{6} \approx 16.67\text{ minutes}. $$

For each resample, six values are drawn with replacement from the six observed times, and their mean is calculated. Resampling with replacement allows a value to appear more than once or not at all in a replicate. Suppose the 2,000 bootstrap means, sorted from smallest to largest, have a value of 13.83 minutes at rank 50 and 20.17 minutes at rank 1,950. With the nearest-rank convention, the endpoint ranks for a 95% interval are $$ \left\lceil 0.025(2000)\right\rceil=50, \qquad \left\lceil 0.975(2000)\right\rceil=1950. $$

Thus the interval is $[13.83,\ 20.17]$ minutes. In context, these data and this procedure give a 95% percentile bootstrap confidence interval of 13.83 to 20.17 minutes for the population mean time. The interval is not a claim that 95% of individual task times fall between those endpoints.

Why the Quantiles Can Act as Confidence Limits

A confidence interval is intended to capture the unknown parameter at a stated long-run rate under repeated sampling. The percentile method uses the bootstrap distribution as an approximation to the sampling behavior of the statistic. The connection is most easily seen when the statistic has an approximately symmetric sampling distribution.

Let $\hat{\theta}$ denote the observed estimate of a parameter $\theta$. For a location-type statistic, imagine that the sampling error $\hat{\theta}-\theta$ has a distribution that is approximately symmetric about zero. Bootstrap errors $T^*-\hat{\theta}$ are used to approximate that sampling error. If their central $1-\alpha$ portion extends from $-c$ to $c$, the bootstrap statistics extend approximately from $\hat{\theta}-c$ to $\hat{\theta}+c$. The percentile interval uses these two bootstrap quantiles as its endpoints. Under the approximation, the event that $\theta$ lies between them corresponds to the sampling error lying between $-c$ and $c$, which has probability approximately $1-\alpha$.

\begin{aligned} T^*-\hat{\theta} &\text{ approximately represents } \hat{\theta}-\theta,\\ q_{\alpha/2}^* &\approx \hat{\theta}-c,\\ q_{1-\alpha/2}^* &\approx \hat{\theta}+c,\\ P\!\left(q_{\alpha/2}^*\leq \theta\leq q_{1-\alpha/2}^*\right) &\approx P(-c\leq \hat{\theta}-\theta\leq c) \approx 1-\alpha. \end{aligned}

This is a justification under a simplifying symmetry approximation, not a guarantee that every percentile interval has its advertised coverage. When the bootstrap distribution is strongly skewed or the statistic is biased, its quantiles may not produce well-calibrated limits. That limitation is one reason to examine the bootstrap distribution rather than treating the endpoint calculation as automatic.

Worked Example: A Percentile Interval for a Median

Consider a hypothetical independent sample of five measurements, in milligrams: 3, 4, 5, 7, and 11. The sample median is 5 mg. To see how a bootstrap interval for the population median is formed, consider all $5^5=3125$ possible ordered resamples of size five from the observed values, each equally likely under resampling with replacement.

A bootstrap median equals 3 mg when at least three of the five resampled values are 3. The chance of this event in the bootstrap distribution is $$ P(\text{at least three 3s}) =\binom{5}{3}(0.2)^3(0.8)^2 +\binom{5}{4}(0.2)^4(0.8) +(0.2)^5 =0.0512+0.0064+0.00032 =0.05792. $$ The same calculation applies to 11 mg, which is also selected with probability 0.2 on each draw. Therefore, 3 mg is the 2.5th percentile of the bootstrap median distribution: its cumulative probability already reaches 0.05792 at 3 mg. Likewise, the 97.5th percentile is 11 mg, since the cumulative probability below 11 mg is $1-0.05792=0.94208$, which is below 0.975.

The 95% percentile interval is therefore $[3,\ 11]$ mg. It gives a bootstrap-based interval for the population median, in milligrams. In this small sample, the interval is broad and its endpoints are restricted to observed values. The calculation illustrates the procedure, but does not establish that the interval will achieve 95% coverage in repeated samples.

Confidence Level and Replicate Quantiles

The chosen confidence level determines how much of the bootstrap distribution is retained in the middle. A higher confidence level leaves less probability in each tail and therefore generally produces a wider interval. The interval uses quantiles of the statistic, not the standard error alone.

Worked Example: Comparing 90% and 95% Intervals

Suppose a hypothetical sample of six independent observations is 12, 14, 15, 17, 18, and 24 minutes, as above. A bootstrap run with $B=2{,}000$ resamples gives the following relevant order statistics: rank 100 is 14.50, rank 1,900 is 19.50, rank 50 is 13.83, and rank 1,950 is 20.17. These are illustrative results from the specified resampling procedure.

For a 90% interval, $\alpha=0.10$, so the nearest-rank endpoints use $$ \left\lceil 0.05(2000)\right\rceil=100, \qquad \left\lceil 0.95(2000)\right\rceil=1900. $$ The resulting interval is $[14.50,\ 19.50]$ minutes.

For a 95% interval, the ranks are $$ \left\lceil 0.025(2000)\right\rceil=50, \qquad \left\lceil 0.975(2000)\right\rceil=1950, $$ so the interval is $[13.83,\ 20.17]$ minutes. The 95% interval includes more of the bootstrap distribution and is wider than the 90% interval. Both estimate the same population mean; the difference is the chosen confidence level, not a change in the data or statistic.

Interpretation and Common Pitfalls

A 95% confidence procedure is designed so that, over repeated samples under its assumptions, approximately 95% of intervals constructed by the procedure would contain the fixed population parameter. For one observed sample, the interval gives a range of plausible parameter values according to that procedure. It does not assign a 95% probability to the fixed parameter being in this particular interval.

Using the wrong resampling unit: Resampling individual values independently can break dependence in paired or clustered data. The bootstrap sample must reflect the original study design.
Confusing a confidence interval with a data range: Bootstrap quantiles here summarize resampled statistics, not the percentiles of the original observations.
Assuming more replicates fix everything: A very large $B$ can stabilize estimated quantiles, but cannot repair bias in the original sample, dependence ignored by the resampling, or poor coverage of the percentile method.
Overlooking shape: If the bootstrap distribution is visibly asymmetric, a percentile interval may not have the intended coverage. The quantiles can be calculated correctly even when the method is not well calibrated.

Key takeaway: Specify the statistic and confidence level, resample the observed data with replacement using a design-appropriate scheme, and use the two tail quantiles of the bootstrap statistics as the percentile interval. Interpret the endpoints in the context and units of the population parameter, while remembering that the method relies on the bootstrap distribution being a useful approximation.

Check Your Understanding

Answer each question using the percentile bootstrap method.

For a 90% percentile interval based on $B=1{,}000$ replicates, which two quantiles are used?
In a resample of size $n$, what does “with replacement” allow that sampling without replacement would not?
A 95% percentile interval for a population median is reported as 4.2 to 8.1 grams. What does the interval estimate, and what does it not say about individual measurements?
Why can increasing the number of bootstrap replicates make the estimated endpoints more stable without correcting bias from an unrepresentative sample?
Why might an interval based on bootstrap quantiles be questionable if the bootstrap distribution is strongly skewed?