Bootstrap Sampling Distributions and Standard Errors

What the Spread of Bootstrap Statistics Tells Us

A statistic such as a sample mean, median, or proportion is calculated from a sample. If we could repeatedly draw new samples under the same conditions, the statistic would usually vary from sample to sample. This variation is its sampling variability. A statistic’s standard error describes the typical size of that variability, in the statistic’s own units.

Usually, we do not have many genuinely new samples from the population. Bootstrap resampling offers a way to study how a statistic might vary using the data already collected. As covered in bootstrap resampling, each bootstrap sample is drawn with replacement from the observed data and has the same size as the original sample. We calculate the statistic for each resample. A histogram of those replicate statistics displays how they vary.

Definition: Let \(\hat{\theta}\) be a statistic calculated from the original sample, and let \(\hat{\theta}_1^*,\ldots,\hat{\theta}_B^*\) be the values of that statistic from \(B\) bootstrap samples. The bootstrap standard error is the sample standard deviation of those \(B\) replicate statistics:

\widehat{SE}_{\mathrm{boot}}(\hat{\theta}) = \sqrt{ \frac{\displaystyle\sum_{b=1}^{B} \left(\hat{\theta}_b^*-\bar{\theta}^*\right)^2}{B-1} }, \qquad \bar{\theta}^*=\frac{1}{B}\sum_{b=1}^{B}\hat{\theta}_b^*.

Here, \(B\) is the number of bootstrap replicates, and \(\bar{\theta}^*\) is their mean. The bootstrap standard error estimates the sampling variability of \(\hat{\theta}\), using the observed sample as a stand-in for the population.

The replicate statistics—not the individual values in the original data—are the observations whose standard deviation is calculated in this formula. For instance, if the statistic is a median, the standard error is calculated from the bootstrap medians, not from the original data’s standard deviation. The standard error has the same units as the statistic: hours for a mean measured in hours, or proportion units for a sample proportion.

Conditions and Interpretation

For the ordinary bootstrap described here, the resampling scheme should reflect how the data were collected. A basic setting is an independent random sample of observational units from a population: draw \(n\) units with replacement from the \(n\) observed units for every replicate, where \(n\) is the original sample size. The bootstrap treats the empirical distribution—the distribution that assigns equal probability to each observed unit—as an approximation to the population distribution.

This method does not repair a biased or unrepresentative sample. If the original data omit important parts of the population, resampling those data does not bring the missing information back. Nor should observations be resampled independently when the study design makes them dependent. The resampling scheme needs to preserve relevant features of that design.

When these conditions are reasonable, the bootstrap histogram approximates the distribution of the statistic across repeated samples. Its spread is informative about sampling variability. It is not a histogram of the population values or of the original observations. The center and shape may also be worth noticing, but the focus here is the spread as an estimate of standard error.

Worked Example: Standard Error of a Bootstrap Mean

In a hypothetical study, a sample of patients’ waiting times is measured in hours, and the sample mean is the statistic of interest. A spreadsheet produces 20 bootstrap samples, each of the same size as the original sample. The resulting bootstrap means are summarized below. This small number of replicates keeps the arithmetic manageable; in practice, more replicates are generally used.

Bootstrap mean (hours)	Number of replicates
1.5	2
2.0	4
2.5	8
3.0	4
3.5	2

The mean of the 20 bootstrap means is

\bar{\theta}^* = \frac{2(1.5)+4(2.0)+8(2.5)+4(3.0)+2(3.5)}{20} = \frac{50}{20} = 2.5\text{ hours}.

The sum of squared deviations from this bootstrap mean is

\begin{aligned} \sum_{b=1}^{20}(\hat{\theta}_b^*-\bar{\theta}^*)^2 &=2(1.5-2.5)^2+4(2.0-2.5)^2+8(2.5-2.5)^2\\ &\quad+4(3.0-2.5)^2+2(3.5-2.5)^2\\ &=2(1)+4(0.25)+8(0)+4(0.25)+2(1)\\ &=6\text{ hours}^2. \end{aligned}

Using \(B-1=19\) in the denominator, the estimated standard error is

\widehat{SE}_{\mathrm{boot}}(\bar{x}) = \sqrt{\frac{6}{19}} \approx 0.5619\text{ hours}.

The histogram would have its tallest bar at 2.5 hours, with fewer replicates at the more distant values. Its spread corresponds to an estimated standard error of about 0.562 hours. In context, the bootstrap estimates that the sample mean waiting time would typically vary by roughly 0.562 hours across repeated samples under the conditions represented by this resampling process. This is an estimate of variability, not a guarantee that every new sample mean will be within 0.562 hours of the population mean.

The ordinary bootstrap conditions are appropriate for this calculation only if the original waiting-time observations can reasonably be treated as an independent sample and the resampling scheme matches the study design. The replicate count is small, so the estimate is also subject to noticeable Monte Carlo variation.

Why the Replicate Standard Deviation Estimates Standard Error

The definition follows the logic of repeated sampling. A standard error is the standard deviation of a statistic’s sampling distribution: the distribution of its values over repeated samples from the same population. Since that distribution is not usually available in full, bootstrap resampling approximates it by treating the observed data as an empirical population and repeatedly drawing samples from it.

Each bootstrap statistic \(\hat{\theta}_b^*\) is one draw from this approximate distribution, conditional on the data that were observed. The average of the \(B\) replicates estimates the center of that distribution, and their sample standard deviation measures its spread. Specifically, subtracting \(\bar{\theta}^*\) gives each replicate’s deviation from the replicate average; squaring and averaging those deviations measures their variance, and taking the square root returns the spread to the statistic’s units. The denominator \(B-1\) is the usual sample-standard-deviation adjustment when the center is estimated from the same replicate values.

This is an approximation, not a claim that bootstrap samples are new observations from the original population. The empirical distribution is based on one observed sample, and the bootstrap’s usefulness depends on that sample and resampling scheme being reasonable representations of the sampling process. A larger \(B\) gives a more stable numerical summary of the bootstrap distribution, but it cannot correct limitations in the original data.

Worked Example: Standard Error of Bootstrap Medians

Suppose a hypothetical sample of patient waiting times is used to estimate the median waiting time. Ten bootstrap samples produce medians, in minutes, of 8, 9, 9, 10, 10, 10, 11, 11, 12, and 12. The original sample median is 10 minutes. The ten replicate medians are small in number, so treat the result as an arithmetic illustration rather than a precise estimate.

First find the mean of the replicate medians:

\bar{\theta}^* = \frac{8+9+9+10+10+10+11+11+12+12}{10} = \frac{102}{10} = 10.2\text{ minutes}.

The sum of squared deviations from 10.2 minutes is

\begin{aligned} \sum_{b=1}^{10}(\hat{\theta}_b^*-\bar{\theta}^*)^2 &=(8-10.2)^2+2(9-10.2)^2+3(10-10.2)^2\\ &\quad+2(11-10.2)^2+2(12-10.2)^2\\ &=4.84+2.88+0.12+1.28+6.48\\ &=15.60\text{ minutes}^2. \end{aligned}

Therefore,

\widehat{SE}_{\mathrm{boot}}(\text{median}) = \sqrt{\frac{15.60}{10-1}} = \sqrt{1.7333} \approx 1.317\text{ minutes}.

The replicate medians vary by an estimated standard error of about 1.317 minutes. The calculation uses the spread of the bootstrap medians, because the target statistic is the median. As in the first example, this interpretation assumes that the original observations and the ordinary resampling scheme reasonably represent independent sampling from the population of interest.

Reading a Histogram Without Confusing Its Message

A bootstrap-statistic histogram is a visual summary of the replicate statistics. A narrow spread indicates that the statistic changed relatively little across the resamples; a wider spread indicates greater estimated sampling variability. The horizontal axis uses the statistic’s units. A histogram of bootstrap proportions, for example, is expressed in proportion units, even if the original data are counts of successes and failures.

A histogram helps reveal the overall scale and pattern of the replicate variation, but it does not replace the standard-error calculation. The histogram’s appearance depends partly on choices such as bin width, while the formula uses the actual replicate values. To estimate the bootstrap standard error, calculate the sample standard deviation of the replicates rather than trying to infer a precise number from the bar heights alone.

The number of bootstrap replicates also matters. With few replicates, the histogram may look rough and the estimated standard error may change noticeably if the resampling is repeated. Increasing \(B\) generally reduces this Monte Carlo noise by giving a more stable estimate of the spread of the bootstrap distribution. It does not make the original sample larger, improve its representativeness, or eliminate the sampling uncertainty in the original statistic.

Worked Example: Standard Error of a Bootstrap Proportion

In a hypothetical sample of 10 people, 5 report a particular symptom. The sample proportion is \(5/10=0.5\). Ten bootstrap samples of size 10 are drawn with replacement from these observed responses. Their bootstrap proportions are 0.3, 0.4, 0.4, 0.5, 0.5, 0.5, 0.6, 0.6, 0.7, and 0.7. Each is a multiple of 0.1 because each replicate contains 10 people.

The mean of these proportions is

\bar{\theta}^* = \frac{0.3+2(0.4)+3(0.5)+2(0.6)+2(0.7)}{10} = \frac{5.2}{10} = 0.52.

The squared deviations add to

\begin{aligned} \sum_{b=1}^{10}(\hat{\theta}_b^*-\bar{\theta}^*)^2 &=(0.3-0.52)^2+2(0.4-0.52)^2+3(0.5-0.52)^2\\ &\quad+2(0.6-0.52)^2+2(0.7-0.52)^2\\ &=0.0484+0.0288+0.0012+0.0128+0.0648\\ &=0.156. \end{aligned}

Thus the bootstrap standard error is

\widehat{SE}_{\mathrm{boot}}(\hat{p}) = \sqrt{\frac{0.156}{9}} \approx 0.1317.

The estimated standard error is about 0.132 in proportion units, or about 13.2 percentage points. It describes the spread of the bootstrap proportions for this sample and resampling setup. It is not the probability that the population proportion equals 0.5, nor is it a statement that every new sample proportion will differ from the population proportion by exactly 13.2 percentage points. The original sample is small, and only ten replicates were used, so this numerical estimate is illustrative and would benefit from many more replicates.

Common Mistakes and Practical Checks

Using the original data’s standard deviation instead of the replicate statistics. The bootstrap standard error is the standard deviation of the bootstrap values of the statistic. For bootstrap medians, calculate the spread among the medians; for bootstrap means, calculate it among the means.
Calling the bootstrap histogram a distribution of the raw data. Each plotted value is a statistic from one resample, not one person’s measurement. The histogram approximates the sampling distribution of the statistic.
Treating the standard error as an exact bound on error. A standard error summarizes typical variability; it does not guarantee how far any particular estimate is from a population value.
Assuming more replicates fix a poor sample. A large \(B\) can stabilize the calculation, but repeated resampling of unrepresentative data remains repeated resampling of unrepresentative data.
Ignoring the statistic’s units. The standard error of a mean measured in hours is in hours. A standard error for a proportion is in proportion units, which can also be expressed as percentage points.

Key takeaway: The spread of bootstrap replicate statistics estimates how much the statistic would vary across repeated samples, conditional on the observed data and the resampling scheme. Calculate that spread as the sample standard deviation of the replicates, interpret it in context and in the statistic’s units, and remember that its quality depends on the original sample and design.

Check Your Understanding

Use the definitions and examples above to answer these questions.

A bootstrap histogram contains 500 values, each the median from one resample. Which values should be used to calculate the bootstrap standard error?
A statistic is measured in kilograms. What units should its bootstrap standard error have, and why?
Explain what a relatively wide bootstrap-statistic histogram suggests about the statistic’s estimated sampling variability.
Why does increasing the number of bootstrap replicates not correct an unrepresentative original sample?
For 10 bootstrap statistics, the sum of squared deviations from their replicate mean is 18. What is their bootstrap standard error?