The Bootstrap Approximation To A Sampling Distribution

Why Resample an Observed Sample?

A sampling distribution describes how a statistic would vary across repeated samples from a population. In practice, the population distribution is usually unknown, so it cannot be used directly to generate those repeated samples. The bootstrap addresses this difficulty by treating the observed sample as an estimated population and repeatedly drawing new samples from it, with replacement.

For example, if a hypothetical insurer has a small observed set of claim amounts, resampling those observations can show how an estimator such as the sample mean might vary when a sample of the same size is drawn again. The resulting distribution is not the exact sampling distribution under the unknown population. It is a data-based approximation to that distribution, conditional on the observations available.

Definition: Given observed data $x_1,\ldots,x_n$, the empirical distribution assigns probability $1/n$ to each observation, counting repeated observed values according to their multiplicities. A bootstrap sample is a sample of size $n$ drawn independently with replacement from this empirical distribution. Recalculate the statistic of interest on each bootstrap sample. The resulting conditional distribution of the recalculated statistic is the bootstrap approximation to its sampling distribution.

The resampling is with replacement: after an observation is selected, it remains available for the next draw. A bootstrap sample can therefore include an observed value more than once and omit some observed values entirely. Each bootstrap sample has the original sample size $n$, unless a different procedure is explicitly defined.

From the Sample to a Bootstrap Distribution

Let $T$ be a statistic computed from the data. For each bootstrap replication $b$, draw $x_1^{*(b)},\ldots,x_n^{*(b)}$ from the empirical distribution and calculate $T^{*(b)}=T(x_1^{*(b)},\ldots,x_n^{*(b)})$. Repeating this process produces values that approximate the distribution of $T$ across repeated samples.

Choose the statistic.
Specify the quantity whose sampling variation matters, such as a sample mean or sample variance.

Define the empirical population.
Give each of the $n$ observed data positions probability $1/n$. If values are tied, their combined probability is proportional to their number of occurrences.

Resample $n$ times with replacement.
Each draw is independent and uses the same empirical probabilities. A new bootstrap sample is generated for each replication.

Recalculate the statistic.
Apply the original definition of $T$ to that bootstrap sample. Do not hold the original statistic fixed or substitute a different denominator.

Summarize the replicates.
The distribution of the recalculated values approximates the sampling distribution. Enumerating all resamples gives the exact bootstrap distribution; a finite set of simulated resamples gives a Monte Carlo approximation to it.

Why the Bootstrap Mean Has a Simple Expectation

For a sample mean, some bootstrap properties follow directly from the empirical distribution. Write the observed sample mean as $\bar{x}$, and let $X^*$ denote one draw from the empirical distribution. Its expected value is the average of the observed values:

E[X^*\mid x_1,\ldots,x_n] =\frac{1}{n}\sum_{i=1}^{n}x_i =\bar{x}.

A bootstrap sample mean is $\bar{X}^*=(X_1^*+\cdots+X_n^*)/n$, where the draws are independent conditional on the observed data. Linearity of expectation then gives the first result below. Independence also means that the conditional variance of the sum is the sum of the individual conditional variances.

Result: For resampling $n$ independent observations with replacement from the empirical distribution, $$ E[\bar{X}^*\mid x_1,\ldots,x_n]=\bar{x}, \qquad \operatorname{Var}(\bar{X}^*\mid x_1,\ldots,x_n) =\frac{1}{n}\left(\frac{1}{n}\sum_{i=1}^{n}(x_i-\bar{x})^2\right). $$ The variance inside parentheses uses divisor $n$, the variance of one draw from the empirical distribution. It is not the usual unbiased sample variance, which uses divisor $n-1$.

To see the variance result, set $s_n^2=n^{-1}\sum_{i=1}^{n}(x_i-\bar{x})^2$. Each bootstrap draw has conditional variance $s_n^2$. Therefore,

\operatorname{Var}(\bar{X}^*\mid x_1,\ldots,x_n) =\operatorname{Var}\left(\frac{1}{n}\sum_{j=1}^{n}X_j^*\ \middle|\ x_1,\ldots,x_n\right) =\frac{1}{n^2}\sum_{j=1}^{n}s_n^2 =\frac{s_n^2}{n}.

This is a justification for the bootstrap mean’s center and spread under its own resampling mechanism. It does not guarantee that the empirical distribution accurately represents the unknown population, especially when the sample is small or unrepresentative.

Worked Example: Exact Bootstrap Distribution of a Sample Mean

A hypothetical pilot study records three values: $0,1,2$. Treat these as the complete observed sample. Find the exact bootstrap distribution of the sample mean for bootstrap samples of size three, and check its mean and variance.

Method. Each draw independently selects one of $0,1,2$, each with probability $1/3$. There are $3^3=27$ ordered bootstrap samples. The sum of the three selected values can range from 0 to 6. Counting ordered samples by their sum gives the following multiplicities:

Bootstrap sum	Bootstrap mean	Number of ordered samples
0	0	1
1	1/3	3
2	2/3	6
3	1	7
4	4/3	6
5	5/3	3
6	2	1

The multiplicities sum to $1+3+6+7+6+3+1=27$, as required. The expected bootstrap mean is the multiplicity-weighted average:

E[\bar{X}^*] =\frac{0(1)+(1/3)(3)+(2/3)(6)+1(7)+(4/3)(6)+(5/3)(3)+2(1)}{27} =\frac{27}{27}=1.

For a variance check, the squared deviations from 1, weighted by their multiplicities, total $1+4/3+2/3+0+2/3+4/3+1=6$. Thus, $\operatorname{Var}(\bar{X}^*)=6/27=2/9$.

The formula gives the same result. The empirical mean is $\bar{x}=1$, and the empirical variance with divisor three is $(1+0+1)/3=2/3$. Consequently, $\operatorname{Var}(\bar{X}^*)=(2/3)/3=2/9$. Both checks agree.

Common mistake avoided: Do not divide the weighted sum of bootstrap means by the number of resamples and then divide by the sample size again. The table already lists means; their weighted average uses the total multiplicity, 27, as its denominator.

Recalculate the Statistic, Not Just the Data

The bootstrap procedure applies to statistics beyond the mean. For every resample, calculate the statistic using its original definition. For example, if the target statistic is the usual sample variance, use divisor $n-1$ in every replicate. The empirical variance formula for the variance of a bootstrap mean does not change the definition of a sample variance calculated from an individual replicate.

Worked Example: Bootstrap Spread and a Sample-Variance Replicate

A hypothetical sample of four recorded amounts is $2,4,8,10$. Use the empirical distribution to find the conditional variance of the bootstrap sample mean for samples of size four. Then calculate the usual sample variance in the particular bootstrap sample $2,4,4,10$.

Step 1: Find the empirical mean and variance. The mean is $\bar{x}=(2+4+8+10)/4=24/4=6$. The squared deviations are $(2-6)^2=16$, $(4-6)^2=4$, $(8-6)^2=4$, and $(10-6)^2=16$, for a total of 40. The empirical variance of one bootstrap draw is $40/4=10$.

Step 2: Find the bootstrap mean variance. A bootstrap mean averages four independent draws from this empirical distribution, so $\operatorname{Var}(\bar{X}^*\mid x_1,\ldots,x_4)=10/4=2.5$. Its conditional standard deviation is $\sqrt{2.5}=1.5811$, rounded to four decimal places.

Step 3: Recalculate the sample variance for the stated replicate. The replicate is $2,4,4,10$, with mean $(2+4+4+10)/4=20/4=5$. Its squared deviations from 5 are $9,1,1,25$, which sum to 36. The usual sample variance is $36/(4-1)=12$. If the statistic had instead been defined as variance with divisor $n$, its value for this replicate would be $36/4=9$.

Check. The bootstrap mean variance, 2.5, is not the sample variance of a particular bootstrap replicate. They are different quantities: one describes variation of the bootstrap means, while the other is a statistic recalculated on one resample.

Common mistake avoided: Using divisor $n$ for a replicate when the requested statistic is the usual sample variance changes the statistic. Here it would give 9 instead of 12.

Worked Example: A Bootstrap Distribution for the Sample Median

A hypothetical sample consists of $1,3,9$. Find the exact bootstrap distribution of the sample median for bootstrap samples of size three. The median is the middle value after the three resampled observations are sorted.

A bootstrap median is 1 exactly when at least two of the three draws equal 1. The probability is the sum of the probabilities of exactly two 1s and exactly three 1s:

P(\operatorname{median}^*=1) ={3\choose2}\left(\frac13\right)^2\left(\frac23\right) +\left(\frac13\right)^3 =\frac{6}{27}+\frac{1}{27} =\frac{7}{27}.

By the same count, the median is 9 exactly when at least two draws equal 9, also with probability $7/27$. The remaining outcomes have median 3, so $P(\operatorname{median}^*=3)=1-7/27-7/27=13/27$. These probabilities sum to $27/27$.

The expected bootstrap median is therefore \[ E[\operatorname{median}^*] =1\left(\frac{7}{27}\right)+3\left(\frac{13}{27}\right)+9\left(\frac{7}{27}\right) =\frac{7+39+63}{27} =\frac{109}{27} \approx 4.0370. \] The observed-sample median is 3. The bootstrap distribution is discrete and, in this example, its expected value differs from the observed median. This illustrates why the statistic must be recalculated in every resample rather than assuming all statistics behave like the sample mean.

Check. The three probabilities $7/27,13/27,7/27$ total one, and the expected value calculation uses each possible median with its correct probability.

Common mistake avoided: A median of 1 does not require all three draws to equal 1. It occurs whenever at least two draws equal 1, so the exactly-two case must be included.

Interpretation and Conditions

The ordinary bootstrap is most directly motivated when observations can be regarded as independent and identically distributed draws from a population. It approximates that population by placing mass only at the observed values. Consequently, the bootstrap cannot generate a value outside the observed sample’s range, and it cannot reproduce population features that the sample failed to capture.

The approximation is conditional on the observed data. It answers a question of the form: “If the empirical distribution were the population, how would this statistic vary across new samples of this size?” That is different from repeated-sample simulation under a specified population model, where samples are generated from the assumed model rather than from the observed data. The earlier material on repeated-sample simulation concerns that model-based setting.

Ordinary resampling may be inappropriate when observations are dependent, when the sample design has important groups or unequal selection probabilities, or when the data were truncated or censored in ways that must be respected. In those settings, the resampling scheme must reflect the data structure and observation process; independent draws from the raw empirical distribution do not automatically do so.

Common Exam Traps

Sampling without replacement. That produces a different distribution and prevents repeated observations. Bootstrap draws are with replacement.
Changing the sample size. In the ordinary bootstrap, each resample has size $n$, matching the observed sample size.
Using the wrong denominator. Distinguish the empirical variance of one draw, which uses divisor $n$, from the usual sample variance statistic, which uses divisor $n-1$.
Assuming every statistic is centered at its observed value. The sample mean has bootstrap expectation $\bar{x}$, but a nonlinear statistic need not have bootstrap expectation equal to its observed value.
Calling the bootstrap distribution the true sampling distribution. It is an approximation based on the observed sample, and its quality depends on how well that sample represents the population and the relevant sampling process.

Key takeaway: The bootstrap turns the observed sample into an estimated population, repeatedly draws samples of the same size with replacement, and recalculates the statistic. The resulting conditional distribution approximates sampling variation; its validity depends on the sample and resampling scheme representing the data-generating process.

Check Your Understanding

For each problem, use resampling with replacement and treat the observed values as the empirical population.

A sample is $2,5,8$. For bootstrap samples of size three, find the conditional expectation and variance of the bootstrap sample mean. (Answer: expectation $5$; variance $2$.)
The observed sample is $1,4$, and each bootstrap sample has size two. Give the possible bootstrap means and their probabilities. (Answer: mean $1$ with probability $1/4$, mean $5/2$ with probability $1/2$, and mean $4$ with probability $1/4$.)
The observed sample is $0,2,5$. For a bootstrap sample of size three, what is the probability that every resampled value is at most 2? (Answer: $(2/3)^3=8/27\approx0.2963$.)
The observed sample is $1,2,7$. For bootstrap samples of size three, find the probabilities that the sample median is 1, 2, or 7. (Answer: $7/27,\ 13/27,\ 7/27$, respectively.)
The observed sample is $3,5,9,11$. For bootstrap samples of size four, calculate the conditional variance of the bootstrap sample mean. (Answer: the empirical variance is $(16+4+4+16)/4=10$, so the bootstrap mean variance is $10/4=2.5$.)

What You'll Learn