Empirical Bootstrap Samples

Why Resample the Observed Data?

A statistic such as a sample mean or median changes from sample to sample. Its sampling distribution describes that variation, but the population distribution is usually unknown. The nonparametric bootstrap approximates the sampling distribution by treating the observed data as a stand-in for the population and repeatedly drawing new samples from it.

A bootstrap sample has the same size as the original sample and is drawn with replacement. Thus an observation can appear more than once in a bootstrap sample, while another observed value may not appear at all. A bootstrap statistic is calculated from each resample; it is not a new observation from the original population.

Definition: Given observed data $x_1,\ldots,x_n$, the empirical distribution assigns probability $1/n$ to each observation index. Equivalently, its cumulative distribution function is $F_n(t)=n^{-1}\sum_{i=1}^n I(x_i\le t)$. A nonparametric bootstrap sample $X_1^*,\ldots,X_n^*$ consists of $n$ independent draws from this empirical distribution, conditional on the observed data. The bootstrap distribution of a statistic $T$ is the conditional distribution of $T^*=T(X_1^*,\ldots,X_n^*)$.

The index-based definition matters when observed values are tied. If the data are $1,1,2$, the empirical distribution puts probability $2/3$ on the value 1 and $1/3$ on the value 2. One can think of drawing one of the three original indices uniformly; the two indices with value 1 produce the same numerical draw.

The bootstrap does not assume a particular parametric form for the population distribution. Its usual justification is that the empirical distribution estimates the population distribution, so resampling from it can approximate the effect of drawing another sample from that population. This approximation is most directly motivated when the observations are independent and identically distributed. Ordinary resampling does not preserve dependence among time-ordered or clustered observations.

Bootstrap Distributions of Means

For a mean, the bootstrap has a useful exact expectation and variance conditional on the data. Let $\bar{x}=n^{-1}\sum_{i=1}^n x_i$, and define the empirical variance using denominator $n$ as $s_n^2=n^{-1}\sum_{i=1}^n(x_i-\bar{x})^2$. Since each bootstrap draw has mean $\bar{x}$ and variance $s_n^2$, independence of the draws gives the following result.

Result: Conditional on the observed data, $E_*(\bar{X}^*)=\bar{x}$ and $\operatorname{Var}_*(\bar{X}^*)=s_n^2/n$. Here $E_*$ and $\operatorname{Var}_*$ refer to expectation and variance over bootstrap resampling, holding the original data fixed.

To see why, write the bootstrap mean as the average of $n$ independent draws from the empirical distribution. Each draw has conditional expectation $\bar{x}$, so the expectation of their average is also $\bar{x}$. Each draw has conditional variance $s_n^2$; the variance of the average of $n$ independent draws is therefore $n s_n^2/n^2=s_n^2/n$. This is a statement about the bootstrap distribution, not a claim that its values equal the original sample mean.

Worked Example: An Exact Bootstrap Distribution for a Mean

A hypothetical sample of three policy-processing times, in days, is $2,4,8$. Find the exact bootstrap distribution of the mean for resamples of size 3, and calculate its bootstrap expectation and variance. Use all ordered resamples, each of which has probability $1/27$.

Method. Each of the three draws can be 2, 4, or 8, so there are $3^3=27$ equally likely ordered resamples. It is convenient to group them by the sum of their three values; the bootstrap mean is that sum divided by 3.

The possible sums and their counts are as follows. For sums 6, 8, 10, and 12 using only 2s and 4s, the counts are respectively 1, 3, 3, and 1. With one 8, the sums 12, 14, and 16 have counts 3, 6, and 3. With two 8s, sums 18 and 20 each have count 3; three 8s give sum 24 with count 1. Combining the two ways to obtain sum 12 gives count 4.

\begin{array}{c|rrrrrrrrr} \text{Sum} & 6&8&10&12&14&16&18&20&24\\ \text{Count} & 1&3&3&4&6&3&3&3&1 \end{array}

The counts total $1+3+3+4+6+3+3+3+1=27$, as required. Therefore the bootstrap mean takes each listed sum divided by 3, with probability equal to its count divided by 27. For example, $P_*(\bar{X}^*=4)=4/27$, because a sum of 12 occurs in 4 ordered resamples.

The original mean is $\bar{x}=(2+4+8)/3=14/3$. The exact bootstrap expectation is $\{6(1)+8(3)+10(3)+12(4)+14(6)+16(3)+18(3)+20(3)+24(1)\}/(3\cdot27)=14/3$. For the variance formula, $s_n^2=\{(2-14/3)^2+(4-14/3)^2+(8-14/3)^2\}/3=(64/9+4/9+100/9)/3=56/9$. Hence $\operatorname{Var}_*(\bar{X}^*)=(56/9)/3=56/27$ days squared.

Check. The exact distribution has 27 equally likely resamples and its mean is $14/3$, matching the original mean. The variance $56/27$ also follows from the empirical-variance result. The common mistake is to treat the three observed values as a bootstrap sample that must contain one of each; that would exclude valid resamples such as $(2,2,2)$.

Bootstrap Medians and Order Statistics

For an odd resample size $n$, the sample median is the middle order statistic, $X^*_{((n+1)/2)}$, where the observations are sorted from smallest to largest. A useful way to find its distribution is to count how many bootstrap draws fall at or below a threshold. The same approach works for any order statistic.

Result: Let $x_{(1)}<\cdots<x_{(r)}$ denote the distinct observed values, and let $m_j=\#\{i:x_i\le x_{(j)}\}$. For a bootstrap sample of size $n$, the number of draws at or below $x_{(j)}$ has a $\operatorname{Bin}(n,m_j/n)$ distribution. Consequently, for the $k$th bootstrap order statistic, $$ P_*(X^*_{(k)}\le x_{(j)})=P\{\operatorname{Bin}(n,m_j/n)\ge k\}. $$

The event $X^*_{(k)}\le x_{(j)}$ occurs exactly when at least $k$ of the $n$ resampled observations are no greater than $x_{(j)}$. Each draw has probability $m_j/n$ of meeting that condition, independently of the other draws. The count is therefore binomial, which proves the result. The count $m_j$, not the distinct-value index $j$, determines the probability when data include ties.

Worked Example: A Bootstrap Median with Tied Observations

A hypothetical sample of three claim-review durations, in hours, is $0,0,10$. Find the exact distribution of the bootstrap median for resamples of size 3.

Each bootstrap draw is 0 with probability $2/3$ and 10 with probability $1/3$. For three observations, the median is the second order statistic. It equals 0 if at least two draws are 0. If $N_0$ is the number of zeros, then $N_0\sim\operatorname{Bin}(3,2/3)$, so $P_*(\text{median}=0)=P(N_0\ge2)=\binom32(2/3)^2(1/3)+(2/3)^3=12/27+8/27=20/27$.

The median equals 10 if at least two draws are 10. Thus $P_*(\text{median}=10)=\binom32(1/3)^2(2/3)+(1/3)^3=6/27+1/27=7/27$. These probabilities sum to $20/27+7/27=1$, so the distribution is complete. The original sample median is 0, but the bootstrap median is 10 in $7/27$ of resamples.

Check. The result respects the repeated value in the original data: each draw has probability $2/3$ of being zero. Treating the three distinct-value positions as though they had equal probability would give the wrong distribution.

Bootstrap Quantiles When Values Are Tied

A quantile convention must be specified before calculating a bootstrap quantile. Here, for $0<p<1$, use the inverse empirical distribution convention: the sample $p$-quantile is the $k$th order statistic, where $k=\lceil np\rceil$. Other software conventions can interpolate between order statistics, so they need not produce the same statistic.

Worked Example: The 75th Percentile with Ties

A hypothetical set of four recorded repair costs, in hundreds of dollars, is $1,1,2,5$. For bootstrap samples of size 4, find the distribution of the 75th-percentile statistic using $k=\lceil4(0.75)\rceil=3$.

The distinct observed values are 1, 2, and 5. Their cumulative counts are $m_1=2$, $m_2=3$, and $m_3=4$. Thus the probabilities of a single bootstrap draw being at or below those values are $1/2$, $3/4$, and 1.

For the third order statistic, the probability it is at most 1 is $P\{\operatorname{Bin}(4,1/2)\ge3\}=\{\binom43+\binom44\}/16=5/16=80/256$. The probability it is at most 2 is $P\{\operatorname{Bin}(4,3/4)\ge3\}=\binom43(3/4)^3(1/4)+(3/4)^4=27/64+81/256=189/256$. It is at most 5 with probability 1.

Subtracting consecutive cumulative probabilities gives the probability at each possible value: $P_*(X^*_{(3)}=1)=80/256$, $P_*(X^*_{(3)}=2)=189/256-80/256=109/256$, and $P_*(X^*_{(3)}=5)=1-189/256=67/256$. The probabilities sum to $(80+109+67)/256=1$.

Check and common trap. The second distinct value is 2, but three of the four observations are at or below 2. Its binomial probability is therefore $3/4$, not $2/4$. Substituting the distinct-value index for the cumulative count would give an incorrect answer whenever ties change that count.

Using Bootstrap Samples Carefully

The bootstrap distribution is a practical approximation to the sampling distribution, not a guarantee that the approximation is accurate in every setting. It can be computed exactly for small samples, as above, or approximated by generating many resamples and recording the statistic from each. The examples here use exact probabilities to make the resampling logic visible.

Keep the target statistic and the resampling rule separate. First specify the original data, the resample size, and the statistic—including the quantile convention if relevant. Then sample $n$ observations with replacement from the empirical distribution and recalculate that same statistic. For order statistics, use the cumulative number of original observations at or below each distinct value. For means, repeated observations contribute repeatedly, just as they do in any resampled statistic.

With replacement means some observed values may repeat and others may be absent.
Conditional on the data means the bootstrap distribution treats the observed sample as fixed.
Ties accumulate probability. A value observed several times receives that many shares of the empirical probability.
A bootstrap distribution is not itself an interval. It describes resampled statistics; constructing and interpreting bootstrap intervals is a separate step.

These calculations are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus because they require interpreting resampling rules, tracking empirical probabilities, and evaluating statistics under repeated sampling.

Check Your Understanding

For each problem, treat the bootstrap sample size as equal to the original sample size and use sampling with replacement. For quantiles, use the inverse empirical distribution convention $k=\lceil np\rceil$.

A sample of three hypothetical settlement durations is $3,6,9$ days. What are the bootstrap mean’s expectation and variance? Use the empirical variance with denominator $n$. (Answer: $E_*(\bar{X}^*)=6$ days and $\operatorname{Var}_*(\bar{X}^*)=2$ days squared.)
A sample is $2,2,8$. What is the probability that a size-3 bootstrap sample has median 2? (Answer: $20/27$.)
For data $1,4,4,10$, find the probability that the bootstrap second order statistic is at most 4. (Answer: $243/256$.)
A sample is $0,4$. Give the exact bootstrap mean distribution for resamples of size 2, and its variance. (Answer: Means $0,2,4$ have probabilities $1/4,1/2,1/4$, respectively; the variance is $2$.)
For data $1,1,3,7$, find the probability that the bootstrap third order statistic is at most 3. (Answer: $189/256$.)

What You'll Learn