Bootstrap Resampling and the Empirical Distribution

Why Resample the Data We Have?

A statistic such as a sample mean or median describes the data collected, but it can change if a different sample is collected. In many settings, the population is too large to measure completely, so we cannot directly see how much a statistic would vary across repeated samples. The bootstrap provides a way to study that variation by treating the observed data as a stand-in for the population.

The central idea is simple: make repeated samples from the data already observed, allowing the same observation to be selected more than once. Each resample has the same size as the original sample. We calculate the statistic of interest for each resample and examine how those results vary. These resamples are not new observations from the real population; they are generated from a model built from the observed data.

The Empirical Distribution as a Stand-In

Suppose the observed values are \(x_1,x_2,\ldots,x_n\). The empirical distribution says that one draw has probability \(1/n\) of selecting each position in this list. A value appearing several times therefore has a greater chance of being selected than a value appearing once. For example, if the data are \(2,2,5,9\), a draw has probability \(2/4\) of being 2, and probability \(1/4\) each of being 5 or 9.

“Draw with replacement” means that after selecting a value, we put it back before the next draw. Thus each draw has the same empirical distribution, and the same observed position can be selected repeatedly. A bootstrap sample can contain duplicates and can omit some original observations. It cannot contain a value that was absent from the original data.

The empirical distribution is also a compact way to describe the observed data as a probability model. For a threshold \(x\), its cumulative probability is the fraction of observed values at most \(x\):

where \(I(x_i\leq x)\) equals 1 when \(x_i\leq x\) and 0 otherwise. This formula simply counts the observations at or below \(x\), then divides by \(n\).

How a Bootstrap Replicate Is Generated

For an ordinary bootstrap, take \(n\) draws from the empirical distribution, independently and with replacement. This gives a resample of size \(n\), even though some original observations may appear multiple times. Apply the statistic of interest to that resample in exactly the same way as to the original data.

For a sample of size \(n\), there are \(n^n\) possible ordered bootstrap samples: each of the \(n\) draws has \(n\) possible observed positions. Every particular ordered sequence has probability \((1/n)^n\) under the empirical model. In practice, it is usually more convenient to generate many random resamples than to list all possible sequences.

Why the Empirical Distribution Can Approximate Uncertainty

The population distribution is unknown, which is why the original sample was collected. If the observations provide a reasonable representation of that population, their empirical distribution can serve as an approximation to it. The bootstrap then repeats the sampling mechanism using this estimated distribution in place of the unknown population distribution.

One short justification comes from looking at the empirical probability below a fixed value \(x\). If the original observations are independent draws from a population with cumulative distribution function \(F(x)\), then each indicator \(I(X_i\leq x)\) has expectation \(F(x)\). By linearity of expectation,

So the empirical fraction at or below \(x\) is, across repeated original samples, centered on the corresponding population probability. The bootstrap uses this empirical distribution as a practical estimate of the population distribution. Repeating draws from it provides a way to approximate how a statistic would vary if sampling were repeated under that estimated model. This argument explains the idea; it does not guarantee that the approximation is accurate for every statistic, data set, or sampling design.

The key assumptions for the ordinary procedure are that the original observations are reasonably representative of the population relevant to the question and that they can be treated as independent, identically distributed observations. The resampling must also preserve the observational unit and the structure relevant to the statistic. A biased convenience sample does not become representative merely because it is resampled many times. Likewise, if observations are naturally paired, clustered, or selected using a complex design, drawing individual rows independently may fail to preserve important structure.

What the Bootstrap Does—and Does Not—Tell Us

A collection of bootstrap replicates is a model-based approximation to the statistic’s repeated-sampling behavior. It reflects the observed values and their relative frequencies. If the original sample is small, unusual features of the population may not appear in the data, and the bootstrap cannot recreate them. If an extreme value is observed only once, for example, it has empirical probability \(1/n\) on each draw; if an important population value is never observed, it has probability zero under the empirical distribution.

The number of resamples and the quality of the original sample answer different questions. Generating more resamples can make a computer’s description of the empirical model more stable, but it does not add new information about the real population. A very large number of bootstrap replicates from a poor or unrepresentative sample can still provide a poor approximation to population uncertainty.

The basic idea is useful because it applies to many statistics: means, medians, and other quantities calculated from data. The specific ways to summarize bootstrap replicates and construct inferential results require additional methods. Here, the essential distinction is between the original sample, the empirical distribution defined by that sample, and the bootstrap samples drawn from that distribution.

Common Mistake

• Wrong reasoning: “A bootstrap sample is a new independent sample from the population, so repeated resampling gives us more real data.”
• Correction: A bootstrap sample is drawn from the empirical distribution of the original observations, not from the population itself. It can help approximate variation under that empirical model, but it does not add observations or repair a sample that poorly represents the population.

A second common error is to sample without replacement. That produces a rearrangement or subset of the observed data, not the ordinary bootstrap described here. Replacement is what allows some observations to recur while others are omitted, creating variation among resamples.