Why More Resamples Do Not Always Mean More Information
A bootstrap distribution can look detailed: thousands of replicate statistics may form a smooth histogram, and an interval can be calculated from its percentiles. But the replicates are generated from the observed data. They are not new observations from the population. If the original sample is too small, misses an important kind of outcome, or contains dependent observations treated as independent, resampling can repeat the problem rather than fix it.
The ordinary nonparametric bootstrap is useful when the observations provide a reasonable representation of the population and the resampling scheme respects how the data were collected. This tutorial focuses on situations where those conditions are doubtful. The key question is not simply whether a computer can generate many replicates; it is whether those replicates capture the uncertainty relevant to the population quantity of interest.
What the Ordinary Bootstrap Assumes
Suppose the observed data are \(X_1,\ldots,X_n\), and the statistic of interest is \(T\), such as a sample mean or a sample proportion. The ordinary nonparametric bootstrap forms each replicate by drawing \(n\) observations with replacement from the observed data. It calculates \(T^*\) for each replicate and uses the resulting bootstrap distribution to estimate how \(T\) would vary across samples.
The method treats the observed values as a stand-in for the population. Its usual justification requires, among other things, that the observations are independent and identically distributed (or that the resampling unit is an appropriate independent unit), and that the empirical distribution represents the population well enough for the statistic under study. The statistic also needs to behave regularly enough for the bootstrap distribution to approximate its sampling distribution. These are substantive conditions, not consequences of running a large number of replicates.
A bootstrap interval is based on the variation visible under repeated sampling from the empirical distribution. It does not automatically include uncertainty about data features that never appeared in the sample. Nor does resampling make dependent observations independent. Those limitations matter especially with tiny samples, rare outcomes, statistics at their boundaries, and clustered or sequential data.
Failure Mode 1: Rare Outcomes Missing From the Sample
If a category or event is absent from the observed data, the ordinary bootstrap cannot include it in a replicate. Every bootstrap draw comes from observed values only. A sample with no events therefore produces bootstrap samples with no events, even if the population has a nonzero event rate.
The chance of this problem can be substantial when the event is rare. If the population event probability is \(q\), and \(n\) independent observations are collected, the probability of seeing no events is \((1-q)^n\). This calculation describes a possible weakness in the original sample; it is not something that bootstrap replication can undo.
Worked Example: No Rare Events Observed
Suppose a hypothetical sample of \(n=12\) patients records whether a particular adverse event occurred. None of the 12 patients has the event, so the observed sample proportion is \(\hat{p}=0/12=0\). Consider the ordinary percentile bootstrap interval for the population event proportion.
Bootstrap distribution. Every resampled observation is drawn from the 12 observed zeros. Thus every bootstrap sample also has zero events, and every bootstrap proportion is \(0/12=0\). The bootstrap distribution is concentrated entirely at zero, so its percentile interval is \([0,0]\).
What the data could have missed. If the true event probability were \(q=0.10\), the probability that all 12 observations would nevertheless contain no event is \((1-q)^{12}=0.90^{12}=0.2824\), or about 28.2%. The observed sample is therefore compatible with a nonzero event rate, even though its ordinary bootstrap distribution has no variation.
The interval \([0,0]\) reflects the lack of variation in resamples from this particular empirical distribution; it does not establish that the population event probability is zero. A larger sample or a method designed for rare-event data may be needed. The appropriate choice depends on the design and assumptions, not on the number of bootstrap replicates.
Failure Mode 2: Tiny Samples and Unseen Features
A tiny sample gives the empirical distribution only a few possible values. Bootstrap samples can repeat those values in different combinations, but they cannot reveal unobserved parts of the population distribution. This limitation is especially important when the population is strongly skewed or has a tail that the sample represents poorly.
Worked Example: A Skewed Sample of Three Values
A hypothetical sample of three measurements, in minutes, is \(4, 5, 30\). The target is the population mean. The sample mean is \(\bar{x}=(4+5+30)/3=13\) minutes.
The sample variance and standard deviation are \(s^2=[(4-13)^2+(5-13)^2+(30-13)^2]/(3-1)\) \(=(81+64+289)/2=217\) minutes squared, and \(s=\sqrt{217}=14.73\) minutes. The usual estimate \(s/\sqrt{n}\) of the standard error of the sample mean is \(14.73/\sqrt{3}=8.505\) minutes.
For a bootstrap sample, each draw has probability \(1/3\) of being 4, 5, or 30. The variance of one draw from this empirical distribution is \([(4-13)^2+(5-13)^2+(30-13)^2]/3=434/3=144.7\) minutes squared. The mean of three independent draws from it therefore has variance \(144.7/3=48.22\) minutes squared and bootstrap standard error \(\sqrt{48.22}=6.944\) minutes.
These calculations describe the bootstrap distribution conditional on the three observed values. In particular, the bootstrap cannot generate a value below 4 or above 30 minutes. In \( (2/3)^3=8/27=0.2963\), or about 29.6%, of bootstrap samples, none of the three draws is 30; those replicates omit the only observed value in the high tail. If the population has still larger values that this sample missed, the bootstrap has no way to represent them.
The difference between the two standard-error calculations is not by itself a test that the bootstrap fails. The warning is that a sample of three skewed observations provides very limited information about the population shape and tails. The bootstrap can quantify variation under its empirical stand-in for the population, but cannot certify that the stand-in is adequate.
Failure Mode 3: Statistics at a Boundary
Some statistics have a boundary that cannot be crossed. A sample proportion, for example, must lie between 0 and 1; a variance cannot be negative. If a sample lands at a boundary, bootstrap replicates can be unusually restricted, or even identical, because resampling cannot add values that would move the statistic away from that boundary.
Worked Example: A Zero Sample Variance
Suppose four hypothetical measurements are all 7 units. The sample mean is \(\bar{x}=7\), and the sample variance is \(s^2=[(7-7)^2+(7-7)^2+(7-7)^2+(7-7)^2]/(4-1)=0\) units squared.
Every ordinary bootstrap sample consists of four 7s. Its sample variance is also 0, so the bootstrap distribution of the variance is concentrated at zero. A percentile interval for the population variance based on these replicates would be \([0,0]\) units squared.
Yet a population can have nonzero variation and still produce this sample. For example, consider a hypothetical population that gives probability 0.90 to 7 units and probability 0.05 each to 6 and 8 units. Its mean is 7 units and its variance is \(0.05(6-7)^2+0.05(8-7)^2=0.10\) units squared. The probability of drawing four 7s from this population is \(0.90^4=0.6561\). Thus the sample could occur even when the population variance is positive. The bootstrap’s zero-width result reflects the observed values, not proof that the population has no variation.
A boundary result should prompt a question about what values the sample could have missed. The same reasoning applies when every observed binary outcome is a success or every one is a failure: the bootstrap proportion is fixed at 1 or 0. A narrow or degenerate bootstrap distribution at a boundary is not automatically strong evidence of high precision.
Failure Mode 4: Resampling the Wrong Units
The ordinary bootstrap’s independent draws are appropriate only when the resampling units can be treated as independent observations. Repeated measurements on the same person, observations grouped within clinics, or measurements taken in a time sequence may be dependent. Resampling individual rows as if they were unrelated can break the structure that creates their dependence and misrepresent uncertainty.
Worked Example: Repeated Measurements Within Households
Imagine a hypothetical dataset with two measurements from each of four households. Both readings in each household are the same: the household-level values are 0, 0, 10, and 10 units, each repeated twice. The mean across the eight recorded readings is 5 units. The independent sampling units are the four households, not the eight readings.
If the eight readings are resampled independently, the empirical variance of one draw is \([(0-5)^2+(0-5)^2+(0-5)^2+(0-5)^2+(10-5)^2+(10-5)^2+(10-5)^2+(10-5)^2]/8=25\) units squared. The bootstrap mean of eight draws then has variance \(25/8=3.125\) units squared and standard error \(\sqrt{3.125}=1.768\) units.
If households are resampled as clusters, each bootstrap draw selects a household and carries both of its readings with it. Because the cluster sizes are equal, the mean of the eight readings is also the mean of the four household values. The empirical variance of one cluster-level draw is 25 units squared, so the bootstrap mean across four selected households has variance \(25/4=6.25\) units squared and standard error \(\sqrt{6.25}=2.500\) units.
Here, resampling individual readings gives less estimated variation than resampling the independent clusters, because it treats repeated readings within a household as additional independent information. Cluster resampling respects the sampling units, but four households are still a very small number of independent clusters; using the right unit does not guarantee a reliable interval when the sample is tiny.
A Short Justification: Why a Missing Event Stays Missing
Let \(K\) be the number of observed events among \(n\) observations. Under the empirical distribution, the probability of an event on any bootstrap draw is \(K/n\), because exactly \(K\) of the \(n\) observed values are events. The probability that a bootstrap sample contains no events is therefore
If the original sample has \(K=0\), this probability is \(1\). Every bootstrap sample has zero events. The derivation makes the limitation explicit: the bootstrap reproduces the empirical distribution, and when that distribution assigns zero probability to an outcome, resampling cannot give that outcome positive probability.
How to Respond to a Warning Sign
A warning sign does not mean that all bootstrap methods are invalid. It means the ordinary resampling distribution may not answer the question reliably without further justification. Before interpreting an interval, check the data and the design:
- Check what is absent. Ask whether a rare category, tail value, or meaningful source of variation could have been missed.
- Check the independent unit. Resample whole independent clusters or other design units when individual records are dependent; do not treat repeated observations as independent merely because they occupy separate rows.
- Check boundary estimates. If the observed statistic is at its possible minimum or maximum, examine whether the data could reasonably have missed values away from the boundary.
- Check sample size in context. There is no single sample-size cutoff that makes the ordinary bootstrap reliable for every statistic and distribution. Tiny samples are especially concerning when data are skewed, rare, or highly structured.
- Consider what could add information. More observations, better coverage of rare outcomes, or a model-based approach with explicit assumptions may be more useful than more bootstrap replicates.
Common Mistake: Treating Replicate Count as Sample Size
A common mistake is to reason, “I generated 100,000 bootstrap replicates, so my interval must be precise.” The correction is that the replicate count controls how accurately the computer represents the bootstrap distribution; it does not increase the number of original observations or repair the resampling assumptions. A million resamples of 12 zeros still contain no event, and a million resamples of dependent rows still use the wrong independent unit if the rows are clustered.
Check Your Understanding
For each question, focus on what the observed data allow the bootstrap to represent.
- A sample of 20 observations contains no cases of a rare outcome. What values can the ordinary bootstrap generate for the event count?
- In the sample \(2, 3, 25\), why can the bootstrap not describe population values above 25?
- Four identical observations produce a bootstrap distribution of sample variances concentrated at zero. Does this establish that the population variance is zero? Explain.
- A study records five measurements from each participant. What should be considered before resampling individual measurements?
- What does increasing the number of bootstrap replicates improve, and what does it leave unchanged?