What a Bootstrap Histogram Can Tell You
A bootstrap analysis produces many estimates by repeatedly resampling the observed data and recalculating the statistic. The resulting histogram is more than a picture of uncertainty: its center can indicate estimated bias, and its shape can reveal asymmetry, multiple clusters, or sensitivity to particular observations. These features matter because a confidence interval may rely on the bootstrap distribution’s location and tails.
The histogram is conditional on the data you observed. It is not a direct picture of the unknown parameter, nor does it show every source of uncertainty in the study. Its usefulness depends on how well the resampling procedure represents the data-generating process and on whether the statistic can be sensibly calculated across resamples.
The bootstrap standard error \(SE\) describes the spread of the bootstrap replicates. Bias and spread answer different questions: bias concerns how far the distribution’s center is from \(\hat{\theta}\), while spread concerns how variable the statistic is across resamples. A small bias compared with \(SE\) may be practically modest; a bias of the same size as, or larger than, \(SE\) deserves more attention. Neither comparison alone proves that an interval is valid.
Center and Estimated Bias
For a roughly symmetric bootstrap histogram, compare its center with the vertical reference at \(\hat{\theta}\). The mean of the replicates is used in the estimated bias formula, although the median can also help describe a skewed distribution’s typical location. A histogram centered noticeably above \(\hat{\theta}\) has positive estimated bootstrap bias; one centered below it has negative estimated bootstrap bias.
Worked Example: Estimating Bootstrap Bias
Suppose a hypothetical sample gives an estimated mean waiting time of \(\hat{\theta}=12.0\) minutes. In \(B=1000\) bootstrap resamples, the bootstrap estimates sum to 12,400 minutes, so their mean is \(12{,}400/1000=12.4\) minutes. The bootstrap standard error is \(1.8\) minutes.
The estimated bootstrap bias is $$ \widehat{\operatorname{bias}}_{\mathrm{boot}} = 12.4 - 12.0 = 0.4 \text{ minutes}. $$
The bootstrap distribution’s mean is 0.4 minutes above the original estimate. Relative to the bootstrap standard error, this is \(0.4/1.8\approx0.222\), so the estimated shift is small compared with the replicate-to-replicate spread in this example. This comparison describes the bootstrap output; it does not establish that the true bias is exactly 0.4 minutes or that it can be ignored in every analysis.
A finite number of resamples adds simulation variation to summaries such as the bootstrap mean and quantiles. If the histogram looks ragged or a reported feature depends on only a few extreme replicates, the number of resamples may be too small for a stable visual assessment. Increasing the number of resamples can reduce this simulation noise, but cannot repair a mismatch between the resampling scheme and the study design.
Skewness and Unequal Tails
A bootstrap distribution is skewed when one tail extends farther than the other. In a right-skewed distribution, a relatively long tail stretches toward larger estimates; in a left-skewed distribution, it stretches toward smaller estimates. The mean is often pulled toward the longer tail, so it can differ appreciably from the median. Report the direction of skewness as well as the center: saying only that a distribution is “not symmetric” leaves out useful information.
Skewness matters for interval interpretation because the two tails may not contribute equal amounts of distance on either side of the estimate. A percentile interval uses lower and upper quantiles of the bootstrap distribution, so its endpoints need not be equally far from \(\hat{\theta}\). Unequal distances are not, by themselves, evidence of a calculation error. They reflect the distribution’s asymmetry, though the histogram alone does not establish that the interval has the desired coverage.
Worked Example: Reading a Right-Skewed Distribution
Suppose a hypothetical sample gives an estimated median recovery time of \(\hat{\theta}=5.0\) days. The bootstrap replicate median is 4.6 days and the replicate mean is 5.3 days. The 2.5th and 97.5th percentiles of the replicates are 2.1 and 8.4 days.
The mean exceeds the median, and the upper endpoint is farther from the sample estimate than the lower endpoint: $$ 5.0 - 2.1 = 2.9 \text{ days}, \qquad 8.4 - 5.0 = 3.4 \text{ days}. $$
These summaries are consistent with a right-skewed distribution: its upper tail extends farther, and its mean is above its median. The interval from 2.1 to 8.4 days is asymmetric around 5.0 days. If used as a 95% percentile bootstrap interval, it should be reported in context as an interval for the population median recovery time, in days. Under repeated use of the stated procedure and its assumptions, such intervals are intended to cover the parameter about 95% of the time; this particular interval does not mean there is a 95% probability that the fixed parameter lies between 2.1 and 8.4 days.
A short derivation shows why skewness can also make a basic bootstrap interval asymmetric. Let \(q_L\) and \(q_U\) be lower and upper quantiles of the bootstrap estimates. The basic interval uses the approximation that the bootstrap error \(\hat{\theta}^{*}-\hat{\theta}\) represents the sampling error \(\hat{\theta}-\theta\). Reversing the error’s sign reverses the quantile order: $$ q_L \leq \hat{\theta}^{*} \leq q_U \quad\Longrightarrow\quad q_L-\hat{\theta} \leq \hat{\theta}^{*}-\hat{\theta} \leq q_U-\hat{\theta} $$ $$ \hat{\theta}-\theta \approx \hat{\theta}^{*}-\hat{\theta} \quad\Longrightarrow\quad 2\hat{\theta}-q_U \leq \theta \leq 2\hat{\theta}-q_L. $$ Thus, the basic interval reflects the bootstrap quantiles around \(\hat{\theta}\), swapping their order. If \(q_L\) and \(q_U\) are not equally spaced around the estimate, the reflected endpoints will not be equally spaced either. This reasoning explains the interval’s construction; it does not guarantee the approximation is good.
Multiple Modes and Outlier Influence
A multimodal bootstrap histogram has two or more distinct peaks. A single estimate near the middle of two peaks may not describe a typical resample well: few replicates may fall near that middle, while many cluster around separate values. The clusters can arise when different kinds of resamples produce different estimates—for example, when a small number of observations strongly affect a statistic. A histogram cannot by itself identify the cause. Examine the data and the resampling setup before assigning an explanation.
In regression, an observation is influential when removing it substantially changes a fitted result, such as the estimated slope. Being unusual in the response direction is not enough to establish influence: the effect depends on how the observation sits relative to the rest of the data, including its predictor value. A leave-one-out comparison is a direct diagnostic. It measures sensitivity of the estimate to deleting one observation, not the probability that the observation is erroneous.
Worked Example: Checking a Regression Slope for Influence
Consider a hypothetical four-point data set, with predictor \(x\) measured in predictor units and response \(y\) in outcome units:
| \(x\) | \(y\) |
|---|---|
| 0 | 0 |
| 1 | 0 |
| 2 | \(23/14\) |
| 3 | \(247/42\) |
For the full data, \(\bar{x}=1.5\), \(\bar{y}=79/42\), and the least-squares slope is the centered cross-product divided by the centered sum of squares: $$ \hat{\beta}_1 = \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} = \frac{135/14}{5} = \frac{27}{14} = \frac{81}{42} \approx 1.929 $$
outcome units per predictor unit. Removing the point with \(x=3\) leaves \(x=0,1,2\), for which \(\bar{x}=1\), \(\bar{y}=23/42\), the centered cross-product is \(23/14\), and the centered sum of squares is \(2\). The slope without that point is therefore $$ \hat{\beta}_{1,(-3)} = \frac{23/14}{2} = \frac{23}{28} \approx 0.8214 $$
outcome units per predictor unit. The leave-one-out change, defined here as the full slope minus the slope without the point, is calculated before rounding: $$ \frac{81}{42}-\frac{23}{28} = \frac{54}{28}-\frac{23}{28} = \frac{31}{28} \approx 1.107 $$
outcome units per predictor unit. This is a substantial change relative to the full slope, so the point is influential for this fitted slope in this small constructed data set. The change is a sensitivity measure, not proof that the observation is wrong. This four-point calculation illustrates influence; it is not used to claim a particular bootstrap histogram for these data.
Now suppose a separate, larger hypothetical regression sample has 2,000 valid bootstrap slope estimates with two clear peaks, one near 0.8 and another near 2.0 outcome units per predictor unit. If inspection shows that resamples containing a particular high-leverage observation tend to form the higher peak, that supports a connection between the observation and the distribution’s shape. A percentile interval may span the gap between peaks even though few replicates lie there. Reporting just the interval endpoints would conceal this feature; the histogram and the sensitivity check are important context for interpreting the result.
A Practical Reading Sequence
Use a consistent sequence when examining a bootstrap histogram. The aim is not to diagnose every irregular shape with a single label, but to connect what is visible to what the summaries and interval can reasonably say.
Compare \(\hat{\theta}\) with the bootstrap distribution’s center. If calculating a bias estimate, state whether it uses the replicate mean and report its units.
Look for symmetry, a longer tail, or extreme replicates. Compare tail behavior with the interval endpoints rather than assuming equal distances around the original estimate.
Ask whether the histogram has distinct peaks and whether the statistic’s value changes according to which observations occur in a resample.
For a suspected influential observation, compare the statistic with and without it. Describe the change and its units; do not treat a single cutoff as a universal definition of influence.
Report the interval and its method, then mention notable asymmetry or multiple modes. Do not infer that the interval is reliable merely because software produced endpoints.
Why the Shape Matters
A bootstrap interval compresses a distribution into two endpoints. That can be useful, but it can hide whether uncertainty is roughly balanced around the estimate, concentrated in a long tail, or separated into distinct clusters. A clear description should name the estimated quantity and its units, summarize the distribution’s notable features, and connect those features to the interval’s shape.
A highly skewed distribution can make symmetric approximations less descriptive; a multimodal distribution can make a single center unrepresentative; and an influential observation can cause the bootstrap estimates to vary markedly depending on whether that case is resampled. These are diagnostic warnings, not automatic verdicts. The proper response is to investigate how the data and statistic produce the observed pattern and to avoid presenting a compact interval as if it showed the entire uncertainty structure.
- Mistake: “The bootstrap histogram is centered above the sample estimate, so the parameter is definitely biased upward by that amount.”
- Correction: The center difference is an estimated bootstrap bias, conditional on the observed data and resampling method. It is not the known true bias.
- Mistake: “The percentile interval is uneven around the estimate, so it must be wrong.”
- Correction: Unequal distances can reflect a skewed bootstrap distribution. Assess the method and assumptions rather than demanding symmetry.
- Mistake: “Two peaks mean the parameter has two possible values.”
- Correction: The peaks describe clusters of bootstrap estimates. Investigate why resamples form those clusters; do not interpret them as two values of a fixed parameter.
Check Your Understanding
For each question, explain your reasoning in context rather than naming a feature alone.
- A bootstrap distribution has mean 18.7 and the original estimate is 18.2. What is the estimated bootstrap bias, and what does it not establish?
- A bootstrap histogram has a long tail toward smaller values. Which direction of skewness does this indicate, and what might happen to percentile interval endpoints?
- Why might an interval spanning two peaks provide an incomplete description of a multimodal bootstrap distribution?
- A regression slope changes substantially when one observation is removed. What does this say about influence, and what does it not prove about that observation?
- Why is it incorrect to interpret a 95% bootstrap interval as giving a 95% probability that the fixed population parameter lies in the particular interval?