Bootstrap Prediction Intervals from Regression Residuals

Why Bootstrap a Prediction Interval?

A prediction interval describes a range of plausible values for one new outcome at specified predictor values. The usual regression interval relies on a model for the errors, often including a normality assumption. A residual bootstrap offers another way to approximate the distribution of a future outcome: it repeatedly resamples the fitted residuals, reconstructs data sets, refits the regression, and simulates a new error.

The new predictor values are treated as fixed, as in the earlier material on treating new predictor values as fixed. The target is still an individual outcome, not just the mean response at those predictor values. Consequently, each bootstrap replicate needs both a fitted prediction and a simulated future error. Refitting the regression also allows the bootstrap distribution to reflect variation in the estimated regression fit.

Definition: A residual-bootstrap prediction interval is an interval formed from simulated future outcomes. Each simulation is produced by resampling fitted residuals to create a bootstrap training data set, refitting the regression, predicting at the fixed new predictor values, and adding an independently resampled residual as the simulated future error.

Model and Assumptions

Suppose the response follows a regression model \(Y_i=m(x_i)+\varepsilon_i\), where \(x_i\) denotes the predictors for training case \(i\), \(m(x_i)\) is the model's mean response, and \(\varepsilon_i\) is its error. At the fixed new predictor values \(x_{\mathrm{new}}\), the new outcome is \(Y_{\mathrm{new}}=m(x_{\mathrm{new}})+\varepsilon_{\mathrm{new}}\). A prediction must account for the future error \(\varepsilon_{\mathrm{new}}\), as well as uncertainty in estimating the mean model.

The basic residual bootstrap is most defensible when the mean model is adequately specified and training errors are independent, have mean zero, and share a common distribution across predictor values. A common distribution implies equal error variance; it is the reason a single pooled residual collection can stand in for future errors at different predictor values. The procedure also assumes that the new case is drawn under the same error-generating conditions as the training cases. It does not make an unreliable mean model reliable, and it does not ensure accurate coverage in a small sample.

After fitting the regression, write the fitted residual for case \(i\) as \(e_i=y_i-\hat{y}_i\). Residuals are estimates of unobserved errors, not the errors themselves. Center them before resampling so the empirical pool has mean zero. Denote these centered residuals by \(e_i^c=e_i-\bar{e}\), where \(\bar{e}\) is the average fitted residual. In ordinary least squares with an intercept, residuals sum to zero; explicit centering is still a useful general instruction.

The Residual-Bootstrap Algorithm

For each bootstrap replicate, keep the original predictor values fixed. Resample \(n\) centered residuals with replacement, add them to the original fitted responses to construct a bootstrap response, and refit the same regression model. Then predict at \(x_{\mathrm{new}}\) and add one more residual drawn independently from the centered residual pool. This last draw represents the error of the future individual outcome.

Fit the stated regression.
Calculate fitted values \(\hat{y}_i\) and residuals \(e_i=y_i-\hat{y}_i\). Center the residuals to form the resampling pool \(e_1^c,\ldots,e_n^c\).

Construct a bootstrap training response.
Draw \(n\) residuals independently with replacement from the centered pool. For each original case, set \(y_i^*=\hat{y}_i+e_i^*\), keeping its original predictor values \(x_i\).

Refit and simulate the new outcome.
Fit the same regression to \((x_i,y_i^*)\). At the fixed new predictor values, calculate \(\hat{y}_{\mathrm{new}}^*\), then draw an independent residual \(e_{\mathrm{new}}^*\) from the centered pool and set \(y_{\mathrm{new}}^*=\hat{y}_{\mathrm{new}}^*+e_{\mathrm{new}}^*\).

Repeat and take quantiles.
Repeat the simulation \(B\) times. For a nominal 95% percentile prediction interval, use the 2.5th and 97.5th percentiles of the \(B\) simulated future outcomes.

The refitting step matters. If we only add a resampled residual to the original fitted prediction, we simulate outcome noise but do not reproduce variation in the estimated regression fit. That shortcut may be useful for describing residual spread, but it is not the full procedure just defined for a prediction interval. In each full replicate, the bootstrap training responses generate a new fitted regression, and a separate residual represents the future case's error.

Worked Example: One Bootstrap Replicate by Hand

Consider a hypothetical regression of an outcome \(Y\), measured in units, on one predictor \(x\). The four training observations are \((0,9),(1,13),(2,15),(3,15)\). The fitted line is \(\hat{y}=10+2x\), with fitted values \(10,12,14,16\) and residuals \(-1,1,1,-1\). These residuals already have mean zero. We want a prediction at the fixed value \(x_{\mathrm{new}}=4\).

For one bootstrap replicate, suppose the four residuals drawn with replacement, in the original predictor order, are \(1,-1,1,-1\). Adding them to the original fitted values gives the bootstrap response \(11,11,15,15\). The bootstrap data are therefore \((0,11),(1,11),(2,15),(3,15)\).

The mean bootstrap response is \(\bar{y}^*=13\), and the mean predictor is \(\bar{x}=1.5\). The refitted slope is

b_1^*= \frac{\sum_{i=1}^{4}(x_i-\bar{x})(y_i^*-\bar{y}^*)} {\sum_{i=1}^{4}(x_i-\bar{x})^2} = \frac{8}{5}=1.6.

The refitted intercept is \(b_0^*=\bar{y}^*-b_1^*\bar{x}=13-(1.6)(1.5)=10.6\). Thus the refitted prediction at \(x=4\) is \(10.6+(1.6)(4)=17.0\) units. If the independently drawn future residual for this replicate is \(1\), the simulated future outcome is \(17.0+1=18.0\) units. This is one bootstrap future outcome, not the interval itself.

Why the Simulations Represent Prediction Uncertainty

Each simulated future outcome has two pieces: a refitted mean prediction and a simulated future error. The bootstrap training data vary from replicate to replicate, so the refitted mean prediction varies. The additional residual varies the future outcome around that prediction. The percentile interval summarizes the combined variation.

This reasoning can be written as a simple decomposition. In the original model, the difference between a new outcome and its true mean is \(\varepsilon_{\mathrm{new}}\). In a bootstrap replicate, the corresponding difference from the original fitted prediction can be written as a change in fitted mean plus the simulated future error:

y_{\mathrm{new}}^*-\hat{y}_{\mathrm{new}} = \left(\hat{y}_{\mathrm{new}}^*-\hat{y}_{\mathrm{new}}\right) + e_{\mathrm{new}}^*.

The first term represents variation in the estimated mean prediction across bootstrap refits. The second represents individual outcome noise. Omitting the second term targets uncertainty about the mean rather than prediction of an individual outcome; omitting refitting leaves out variability from estimating the regression. This decomposition explains why both operations belong in the full residual-bootstrap prediction procedure.

Worked Example: Turning Bootstrap Outcomes Into an Interval

Suppose a hypothetical regression is used to predict a concentration in milligrams per litre for one new case with fixed predictor values. After applying the full procedure for \(B=10\) replicates, the simulated future outcomes, already sorted, are \(17.2,17.6,17.8,18.0,18.1,18.4,18.7,19.0,19.3,19.8\) mg/L. For this small teaching example, form an 80% percentile interval, using linear interpolation between adjacent ordered values.

For \(B=10\), the 10th-percentile position under the interpolation convention used here is \(1+(10-1)(0.10)=1.9\). It lies 0.9 of the way from the first value, \(17.2\), to the second, \(17.6\). The lower endpoint is \(17.2+0.9(17.6-17.2)=17.56\) mg/L.

The 90th-percentile position is \(1+(10-1)(0.90)=9.1\). It lies 0.1 of the way from the ninth value, \(19.3\), to the tenth, \(19.8\). The upper endpoint is \(19.3+0.1(19.8-19.3)=19.35\) mg/L. The resulting illustrative 80% interval is approximately \([17.56,19.35]\) mg/L. It describes a range produced by the bootstrap method for one future outcome under the model assumptions; it is not a probability statement about a fixed parameter. Ten replicates are far too few for stable practical endpoints, especially for a 95% interval.

Assumptions and Limitations in Practice

Residual resampling retains the observed predictor values but pools residuals across cases. This is appropriate only when that pooling is a reasonable representation of the error distribution at the new predictor values. The following checks and cautions are therefore important:

Mean-model form: systematic structure left in residuals can signal that the fitted regression does not capture the mean relationship. Resampling those residuals does not repair that misspecification.
Independence: the basic procedure resamples residuals as if errors were independent. If errors are clustered or ordered with dependence, independently resampling individual residuals breaks that structure.
Common error distribution: pooled residuals assume errors have a similar distribution at all predictor values. Unequal error variance or changing distribution shape can make the simulated future error inappropriate for \(x_{\mathrm{new}}\).
Extrapolation and sparse regions: a bootstrap does not create information at predictor combinations that are poorly represented in the data. A simulated interval can look precise even when a prediction relies on extrapolation.
Residuals are estimates: fitted residuals can be less variable than the underlying errors, particularly for observations with high leverage. The simple procedure may then understate error variability. More specialized adjustments may be needed; they are not guaranteed to fix other model problems.

Worked Example: Why Pooling Can Fail With Unequal Variance

Suppose a hypothetical regression has two predictor regions. At low predictor values, errors are typically near \(-1\) or \(1\) outcome unit; at high values, they are typically near \(-8\) or \(8\) units. Imagine that the fitted residual pool contains \(-8,-1,1,8\), and the new case has a high predictor value.

A pooled residual draw gives each of these four values probability \(1/4\). Its empirical variance is \(\{(-8)^2+(-1)^2+1^2+8^2\}/4=(64+1+1+64)/4=32.5\) units squared, corresponding to a standard deviation of about \(5.70\) units. But the stated high-value error pattern has variance \(\{(-8)^2+8^2\}/2=64\) units squared and standard deviation \(8\) units. The pooled draw mixes in smaller low-region errors, so it does not represent the larger high-region errors adequately. Refitting the regression does not correct this mismatch in the simulated future error.

The point is not that every pooled residual interval will be too narrow by exactly this amount. It is that the common-distribution assumption is substantively important: when it fails, the basic residual bootstrap can misrepresent the new case's prediction uncertainty.

Common Mistake and Practical Interpretation

A common mistake is to resample residuals and add them directly to the original fitted prediction, then call the resulting quantiles a complete residual-bootstrap prediction interval. The reasoning is that residuals represent all uncertainty in a future outcome. The correction is that this shortcut includes simulated outcome noise but keeps the fitted regression fixed. The full procedure constructs bootstrap training responses, refits the regression, and then adds an independent simulated future error.

Another mistake is to assume the bootstrap automatically removes the need to check assumptions. It does not. Residual resampling is a method for approximating a sampling distribution under assumptions; it is not a safeguard against an unsuitable mean model, unequal variance, dependence, or weak data support near the new predictor values. The familiar Gaussian-error method and the bootstrap are different approaches, but neither removes the need to state what model is being used and what conditions make its predictions credible.

Key takeaway: A basic residual-bootstrap prediction interval is built by resampling centered fitted residuals to create bootstrap training responses, refitting the regression, predicting at the fixed new predictor values, and adding a separately resampled future error. Its reliability depends on the mean model and on residuals being a reasonable, independent, common-distribution stand-in for errors.

Check Your Understanding

Try these questions without looking back.

Why does a residual-bootstrap prediction interval include a separately sampled residual after refitting the regression?
In the bootstrap training step, which values stay fixed, and which quantities are resampled?
Explain why adding one sampled residual to the original fitted prediction does not reproduce the full residual-bootstrap procedure.
What assumption about error variation is made when residuals from all predictor values are pooled into one resampling pool?
A residual plot shows a much larger spread at high predictor values than at low values. Why might the basic pooled residual bootstrap give an unsuitable interval for a new case at a high predictor value?