Bootstrap Intervals for Regression Slopes

Why Bootstrap a Regression Slope?

A fitted regression slope describes the estimated change in the response associated with a one-unit increase in the predictor. But the slope is calculated from a sample, so another sample from the same population would generally produce a different slope. A confidence interval expresses uncertainty about the population slope. When a formula-based interval may be sensitive to model assumptions, resampling the observed predictor-response pairs offers a way to estimate how the slope varies under repeated sampling.

The pairing is essential. Each observation consists of an \(x\)-value and its corresponding \(y\)-value. A bootstrap sample draws these complete rows, with replacement, so the observed relationship between predictor and response is preserved. Drawing predictor values and response values separately would break that relationship and would not estimate uncertainty for the same sampling process.

Definition: A pairs bootstrap interval for a regression slope is an interval constructed from slopes calculated on bootstrap samples. Each bootstrap sample contains \(n\) rows drawn with replacement from the \(n\) observed \((x,y)\) pairs. The slope is recalculated for each sample, and the resulting bootstrap distribution is used to estimate the sampling uncertainty of the observed slope.

Resampling Complete Rows

For \(n\) observed pairs, the least-squares slope is

b_1=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{n}(x_i-\bar{x})^2}.

Here \(b_1\) is the sample slope, \(\bar{x}\) is the sample mean of the predictor values, and \(\bar{y}\) is the sample mean of the response values. A bootstrap replicate consists of \(n\) draws from the observed rows, allowing the same row to appear more than once and some rows not to appear. Recalculate the slope using that replicate’s pairs, including the replicate’s own means.

Worked Example: Calculating One Bootstrap Slope

Suppose a hypothetical data set has four paired observations: \((1,2)\), \((2,3)\), \((3,2)\), and \((4,5)\). One bootstrap draw selects row numbers \(2,2,3,4\), so the bootstrap sample is \((2,3)\), \((2,3)\), \((3,2)\), and \((4,5)\). The repeated row is allowed because sampling is with replacement.

The bootstrap means are \(\bar{x}^{*}=(2+2+3+4)/4=2.75\) and \(\bar{y}^{*}=(3+3+2+5)/4=3.25\). The numerator of the slope is

$$ (2-2.75)(3-3.25)+(2-2.75)(3-3.25)+(3-2.75)(2-3.25)+(4-2.75)(5-3.25) =0.1875+0.1875-0.3125+2.1875=2.25. $$

The denominator is

$$ (2-2.75)^2+(2-2.75)^2+(3-2.75)^2+(4-2.75)^2 =0.5625+0.5625+0.0625+1.5625=2.75. $$

Thus this bootstrap replicate has slope \(b_1^{*}=2.25/2.75\approx0.8182\) response units per predictor unit. One replicate is not an interval; it is one draw from the bootstrap distribution of slopes.

From Bootstrap Slopes to an Interval

Repeat the row-resampling process \(B\) times, calculating a slope \(b_1^{*}\) each time. The collection of slopes approximates the sampling distribution of the slope, conditional on the observed data. A bootstrap standard error is the standard deviation of these \(B\) slopes. A percentile interval uses quantiles of their distribution. For example, a 95% percentile interval uses the 2.5th and 97.5th percentiles.

The justification is based on the empirical distribution: it assigns probability \(1/n\) to each observed row. Drawing \(n\) rows from this distribution mimics drawing a new sample from a population approximated by the observed data. If the estimator’s sampling behavior under that empirical distribution is a useful approximation to its behavior under the population, the bootstrap slopes approximate the estimator’s sampling distribution. This is an approximation, not a guarantee; the data must contain enough information to represent the relationship of interest.

Procedure: To construct a pairs-bootstrap percentile interval for a slope, draw \(B\) samples of \(n\) complete rows with replacement. Refit the regression and record the slope for each sample. The 95% percentile interval is the interval from the 2.5th to the 97.5th percentile of the recorded slopes. Larger \(B\) reduces Monte Carlo variability in the estimated quantiles, but cannot remedy a poor or unrepresentative original sample.

Worked Example: A Percentile Interval and a Formula-Based Interval

Consider a hypothetical sample of eight paired observations:

\((1,2),(2,3),(3,5),(4,4),(5,6),(6,7),(7,8),(8,10)\), where \(x\) is measured in predictor units and \(y\) in response units. The sample means are \(\bar{x}=4.5\) and \(\bar{y}=5.625\). The sums of squares and cross-products are \(S_{xx}=\sum(x_i-\bar{x})^2=42\) and \(S_{xy}=\sum(x_i-\bar{x})(y_i-\bar{y})=44.5\). Therefore,

b_1=\frac{S_{xy}}{S_{xx}}=\frac{44.5}{42}=1.0595.

A hypothetical run of \(B=10{,}000\) pairs-bootstrap samples gives a 2.5th percentile of 0.82 and a 97.5th percentile of 1.30. The bootstrap 95% percentile interval is therefore \([0.82,1.30]\) response units per predictor unit. In context, this method estimates a range of plausible values for the population slope describing the change in mean response associated with a one-unit increase in the predictor.

For comparison, a formula-based interval for the slope uses the simple linear regression model

Y_i=\beta_0+\beta_1x_i+\varepsilon_i,

where the predictor values are treated as fixed, and the errors are independent with mean zero, common variance \(\sigma^2\), and a normal distribution. Under these assumptions, the estimated standard error of the slope is \(SE(b_1)=s_e/\sqrt{S_{xx}}\), where \(s_e=\sqrt{SSE/(n-2)}\) and \(SSE=\sum(y_i-\hat{y}_i)^2\). Normal errors give the \(t\)-based interval with \(n-2\) degrees of freedom.

For these data, \(S_{yy}=\sum(y_i-\bar{y})^2=49.875\), so

SSE=S_{yy}-\frac{S_{xy}^2}{S_{xx}} =49.875-\frac{44.5^2}{42} =49.875-47.1488 =2.7262.

Then \(s_e=\sqrt{2.7262/(8-2)}=\sqrt{0.4544}\approx0.6741\), and

SE(b_1)=\frac{0.6741}{\sqrt{42}}\approx0.1040.

Using \(t^{*}=2.447\) for a 95% interval with \(df=6\), the formula-based interval is

1.0595\pm 2.447(0.1040)=1.0595\pm0.2545, \qquad [0.8050,1.314].

Both intervals are centered near a slope of 1.06 and have similar endpoints here. That agreement is reassuring but does not prove that either method is valid. The formula-based interval relies on its stated regression assumptions; the bootstrap interval relies on the observed pairs being a useful stand-in for the population and on the resampling distribution adequately approximating slope uncertainty.

Why the Row Pairing Matters

In a pairs bootstrap, a sampled row carries its predictor and response together. This keeps intact the observed association and the observed spread of predictor values. If the \(x\)-values and \(y\)-values are resampled independently, the new rows combine values that were not paired in the original data. The resulting slopes reflect an artificially altered relationship, so their distribution cannot serve as a bootstrap approximation for the original slope.

Worked Example: Recognizing an Invalid Resampling Scheme

Suppose the four rows are \((1,2),(2,3),(3,2),(4,5)\). A pairs-bootstrap draw might select rows \(1,1,3,4\), producing \((1,2),(1,2),(3,2),(4,5)\). Every response remains attached to the predictor value with which it was observed.

An invalid approach might separately resample predictor values \((1,1,3,4)\) and response values \((3,5,2,2)\), then pair them in draw order. Those pairs include \((1,3)\) and \((1,5)\), which do not preserve the original observations. The altered slope is not a replicate from the pairs bootstrap. To estimate uncertainty for the observed regression relationship, resample whole rows, not separate columns.

Conditions, Limitations, and Interpretation

Before using either method, consider how the observations were obtained and whether a straight-line summary is sensible. For the formula-based \(t\) interval, the model should be approximately linear in the predictor, errors should be independent with constant variance, and the errors should be approximately normal for exact small-sample \(t\) inference. Independence comes from the study design, not from a calculation. A scatterplot and residual checks can help assess the linearity, spread, and unusual residuals; they cannot establish independence by themselves.

The pairs bootstrap does not require normal errors for its resampling step, but it is not assumption-free. The observed pairs should be representative of the population or sampling process of interest, and the data should provide enough variation in \(x\) to estimate a slope. A bootstrap replicate with no variation in its predictor values has an undefined slope; very small or unusual samples can also yield unstable intervals. Resampling cannot fix confounding, selection bias, measurement error, or a relationship that is poorly summarized by a line.

A slope interval is about the population linear association in the setting represented by the sampling process. It does not, by itself, establish that changing the predictor causes a change in the response. A causal interpretation requires a suitable study design and assumptions beyond regression calculation.

Common Mistake

Wrong reasoning: “I can bootstrap the predictor values and response values separately because both columns have the right data values.” That destroys the observed pairing and changes the relationship the slope is intended to summarize.
Correction: Sample complete \((x,y)\) rows with replacement, refit the regression on each sample, and use the resulting slopes to estimate uncertainty.
Another pitfall: Treating a bootstrap interval as automatically assumption-free. It still depends on a representative sample and an informative observed distribution; it cannot repair defects in the data or study design.

Key takeaway: A pairs bootstrap estimates slope uncertainty by resampling complete predictor-response observations and recalculating the slope. Compare its interval with the formula-based interval in light of each method’s assumptions; neither interval alone validates the regression model or supports a causal claim.

Check Your Understanding

Try these questions without looking back.

Why must the predictor and response values be resampled as complete rows when bootstrapping a regression slope?
A bootstrap sample selects row numbers \(1,3,3,4\) from a four-row data set. Is this a valid draw with replacement? Explain.
What two quantiles define a 95% percentile bootstrap interval for a slope?
State the error assumptions used for the usual formula-based \(t\) interval for a regression slope.
Why does a bootstrap interval not, by itself, establish that the predictor causes changes in the response?