Pairs Bootstrap for Heteroskedastic Regression

Why Resample Pairs?

In heteroskedastic regression, the variability of the outcome around its conditional mean can change with the predictors. A resampling method should reflect that the observed predictor values and outcomes occur together: an outcome should not be reassigned to an unrelated predictor just to generate a new sample. The pairs bootstrap does this by resampling whole predictor–outcome observations.

This method is especially natural when observations are independent draws from a population and the predictors are random. It generates new samples from an approximation to the joint distribution of predictors and outcomes, then refits the regression in each sample. The resulting coefficient distribution describes how the estimator varies under repeated sampling from that empirical approximation. It is not a guarantee of accurate inference: the sampling design, model target, sample size, and stability of the design still matter.

Algorithm and Regression Target

For a regression with an intercept and \(k\) predictors, let \(x_i\) denote the predictor vector for observation \(i\), including the intercept column, and let \(y_i\) be its outcome. Write the ordinary least-squares estimate as \(\hat{\beta}\). For bootstrap replication \(b\), draw \(n\) indices independently and uniformly from \(\{1,\ldots,n\}\). Refit the regression to the pairs selected by those indices, producing \(\hat{\beta}^{*(b)}\).

The target need not be a literally correct conditional-mean equation. If the linear conditional mean is misspecified, ordinary least squares can target the best linear projection under the population distribution of the predictors. Pairs resampling then approximates repeated sampling of both predictors and outcomes, and thus the variability of that projection estimator, provided the observations are independent and the required moments and design conditions hold. It does not turn a misspecified linear description into the true conditional mean.

What the Bootstrap Distributions Mean

For \(B\) replications, the collection \(\hat{\beta}^{*(1)},\ldots,\hat{\beta}^{*(B)}\) is the bootstrap distribution of the coefficient estimator. For a single coefficient, its bootstrap standard error is the sample standard deviation of its \(B\) replicate values:

This is an estimate of the sampling standard deviation of the coefficient estimator. It is distinct from the standard error reported within one regression fit. In particular, if each replicate uses HC0, the replicate HC0 standard errors form their own distribution. The average of those reported standard errors is not, by definition, the bootstrap standard error; the latter is calculated from the variation among the replicate coefficients.

For a regression with an intercept, the HC0 covariance estimate in a bootstrap sample has the sandwich form

where \(X^*\) is the resampled design matrix and \(\hat{u}_i^*\) are residuals from the refitted regression. The square root of the relevant diagonal entry is the HC0 standard error for that coefficient in that replication. Other robust covariance choices can be used, but the choice should be stated and applied consistently.

Worked Examples

Why Resampling Complete Pairs Preserves the Relationship

The empirical distribution of the observed pairs assigns probability \(1/n\) to each observed \((x_i,y_i)\). A pairs-bootstrap sample draws from this joint empirical distribution. Consequently, each selected outcome remains attached to its observed predictor value, and repeated copies preserve that same pairing. If the residual spread differs across predictor values in the observed data, resampling pairs carries those different predictor–outcome combinations into the bootstrap samples.

This is what it means to preserve the observed design relationship: the method samples from the observed joint distribution rather than separately rearranging predictors and outcomes. It does not hold the exact set of predictor values fixed in every replication. Predictor values can appear multiple times or not at all, so pairs resampling reflects random-design variation as well as outcome variation.

If the predictors are fixed by design and the inferential target conditions on those exact values, resampling whole pairs changes the design and may not match the intended sampling scheme. Further, independently sampled pairs are essential: resampling individual observations as if independent is not justified when observations have dependence such as clustering or serial correlation. The method also requires enough variation in the resampled predictors to estimate the regression. A bootstrap draw with a rank-deficient design has no ordinary least-squares coefficient for all regressors; frequent such failures indicate that the design or sample size is too fragile for routine pairs-bootstrap inference.

Common Pitfalls and Practical Checks

• Breaking the pairing. Resampling predictors and outcomes separately destroys their observed joint relationship. Draw complete observations instead.
• Confusing the two standard errors. The standard deviation of replicate coefficients estimates bootstrap sampling variability. A robust standard error computed within each replicate is a different quantity; record it as a separate distribution.
• Assuming pairs resampling fixes every problem. It does not correct dependence, a poor sampling design, severe lack of predictor support, or an unsuitable regression target.
• Reporting only the number of replications. State how pairs were sampled, which coefficient or standard error was summarized, the covariance estimator used within replicates, and how an interval was formed.
• Ignoring instability. With small samples, a few observations can dominate the fit, and some resamples can have very different designs. Inspect the replicate distribution and any rank-deficient samples rather than treating the output as automatically trustworthy.