Why Use a Wild Bootstrap?
In regression, a permutation test often relies on exchangeability: under the null, the observations being rearranged must have a joint distribution unchanged by the proposed rearrangement. Unequal error variances generally break that condition. If observations with different variances are freely permuted, the resampled data no longer reflect which design points had more variable errors. The earlier material on residual permutation under heteroskedasticity motivates a different strategy: keep each observation attached to its design variables, and generate new errors with an observation-specific scale.
The wild bootstrap does this by multiplying each fitted residual by a random variable with mean zero and variance one. Because the residual for observation \(i\) stays attached to its own design vector, the method can represent changing error variance across the design. It is a model-based, generally asymptotic procedure, not an exact randomization test.
Model, Hypothesis, and Assumptions
The design matrix \(Z\) has row \(i\) equal to \(z_i^\mathsf{T}\), so \(Z\) is \(n\) by \(p\). Assume it has full column rank, and that the null constraints are estimable. For useful large-sample approximation, no small number of observations should dominate the regression fit or the variance estimate; errors should have finite moments sufficient for the chosen studentized statistic, and the design should not become increasingly dominated by extreme leverage. These conditions support approximate validity as \(n\) grows. They do not make a small-sample wild-bootstrap test exact.
The ordinary least-squares estimate is \(\hat{\beta}=(Z^\mathsf{T}Z)^{-1}Z^\mathsf{T}Y\). A heteroskedasticity-consistent covariance estimate, using the HC0 form, is
The dimensions are important: \(z_i z_i^\mathsf{T}\) is \(p\) by \(p\), as required for the middle, or “meat,” matrix. For testing \(R\beta=r\), form the robust Wald statistic
When the usual large-sample conditions hold, \(W\) is approximately distributed as a chi-square random variable with \(q\) degrees of freedom under \(H_0\). The wild bootstrap instead approximates the null distribution by repeatedly generating responses under the null and recalculating the same studentized statistic.
Constructing the Wild Bootstrap
First fit the model constrained by \(H_0\), producing fitted values \(\tilde{Y}_i\) and restricted residuals \(e_{0i}=Y_i-\tilde{Y}_i\). In a restricted linear model, let \(h_{0ii}\) be the \(i\)th diagonal of its projection matrix. Restricted residuals tend to be smaller than the underlying errors, especially at high-leverage observations. One common correction is HC2: set \(\tilde{e}_i=e_{0i}/\sqrt{1-h_{0ii}}\). This correction requires \(h_{0ii}<1\). It is a useful adjustment, not a cure for extreme leverage.
Choose independent multipliers \(v_i\), independent of the observed data, with \(E(v_i)=0\) and \(\operatorname{Var}(v_i)=1\). Generate the bootstrap response under the null as \(Y_i^*=\tilde{Y}_i+\tilde{e}_i v_i\). Fit the unrestricted model to \(Y^*\), calculate its heteroskedasticity-robust covariance estimate, and calculate \(W^*\) using the same \(R\) and \(r\). Repeating this process gives a reference distribution for \(W\).
Two common multiplier choices are:
- Rademacher: \(v_i=-1\) or \(+1\), each with probability \(1/2\). These multipliers have mean zero, variance one, and third moment zero.
- Mammen: \(v_i=(1-\sqrt{5})/2\) with probability \((\sqrt{5}+1)/(2\sqrt{5})\), and \(v_i=(1+\sqrt{5})/2\) with probability \((\sqrt{5}-1)/(2\sqrt{5})\). They have mean zero, variance one, and third moment one.
Rademacher multipliers are simple and are often used when symmetric errors are plausible. Mammen multipliers can better reproduce a nonzero third moment in settings where error asymmetry matters for the approximation. Neither choice guarantees validity under severe leverage, dependence that the resampling scheme ignores, or a misspecified regression mean.
Why the Resampling Mimics Unequal Variance
The basic justification is conditional variance matching. Given the observed data, the fitted null values and adjusted residuals are fixed. The bootstrap perturbation for observation \(i\) has conditional mean zero and variance \(\tilde{e}_i^2\), since \(E(v_i)=0\) and \(E(v_i^2)=1\). Thus the bootstrap does not assign a common error scale to all observations.
For example, the random part of the least-squares score is a sum of design vectors times errors. Conditional on the data, its bootstrap counterpart has covariance
Here \(\operatorname{Var}^*\) denotes variance over the bootstrap multipliers, conditional on the observed data. The equality follows because the multipliers are independent, have variance one, and have zero mean, so cross-covariances between distinct observations vanish. The expression has the same structure as the heteroskedasticity-consistent covariance “meat.” In large samples, residuals can estimate the observation-specific error scales well enough for this resampling distribution to approximate the null distribution of a studentized statistic. This argument explains the method; it does not establish exact finite-sample calibration.
Worked Example: A Robust Test and One Bootstrap Replicate
Consider a hypothetical experiment with four observations, an intercept and a group indicator \(x_i\), where \(x_i=0\) for the first two observations and \(x_i=1\) for the last two. The responses are \(8,12,9,15\). Test whether the group coefficient is zero. This tiny data set illustrates the arithmetic only; it is far too small to rely on a large-sample calibration.
The fitted group means are \(10\) and \(12\), so the estimated group difference is \(2\). The residuals are \(-2,2,-3,3\). With two observations in each group, the HC0 variance estimate for the difference of means is the sum of the within-group squared residuals divided by \(2^2\):
The standard error is \(\sqrt{6.5}\approx2.550\), and the one-restriction Wald statistic is \(W=(2/\sqrt{6.5})^2=4/6.5\approx0.6154\). Equivalently, the signed robust \(t\)-statistic is \(2/\sqrt{6.5}\approx0.7845\). These numbers alone are not a calibrated wild-bootstrap p-value.
Under the null of equal group means, the restricted intercept-only fit is the grand mean, \(11\). Its four residuals are \(-3,1,-2,4\). The restricted-model leverage is \(h_{0ii}=1/4\) for every observation, so the HC2-adjusted residuals are each divided by \(\sqrt{3/4}=0.8660\): approximately \(-3.464,1.155,-2.309,4.619\).
For one illustrative Rademacher draw, take multipliers \(+1,+1,-1,+1\). The bootstrap responses are then approximately \(7.536,12.155,13.309,15.619\). Their group means are \(9.8455\) and \(14.464\), giving a difference of \(4.619\). The residuals around those group means are approximately \(-2.309,2.309,-1.155,1.155\). The HC0 variance estimate for the difference is
Thus, using the unrounded values, \(W^*=(4/\sqrt{3/4})^2/(10/3)=6.400\). This is one bootstrap statistic, not a p-value. A test requires many independent multiplier draws; its approximate p-value is the fraction of bootstrap statistics at least as large as the observed \(W\), using the same tail convention and studentization in every replicate.
Worked Example: How Multiplier Choice Changes a Perturbation
Suppose one HC2-adjusted residual in a hypothetical null fit is \(2\) response units. With a Rademacher multiplier, its bootstrap perturbation is \(2v_i\), so it is \(+2\) or \(-2\), each with probability \(1/2\). Its mean is \(0\) and variance is \((2^2+(-2)^2)/2=4\).
For a Mammen multiplier, the two possible multiplier values are approximately \(-0.6180\) and \(1.6180\), with probabilities approximately \(0.7236\) and \(0.2764\). The perturbations are therefore approximately \(-1.236\) and \(3.236\) response units. Their weighted mean is approximately \(0.7236(-1.236)+0.2764(3.236)=0\), and their weighted second moment is approximately \(0.7236(1.236^2)+0.2764(3.236^2)=4\). Both choices preserve the residual’s variance contribution; their different shapes affect higher moments. The multiplier choice should reflect the approximation and error-shape concerns, not be selected after seeing which one gives a smaller p-value.
Worked Testing Plan and Diagnostics
State the regression mean model, the restriction \(R\beta=r\), and a robust, studentized statistic. Do not switch the statistic between the observed data and bootstrap samples.
Calculate restricted fitted values and residuals. Apply a stated correction such as HC2, and inspect whether high leverage makes the correction unstable.
Keep the design rows fixed and form \(Y_i^*=\tilde{Y}_i+\tilde{e}_i v_i\). Draw new, independent multipliers for each replicate.
Refit the unrestricted model, recompute its robust covariance and statistic \(W^*\), and compare the bootstrap distribution with the observed \(W\). Report the multiplier choice, residual adjustment, number of replicates, and approximate calibration.
A sound workflow checks whether the conditional mean is plausibly represented by the regression, whether the design matrix has adequate rank, and whether a few observations dominate the fit. Residual-versus-fitted plots can reveal mean-pattern misspecification or changing spread, while leverage measures flag observations for which HC2 adjustments become large. These checks do not prove the assumptions, but serious patterns should prompt model revision or a different inferential strategy.
The basic procedure assumes independent errors conditional on the design. If observations are dependent, applying independent multipliers observation by observation generally fails to preserve that dependence. A cluster wild bootstrap can use a common multiplier within each independent cluster, but it requires enough independent clusters and a procedure designed for that sampling structure. It should not be treated as an automatic repair for arbitrary dependence.
Worked Example: Why Permutation Is Not a Substitute
Imagine two design groups with errors that have mean zero but standard deviations \(1\) and \(3\), respectively. The group labels and design values are fixed, and the scientific null says their regression means are equal. Even under this null, the response distributions differ in spread. Swapping residuals or responses across the groups assigns some high-variance errors to the low-variance group and some low-variance errors to the high-variance group.
For instance, if the errors are normal, an error from the first group has distribution \(N(0,1)\), whereas one from the second has distribution \(N(0,9)\). These distributions are not identical, so the errors are not exchangeable across groups. Permuting them does not reproduce the null sampling distribution. A wild bootstrap instead keeps the design points fixed and scales each residual by its own multiplier; residual magnitudes can therefore remain larger in the higher-variance group. This is the relevant advantage, provided the regression and independence assumptions are reasonable.
Comparison, Limitations, and Common Pitfalls
| Approach | What it resamples | Key requirement or limitation |
|---|---|---|
| Raw response permutation | Observed responses across design units | Requires exchangeability under the null; unequal variances commonly violate it. |
| Residual permutation | Residuals, usually after fitting a null model | Residuals must be suitably exchangeable; heteroskedasticity and leverage can undermine this. |
| Wild bootstrap | Each null residual times a random multiplier | Approximates a model-based null distribution; needs an adequate mean model and suitable independence, design, and large-sample conditions. |
| Heteroskedasticity-consistent asymptotic Wald test | No resampling; uses a robust covariance estimate | Can be convenient, but its reference approximation may be poor in small or high-leverage samples. |
The wild bootstrap is attractive when the errors are nonexchangeable because their variances differ, but it has limits. It does not make a wrong mean model correct, remove confounding, or guarantee accurate small-sample tail probabilities. A high-dimensional regression with many fitted parameters relative to observations may also leave residuals that estimate error scales poorly. In such settings, results can be sensitive to leverage corrections and multiplier distributions; report that sensitivity rather than implying it has disappeared.
- Pitfall: treating the wild bootstrap as an exact test. The reasoning “multipliers are randomly assigned, so the p-value is exact” is wrong. The multipliers generate a conditional approximation whose accuracy depends on the model and asymptotic conditions; they are not randomized treatment assignments.
- Pitfall: permuting when variances differ. The reasoning “the null makes all residuals interchangeable” is wrong. A null restriction on regression coefficients does not imply identical error distributions. Use a method whose assumptions match the error structure.
- Pitfall: using different statistics for the data and replicates. A non-robust observed statistic compared with robust bootstrap statistics does not give a coherent reference comparison. Recompute the same studentized statistic in every replicate.
Check Your Understanding
Answer each question, then compare with the solutions.
- Why can unequal error variances invalidate a permutation of residuals across design points?
- In \(Y_i=z_i^\mathsf{T}\beta+\varepsilon_i\), what dimension must \(z_i\) have if \(\beta\) is a column vector, and what is the dimension of \(z_i z_i^\mathsf{T}\)?
- If an adjusted residual is \(1.5\), what are the two possible perturbations under Rademacher multipliers, and what is their variance?
- What does a wild-bootstrap p-value represent, and why is it not generally exact?
- Name two diagnostics or conditions that could make the basic wild bootstrap unreliable.
Solutions. (1) Different variances mean the residual distributions are not identical across design points, so rearrangement does not preserve their joint distribution under the null. (2) \(z_i\) is \(p\) by \(1\); \(z_i z_i^\mathsf{T}\) is \(p\) by \(p\). (3) The perturbations are \(+1.5\) and \(-1.5\), each with probability \(1/2\); their mean is zero and their variance is \((1.5^2+(-1.5)^2)/2=2.25\). (4) It is the approximate null-tail proportion of bootstrap statistics at least as extreme as the observed statistic; the approximation relies on model and large-sample conditions rather than an exact randomization argument. (5) Examples include extreme leverage, a misspecified regression mean, too few independent observations, or dependence not represented by the multiplier scheme.