Bootstrap Calibration of Robust Multivariate Tests

Why Calibrate a Robust Test by Bootstrap?

A robust test may use a location or regression estimator that resists outliers, a robust scatter estimate, or a statistic whose finite-sample distribution is difficult to approximate. A textbook reference distribution can then give poor tail probabilities, especially with small samples, heavy tails, or unequal group scatters. Bootstrap calibration estimates the null distribution by repeatedly generating data that mimic the sampling process, refitting the robust procedure each time, and comparing the observed statistic with the resulting bootstrap statistics.

The design has two separate decisions. First, determine what must be resampled together: an observation, a group-specific observation, a cluster, or a response conditional on a design point. Second, ensure the bootstrap distribution represents the null hypothesis, rather than simply reproducing the observed effect. Getting either decision wrong can give a precise-looking but invalid p-value.

Test Statistics and the Bootstrap Reference Distribution

For a \(d\)-dimensional location parameter \(\mu\), a common robust Wald statistic for \(H_0:\mu=\mu_0\) is

where \(\hat{\mu}\) is a robust location estimator and \(\hat{\Omega}\) estimates the asymptotic covariance matrix of \(\sqrt n(\hat{\mu}-\mu)\). The matrix \(\hat{\Omega}\) is not automatically the robust scatter estimate of the observations: it must estimate the sampling covariance of the chosen estimator. If the statistic tests \(q\) independent linear restrictions, replace the difference by the \(q\)-dimensional restriction residual and use its estimated covariance.

With \(B\) bootstrap replications, a standard Monte Carlo tail estimate for a large-value test is

The added one prevents a reported zero when no simulated value exceeds the observed statistic. It does not make a general bootstrap test an exact finite-sample test. Its calibration is usually approximate and depends on the bootstrap distribution representing the null law well.

How to Impose the Null

For a one-sample location test, one construction starts with residuals \(R_i=X_i-\hat{\mu}\), where \(\hat{\mu}\) is a suitable robust estimate of location. Resample residual vectors as whole units and set \(X_i^*=\mu_0+R_{I_i}\), with indices \(I_i\) sampled from \(1,\ldots,n\). The generated observations are centered at the null value, while their empirical residual distribution preserves estimated shape and dependence among coordinates. In each bootstrap sample, recompute the robust location, its covariance estimate, and the test statistic.

This recipe is not universal. If a null hypothesis concerns a contrast between groups, imposing it generally requires generating all groups under the shared restriction while retaining group-specific nuisance features, such as distinct scatters. For regression, the design and response form the relevant data structure: a pairs bootstrap resamples complete rows when the design is random, while a fixed-design procedure must keep the design points fixed and generate responses under a restricted fit. Resampling individual coordinates independently would destroy their multivariate dependence; resampling regression responses without respecting their design can destroy the model being tested.

Conditions for Valid Calibration

A useful justification is available for regular robust estimators. Let \(X_1,\ldots,X_n\) be independent and identically distributed vectors under a distribution \(P\), and let \(\theta\in\mathbb{R}^d\) be the parameter under test. Suppose the estimator has an asymptotically linear representation

where \(\Omega\) is finite and positive definite. Assume the estimator and covariance estimate are sufficiently regular that the corresponding expansion holds conditionally for the ordinary empirical bootstrap, with \(\hat{\Omega}\xrightarrow{p}\Omega\). For a null-imposed bootstrap, assume its fitted null-generating distribution satisfies the same conditional expansion and covariance convergence. These are substantive requirements, not a consequence of calling an estimator robust. They can fail for some nonsmooth estimators, poorly estimated nuisance parameters, or bootstrap schemes that do not reproduce the sampling structure.

Proof. Under the null, the multivariate central limit theorem and the asymptotic linear representation give \(\sqrt n(\hat{\theta}-\theta_0)\xrightarrow{d}N_d(0,\Omega)\). Since \(\hat{\Omega}\xrightarrow{p}\Omega\), Slutsky's theorem gives \(T_n\xrightarrow{d}Z^\mathsf{T}\Omega^{-1}Z\), where \(Z\sim N_d(0,\Omega)\). Write \(Z=\Omega^{1/2}U\) for \(U\sim N_d(0,I_d)\); then \(Z^\mathsf{T}\Omega^{-1}Z=U^\mathsf{T}U\sim\chi^2_d\).

Conditionally on the data, the assumed bootstrap expansion and covariance convergence give the same limiting normal law for \(\sqrt n(\hat{\theta}^*-\theta_0)\) under the null-imposed generating distribution. Applying the same quadratic-form argument conditionally yields convergence of the bootstrap statistic's conditional distribution to \(\chi^2_d\), in probability. The limiting chi-square distribution is continuous, so its tail probabilities at fixed thresholds are continuous as well. Thus the bootstrap reproduces the limiting null calibration. \(\square\)

A related practical implication follows from the proof: all data-dependent fitting that contributes to the statistic must be repeated. If the scatter or standard error is estimated from the sample, it should usually be re-estimated in every replicate. Otherwise the bootstrap statistic may omit a source of sampling variation, and the theorem's conditional covariance argument no longer describes the computation actually performed.

Resampling Units and Failure Modes

The right resampling unit is determined by how the data were sampled, not by which option is easiest to code. For independent multivariate observations, resample complete vectors. For independent groups, resample within groups when group-specific distributions are permitted under the null. For clustered observations, resample clusters if clusters are independent and within-cluster dependence is part of the data-generating process. For fixed-design regression, retain the design and use a response-generation method justified by the error assumptions. In each case, the null restriction must be built into the bootstrap generation step or otherwise enforced by a valid calibration argument.

Bootstrap consistency is not guaranteed merely because the estimator has bounded influence or resists outliers. The empirical bootstrap may be inconsistent for a nonsmooth estimator or a statistic whose limiting law depends on features not captured by naive resampling. Heavy tails can also undermine covariance estimation or the finite-moment conditions behind a Wald approximation. If the required regularity conditions are doubtful, examine the estimator-specific theory and consider a bootstrap designed for that estimator and sampling model; do not infer validity from robustness alone.

Bootstrap calibration matters because robust inference is often chosen precisely when simple reference distributions may be unreliable. A defensible analysis states the resampling unit, explains how the null is imposed, and names the conditions supporting the bootstrap approximation. Reporting the replication count and Monte Carlo tail estimate makes the numerical resolution visible; it does not replace the justification for the resampling design.