Bootstrap for Robust Location and Scatter Estimators

Why Bootstrap Robust Estimates?

A robust location estimate can resist a small number of extreme observations, and a robust scatter estimate can reduce their effect on estimated variability. That resistance does not by itself tell us how uncertain either estimate is. The pairs bootstrap addresses uncertainty by resampling whole observed vectors and recomputing the location and scatter estimates. Resampling whole vectors preserves the observed association among coordinates.

The bootstrap is not an automatic repair for contamination. It treats the empirical distribution of the observed data as its population, so contaminated observations can be selected repeatedly. Moreover, a resample may contain too few distinct observations for a scatter estimator to behave as it does on the original data. It is therefore important to distinguish robustness of the point estimator from reliability of its bootstrap distribution.

In practice, one repeats this procedure \(B\) times. The resulting estimates describe bootstrap variation around the original estimate; their empirical quantiles can be used for intervals when the estimator and bootstrap approximation justify that interpretation. The number \(B\) controls simulation precision, not whether the bootstrap is statistically appropriate.

Robustness and Bootstrap Validity Are Different Questions

A contamination model makes one aspect of robustness precise. For a distribution \(F\), a point \(x\), and a small contamination fraction \(\varepsilon\), define \(F_{\varepsilon,x}=(1-\varepsilon)F+\varepsilon\delta_x\). For a statistical functional \(T\), its influence function is \( \operatorname{IF}(x;T,F)=\lim_{\varepsilon\downarrow0}\{T(F_{\varepsilon,x})-T(F)\}/\varepsilon \), when this limit exists. A bounded influence function has bounded norm as \(x\) ranges over possible observations. It limits the first-order effect of a small amount of contamination at a single point. It does not guarantee good behavior under large contamination, nor does it by itself guarantee bootstrap validity.

Breakdown concerns a different, more global failure. In a finite sample, the replacement breakdown point is the smallest fraction of observations that can be replaced so that the estimate can be driven arbitrarily far, or, for scatter, made arbitrarily large or degenerate. The precise finite-sample value depends on the estimator and its convention. Bounded influence is a local sensitivity property; breakdown is resistance to extensive replacement. Neither property alone establishes that the empirical bootstrap correctly approximates sampling uncertainty.

A First-Order Validity Result

A useful justification is available when a robust estimator has an asymptotically linear representation. This is an additional regularity property, not a consequence of the word “robust.” Write \(\theta=T(F)\) for the population parameter and let \(\psi:\mathbb{R}^p\to\mathbb{R}^q\) be a vector influence function with \(E_F[\psi(X)]=0\) and \(E_F[\|\psi(X)\|^2]<\infty\). Suppose \( \hat\theta_n-\theta=n^{-1}\sum_{i=1}^n\psi(X_i)+r_n \), where \(\sqrt n\,r_n\xrightarrow{p}0\). For the bootstrap, suppose the corresponding expansion holds conditionally: \( \hat\theta_n^*-\hat\theta_n=n^{-1}\sum_{j=1}^n\{\psi(X_j^*)-P_n\psi\}+r_n^* \), where \(\sqrt n\,r_n^*\xrightarrow{p^*}0\) in probability. Here \(P_n\psi=n^{-1}\sum_i\psi(X_i)\), and conditional convergence in probability means the conditional probability of exceeding any fixed positive threshold tends to zero in probability over the original data.

Proof. The original representation and the multivariate central limit theorem give \( \sqrt n(\hat\theta_n-\theta)=n^{-1/2}\sum_i\psi(X_i)+o_p(1)\xrightarrow{d}N_q(0,V) \). For the bootstrap, condition on the observed data. The vectors \(\psi(X_j^*)-P_n\psi\) are then independent draws from the centered empirical distribution of the influence values. Their conditional covariance is \( V_n=P_n[(\psi-P_n\psi)(\psi-P_n\psi)^\mathsf{T}] \). The law of large numbers applied to the integrable entries of \(\psi\psi^\mathsf{T}\), and to \(\psi\), gives \(V_n\xrightarrow{p}V\).

The conditional vector Lindeberg quantity is \( P_n[\|\,\psi-P_n\psi\,\|^2\,\mathbf{1}\{\|\psi-P_n\psi\|>\eta\sqrt n\}] \). This tends to zero in probability: truncate \(\|\psi\|\) at a fixed level, for which the indicator is eventually zero, and then let the truncation level increase; the expected contribution of the discarded squared tail tends to zero because \(E_F\|\psi\|^2<\infty\). The conditional multivariate Lindeberg–Feller theorem therefore gives a conditional normal limit with covariance \(V\) (including the possibly singular case). Finally, the assumed conditional remainder is negligible at the \(\sqrt n\) scale, so adding it does not change that limit. This proves the result. \(\square\)

The theorem explains the role of influence functions: if \(\psi\) is bounded, its second moment is finite, but boundedness alone does not supply either asymptotic linearity or the required bootstrap expansion. Nonsmooth estimates, data-dependent tuning, singular scatter estimates, and distributions near a breakdown boundary all require particular care before applying this argument.

When Resamples Expose a Problem

A robust scatter estimator may require a minimum number of distinct observations or a nonsingular subset. Bootstrap sampling creates duplicates, so some replicates may fail that requirement even when the original estimate exists. Silently discarding failed replicates changes the conditional distribution being used for inference. Any rule for handling such replicates—reporting failure, using a defined regularization, or restricting the inferential claim—must be stated and justified for the estimator at hand.

Practical Interpretation and Common Pitfalls

Bootstrap uncertainty for robust multivariate estimates is most persuasive when the estimator is well defined over essentially all relevant resamples, its sampling behavior is regular near the data-generating distribution, and contamination is not dominating the empirical distribution. Examine location and scatter replicates separately, as well as jointly when the scientific question concerns both. For scatter, check for zero or nearly degenerate estimates and for unusually large values; for location, check whether a small number of observations dominate the replicate variation.

• Mistake: “The estimator has bounded influence, so its bootstrap must be valid.” Bounded influence limits local sensitivity. Bootstrap validity also requires a suitable sampling expansion and stable behavior of the resampling procedure.
• Mistake: “Robustness removes contamination from the bootstrap.” The empirical distribution retains every observed vector. Contaminated points can appear repeatedly in resamples.
• Mistake: “A failed or singular replicate is just a numerical nuisance.” It may reflect a genuine support or breakdown issue for the estimator. State how it is handled; do not treat deletion as innocuous.