Empirical Bootstrap for Smooth Vector Parameters

Why Resample Multivariate Observations?

A parameter may be a vector even when the data arrive as whole multivariate observations. For example, a study may estimate several population means at once, or the entries of a covariance matrix. When observations are independent and identically distributed, the empirical bootstrap approximates the sampling variation of such estimates by repeatedly drawing entire observed vectors with replacement. Keeping each vector intact preserves the observed dependence among its components.

The central case in this tutorial is a smooth function of an empirical average. This covers means directly, and also covers covariance matrices after expressing their entries through first and second moments. The approximation is conditional on the observed data: it treats the empirical distribution as a stand-in for the unknown population distribution.

The bootstrap sample consists of \(n\) draws, not \(n\) distinct observations. Equivalently, each draw selects one of the original indices uniformly and independently, so a given observation can appear several times or not at all. For multivariate data, a draw selects one complete vector \(X_i\), not separate coordinates sampled independently.

Smooth Vector Parameters and Bootstrap Consistency

Suppose a measurable vector-valued function \(Z=h(X)\) takes values in \(\mathbb{R}^q\), with \(E\|Z\|^2<\infty\). Write \(\mu=E[Z]\) and \(\Sigma=\operatorname{Var}(Z)\). Let \(g\) map an open neighborhood of \(\mu\) in \(\mathbb{R}^q\) into \(\mathbb{R}^r\), and suppose \(g\) is continuously differentiable there. The parameter and its estimator are \(\theta=g(\mu)\) and \(\hat{\theta}=g(\bar Z)\), where \(\bar Z=n^{-1}\sum_i Z_i\). The bootstrap estimate is \(\hat{\theta}^*=g(\bar Z^*)\), with \(\bar Z^*=n^{-1}\sum_i h(X_i^*)\).

These definitions cover a sample mean by taking \(Z=X\) and \(g\) to be the identity. They also cover covariance entries: use \(Z\) to collect the coordinates of \(X\) and the distinct products \(X_jX_k\), then express covariance entries as second moments minus products of means. Finite second moments of this enlarged \(Z\), which require finite fourth moments of \(X\) for the product coordinates, give the needed covariance matrix for its sample average.

Here conditional convergence means that, for every bounded continuous function \(\varphi\), the conditional expectation of \(\varphi(\sqrt n(\hat{\theta}^*-\hat{\theta}))\) converges in probability to its expectation under the stated normal distribution. This formulation handles vector parameters and allows a degenerate limit, for instance if some parameter components are linearly redundant.

Proof. The multivariate central limit theorem gives \(\sqrt n(\bar Z-\mu)\xrightarrow{d}N(0,\Sigma)\). Differentiability yields \(g(\bar Z)-g(\mu)=A(\bar Z-\mu)+R_n\), where \(\|R_n\|/\|\bar Z-\mu\|\xrightarrow{p}0\) whenever \(\bar Z\ne\mu\). Since \(\bar Z-\mu=O_p(n^{-1/2})\), it follows that \(\sqrt n R_n\xrightarrow{p}0\). The ordinary delta method proves the first convergence.

For the bootstrap statement, the empirical covariance of \(Z_1,\ldots,Z_n\) converges in probability to \(\Sigma\). The conditional central limit theorem for iid draws from the empirical distribution therefore gives, conditionally in probability, \(\sqrt n(\bar Z^*-\bar Z)\xrightarrow{d}N(0,\Sigma)\). Its Lindeberg condition holds in probability: for each \(\varepsilon>0\), the empirical average of \(\|Z_i-\bar Z\|^2\mathbf{1}\{\|Z_i-\bar Z\|>\varepsilon\sqrt n\}\) converges to zero in probability. This follows by centering, truncating \(\|Z_i\|^2\) at a fixed level, applying the law of large numbers to the truncated terms, and then letting the truncation level increase; finite \(E\|Z\|^2\) makes the expected tail tend to zero.

Also, \(\bar Z^*-\bar Z=O_p(n^{-1/2})\) under the joint distribution of data and resampling, since its conditional second moment is \(n^{-1}\) times the empirical covariance trace, which is bounded in probability. Applying the differentiability expansion at \(\bar Z\), with derivative \(Dg(\bar Z)\xrightarrow{p}A\), gives \(\sqrt n\{g(\bar Z^*)-g(\bar Z)\}=A\sqrt n(\bar Z^*-\bar Z)+o_p(1)\). The conditional central limit theorem and this negligible remainder give the claimed conditional limit. \(\square\)

Worked Example: Bootstrap Standard Errors for a Vector Mean

Bootstrap Standard Errors and Covariance Estimates

In applications, generate \(B\) independent bootstrap samples and calculate \(\hat{\theta}^{*(b)}\) from each. Their average and empirical covariance are \(\bar{\theta}^*=B^{-1}\sum_{b=1}^B\hat{\theta}^{*(b)}\) and \(\widehat{V}_{\!*}=(B-1)^{-1}\sum_{b=1}^B(\hat{\theta}^{*(b)}-\bar{\theta}^*)(\hat{\theta}^{*(b)}-\bar{\theta}^*)^\mathsf{T}\). The square roots of the diagonal entries of \(\widehat{V}_{\!*}\) estimate componentwise standard errors; the other entries estimate covariation between estimates. Increasing \(B\) reduces Monte Carlo error in these calculations, but does not change the data or remove sampling uncertainty.

The theorem establishes convergence in distribution, which by itself does not guarantee convergence of variances. To interpret the bootstrap covariance as a consistent standard-error estimate, one additionally needs sufficient integrability so the squared, scaled bootstrap errors do not retain substantial probability in increasingly extreme tails. A precise sufficient condition is conditional uniform integrability in probability of \(\|\sqrt n(\hat{\theta}^*-\hat{\theta})\|^2\), together with convergence of the conditional second moments. This condition holds in common smooth finite-dimensional settings under stronger moment and smoothness assumptions; it should not be inferred from weak convergence alone.

For a sample mean, the bootstrap covariance is especially transparent: it equals the empirical covariance of the observations divided by \(n\), exactly, conditional on the data. For a covariance matrix estimate, the bootstrap must recalculate the entire covariance matrix in each resample. If the entries are represented by averages of coordinates and products, the smooth-average result applies when those product variables have finite covariance. The bootstrap then estimates the joint sampling variation of all entries, not merely separate marginal variations.

Percentile and Basic Intervals

For a scalar component \(\theta_j\), let \(q_\alpha^*\) be the conditional \(\alpha\)-quantile of its bootstrap estimates \(\hat{\theta}_j^*\), using the generalized inverse convention \(q_\alpha^*=\inf\{t:P(\hat{\theta}_j^*\le t\mid X_1,\ldots,X_n)\ge\alpha\}\). The percentile interval at nominal level \(1-\alpha\) is \([q_{\alpha/2}^*,q_{1-\alpha/2}^*]\). The basic interval reflects the bootstrap error quantiles around the observed estimate: \([2\hat{\theta}_j-q_{1-\alpha/2}^*,\,2\hat{\theta}_j-q_{\alpha/2}^*]\). These are componentwise intervals; their separate nominal coverages do not imply simultaneous coverage for a whole vector.

Conditions, Interpretation, and a Common Pitfall

The ordinary empirical bootstrap described here is for iid sampling of the chosen observation units. If the independent units are clusters, for example, resampling individual records would not reproduce the sampling design; the units must be chosen to match the dependence structure. The data also need to contain enough information for the target: if a covariance estimate is undefined in a resample, a procedure for handling that event must be specified rather than silently discarding it.

Smoothness matters because the proof transfers the sampling fluctuation of an average through a first-order derivative. A parameter that is not differentiable at the population value, or a statistic whose limit is not captured by a first-order linear approximation, may not satisfy this theorem. The consistency result also concerns the centered bootstrap error \(\hat{\theta}^*-\hat{\theta}\), not the uncentered bootstrap estimates as an approximation to the error around the unknown \(\theta\).

• Common mistake: resampling each coordinate separately. This replaces the observed joint vectors with draws that can combine coordinates from different individuals. It can destroy the cross-covariance structure. Resample whole vectors when the sampling units are iid vectors.
• Common mistake: treating weak convergence as proof of standard-error consistency. Convergence of distributions does not alone control tail contributions to variances. Check suitable moment or uniform-integrability conditions.
• Common mistake: reading componentwise intervals as a joint confidence region. Separate intervals do not automatically provide the stated joint coverage for all components together.