Subsampling When the Ordinary Bootstrap Fails

Why Resampling the Full Sample Can Fail

The ordinary bootstrap draws \(n\) observations with replacement from the empirical distribution and recomputes an estimator. For a sufficiently smooth estimator with a stable first-order approximation, this often reproduces the estimator’s sampling distribution. But the empirical distribution is not the population distribution: its support and local shape are determined by the observed sample. When an estimator is sensitive to a boundary, a kink, or a growing number of coordinates, that difference can persist at the scale relevant to inference.

The problem is not simply that the bootstrap uses repeated observations. The deeper issue is whether resampling from the empirical distribution reproduces the limit law of the statistic. For nonregular estimators, the limiting operation may be nonlinear, depend on an estimated boundary, or fail to remain stable under empirical perturbations. In high dimensions, approximation errors can also accumulate across coordinates, so a method that works for a fixed-dimensional smooth parameter need not work for a maximum or other complex statistic.

The scale matters. If the estimator has rate \(r_n\), then a resampled statistic must be centered and scaled so its conditional variation is on the scale of \(r_n(T_n-\theta)\). For smaller resamples of size \(m\), this generally means using \(r_m\), not \(r_n\). The procedures below rely on \(m\) growing while remaining small relative to \(n\).

A Boundary Example: The Sample Maximum

Suppose \(X_1,\ldots,X_n\) are independent and uniformly distributed on \([0,\theta]\), where \(\theta>0\) is unknown. The maximum \(M_n=\max_i X_i\) estimates the upper endpoint. Its error is not of the usual square-root-\(n\) form: the relevant rate is \(n\).

To verify the result, first consider \(0\leq x\leq n\theta\). Since the observations are independent and uniform,

For \(x>n\theta\), the event is impossible, so its probability is zero. For each fixed \(x\geq0\), the restriction \(x\leq n\theta\) holds for all sufficiently large \(n\); therefore the displayed probability converges to \(e^{-x/\theta}\). In particular, when \(\theta=1\) the limit is \(e^{-x}\). For general \(\theta\), \(n(\theta-M_n)/\theta\) has the rate-1 exponential limit, or equivalently \(n(M_n-\theta)\) converges to a negative exponential variable scaled by \(\theta\). This endpoint example already shows why the scale and parameterization of a limit must be stated precisely.

Subsampling: Use Smaller Subsets Without Replacement

Subsampling avoids drawing repeatedly from the full empirical distribution. Choose a subset \(S\) of \(m\) distinct indices uniformly from \(\{1,\ldots,n\}\), compute the estimator \(T_{m,S}\) on those observations, and form \(r_m(T_{m,S}-T_n)\). Repeat over all subsets, or over a large random collection of them. The centering by \(T_n\) is convenient because \(\theta\) is unknown.

The last condition makes the full-sample centering negligible on the subset scale: \(r_m(T_n-\theta)=(r_m/r_n)\,r_n(T_n-\theta)\xrightarrow{p}0\), provided \(r_n(T_n-\theta)\) is bounded in probability. Subsampling also requires that its conditional distribution actually concentrates around the target law. The following result states this condition explicitly, rather than assuming that every statistic automatically has this property.

Here is the full argument. First, an unconditionally selected subset of \(m\) observations from an independent identically distributed sample has the same distribution as \(m\) independent observations from the population. Hence \(r_m(T_{m,S}-\theta)\xrightarrow{d}J\). Also \(r_m(T_n-\theta)\xrightarrow{p}0\), since \(r_m(T_n-\theta)=(r_m/r_n)r_n(T_n-\theta)\), the second factor is bounded in probability, and the first tends to zero. Subtracting this negligible term gives the unconditional convergence \(r_m(T_{m,S}-T_n)\xrightarrow{d}J\). Consequently, \(E[Q_{n,m}(t)]\to F_J(t)\) at every continuity point. By the assumed vanishing variance, \(Q_{n,m}(t)-E[Q_{n,m}(t)]\xrightarrow{p}0\), using Chebyshev’s inequality. Combining these two limits proves \(Q_{n,m}(t)\xrightarrow{p}F_J(t)\). \(\square\)

The variance condition is a substantive requirement: it says the conditional subsampling distribution is not persistently random across datasets. It can be verified for many statistics with suitable stability properties, but should not be silently inferred from convergence of the full-sample statistic alone.

The \(m\)-out-of-\(n\) Bootstrap

The \(m\)-out-of-\(n\) bootstrap draws \(m\) observations with replacement from the empirical distribution and recomputes the estimator. It uses the same smaller-sample scaling as subsampling, typically \(r_m(T_m^*-T_n)\). Unlike subsampling, its resampled observations can repeat; the reduced resample size, rather than the removal of replacement, is the main change from the ordinary bootstrap.

In the uniform endpoint example, take \(m\to\infty\) and \(m/n\to0\). The number of original observations above \(M_n-x/m\) is approximately \(nx/m\) for fixed \(x>0\): the corresponding interval has length \(x/m\), and its expected count is \(nx/m\), which diverges because \(n/m\to\infty\). The fraction of observations in that interval is therefore approximately \(x/m\). Conditional on the data, the probability that all \(m\) bootstrap draws miss it is approximately

This is the target distribution function at \(-x\) for \(m(M_m^*-M_n)\). The approximation is valid for fixed \(x\), with \(m\to\infty\), \(m/n\to0\), and the stated uniform model; it is not a guarantee for arbitrary estimators. It illustrates how a smaller bootstrap sample can make the empirical endpoint less influential at the resampling scale.

Choosing the Resample Size and Avoiding Overclaims

A practical choice of \(m\) must balance two competing requirements. It must be large enough for the asymptotic approximation to be relevant, yet small enough relative to \(n\) to reduce the empirical-distribution defect that causes the failure. There is rarely a single universally optimal value. A defensible analysis checks whether conclusions are stable across several choices of \(m\) that increase while remaining below \(n\), and explains the rate used to scale each resampled statistic.

For a root-\(n\) estimator, the natural subset scaling is \(\sqrt m\); for the uniform maximum, it is \(m\). More generally, if the established limit uses \(r_n\), the resample statistic should use \(r_m\). Studentization can be useful when the target is a standardized statistic, but it does not automatically remove nonregularity or high-dimensional approximation error. Likewise, increasing the number of resamples reduces simulation noise but does not fix an inconsistent resampling scheme.

In high-dimensional problems, the relevant limit may involve a maximum over a growing number of coordinates, and the validity of any resampling method depends on how the dimension grows and on the dependence and tail assumptions. The ordinary bootstrap should not be declared invalid merely because the dimension is large; nor should its validity be inferred from a fixed-dimensional argument. The limiting approximation and the conditional resampling approximation both need justification for the actual regime.

• Pitfall: using the ordinary sample-size scaling for a smaller resample. If a statistic has rate \(r_m\) on samples of size \(m\), multiplying its resampled error by \(r_n\) can inflate or shrink the distribution incorrectly. Use the scale dictated by the subset-size limit.
• Pitfall: treating \(m/n\to0\) as a complete proof. A smaller resample can help, but consistency still requires a valid limit law and concentration of the conditional resampling distribution.
• Pitfall: reading agreement among resamples as proof of validity. Many resamples can estimate the chosen conditional distribution precisely while that distribution still targets the wrong law. Monte Carlo precision and statistical validity are different questions.