Studentized Bootstrap Tests and Confidence Regions

Why Studentize a Bootstrap Statistic?

A bootstrap distribution of an estimator’s raw error can be difficult to interpret when its coordinates have different scales. For example, a difference of 2 may be substantial for a parameter measured in millimeters but negligible for one measured in thousands of dollars. Studentization addresses this by dividing estimation errors by estimated standard errors. It puts coordinates on comparable scales while retaining their dependence when the statistic is constructed jointly.

This tutorial develops bootstrap-t tests and confidence regions for a fixed-dimensional vector parameter. The resampling scheme is the empirical bootstrap: resample the observed independent sampling units, as in the earlier material on choosing resampling units and the empirical bootstrap for smooth vector parameters. The key addition here is to recompute standard errors in each bootstrap sample and use the resulting studentized statistics for calibration.

Definitions and Construction

Definition: Let $\theta\in\mathbb{R}^p$ be a parameter and $\hat{\theta}$ an estimator based on $n$ independent sampling units. Write $\hat{\theta}^{*}$ for the estimator from an empirical bootstrap resample. Let $\widehat{\operatorname{se}}_j$ and $\widehat{\operatorname{se}}_j^{*}$ estimate the standard errors of $\hat{\theta}_j$ and $\hat{\theta}_j^{*}$, respectively. The coordinate-wise studentized statistics are $$ T_j=\frac{\hat{\theta}_j-\theta_j}{\widehat{\operatorname{se}}_j}, \qquad T_j^{*}=\frac{\hat{\theta}_j^{*}-\hat{\theta}_j}{\widehat{\operatorname{se}}_j^{*}}. $$ The bootstrap statistic is centered at $\hat{\theta}$, because the empirical bootstrap distribution is generated from the observed data, not from a population in which $\theta$ is known.

The numerator and denominator must refer to the same coordinate and the same resampling unit. In each resample, recompute both the estimator and its standard error using the same estimation procedure used for the original data. A bootstrap replicate with a zero or undefined standard error cannot be assigned an ordinary studentized statistic; how such replicates are handled must be specified, and frequent occurrence is a warning that the procedure may be unsuitable.

For a single coordinate, let $q^{*}_{j,\gamma}$ denote the conditional $\gamma$-quantile of $T_j^{*}$ given the observed data. The bootstrap-t interval with nominal coverage $1-\alpha$ is

\left[ \hat{\theta}_j-q^{*}_{j,1-\alpha/2}\widehat{\operatorname{se}}_j,\; \hat{\theta}_j-q^{*}_{j,\alpha/2}\widehat{\operatorname{se}}_j \right].

The quantiles appear in reverse order at the two endpoints. This follows by starting with the pivotal inequality $q^{*}_{j,\alpha/2}\leq T_j\leq q^{*}_{j,1-\alpha/2}$ and solving for $\theta_j$.

Worked Example: A Bootstrap-t Interval With Unequal Units

Suppose two coordinates of an estimator are $\hat{\theta}_1=2.4$ and $\hat{\theta}_2=0.18$, with estimated standard errors $0.8$ and $0.03$. The conditional bootstrap-t quantiles at probabilities $0.025$ and $0.975$ are $(-2.1,2.3)$ for coordinate 1 and $(-1.9,2.2)$ for coordinate 2. Find the two 95% marginal intervals.

For coordinate 1, use the upper quantile at the lower endpoint and the lower quantile at the upper endpoint: $$ [2.4-2.3(0.8),\;2.4-(-2.1)(0.8)] =[0.56,\;4.08]. $$ For coordinate 2, the calculation is $$ [0.18-2.2(0.03),\;0.18-(-1.9)(0.03)] =[0.114,\;0.237]. $$

These are marginal intervals: each is calibrated for its own coordinate. Their individual 95% coverage does not imply that both coordinates are covered with probability 95%. Also, the different numerical widths reflect both different units and different uncertainty; the studentized quantiles, rather than a common raw-error cutoff, determine the calibration.

Joint Studentization and Confidence Regions

Coordinate-wise intervals are useful when separate statements about individual parameters are wanted. If the goal is a single statement about the whole vector, the calibration must account for simultaneous coverage and, where relevant, dependence among coordinates. One option is a maximum of absolute coordinate-wise studentized errors. Another is a quadratic statistic that uses an estimated covariance matrix.

Let $\widehat V$ estimate the covariance matrix of $\sqrt n(\hat{\theta}-\theta)$, and let $\widehat V^{*}$ be its counterpart computed in each bootstrap resample. Assume these matrices are positive definite when their inverses are used. Define

Q=n(\hat{\theta}-\theta)^{\mathsf T}\widehat V^{-1}(\hat{\theta}-\theta), \qquad Q^{*}=n(\hat{\theta}^{*}-\hat{\theta})^{\mathsf T} (\widehat V^{*})^{-1}(\hat{\theta}^{*}-\hat{\theta}).

If $c^{*}_{1-\alpha}$ is the conditional $1-\alpha$ quantile of $Q^{*}$, the corresponding joint confidence region is

\mathcal C_{1-\alpha} =\left\{\theta\in\mathbb R^p: n(\hat{\theta}-\theta)^{\mathsf T}\widehat V^{-1} (\hat{\theta}-\theta)\leq c^{*}_{1-\alpha}\right\}.

An alternative simultaneous region uses $M=\max_{1\leq j\leq p}|T_j|$ and $M^{*}=\max_{1\leq j\leq p}|T_j^{*}|$. If $m^{*}_{1-\alpha}$ is the conditional quantile of $M^{*}$, the region is the intersection of the coordinate constraints $|\hat{\theta}_j-\theta_j|\leq m^{*}_{1-\alpha}\widehat{\operatorname{se}}_j$. Because each bootstrap maximum is computed across all coordinates, its quantile calibrates the joint event rather than $p$ separate marginal events.

Worked Example: A Quadratic Bootstrap Confidence Region

Suppose $p=2$, $n=100$, $\hat{\theta}=(1,2)^{\mathsf T}$, and the estimated covariance matrix of $\sqrt n(\hat{\theta}-\theta)$ is $$ \widehat V=\begin{pmatrix}4&1\\1&1\end{pmatrix}. $$ Assume the bootstrap $0.95$ quantile of $Q^{*}$ is $6.1$. Determine whether two candidate parameter values belong to the region.

The determinant of $\widehat V$ is $4(1)-1(1)=3$, so $$ \widehat V^{-1}=\frac{1}{3}\begin{pmatrix}1&-1\\-1&4\end{pmatrix}. $$ For $\theta=(1.4,2.2)^{\mathsf T}$, the difference $\hat{\theta}-\theta=(-0.4,-0.2)^{\mathsf T}$. Its quadratic form is $$ 100\,\frac{(-0.4)^2-2(-0.4)(-0.2)+4(-0.2)^2}{3} =100\,\frac{0.16-0.16+0.16}{3} \approx 5.33. $$ Since $5.33\leq6.1$, this candidate is inside the region.

For $\theta=(1.5,2.2)^{\mathsf T}$, the difference is $(-0.5,-0.2)^{\mathsf T}$, and the statistic is $$ 100\,\frac{0.25-0.20+0.16}{3}=7. $$ Since $7>6.1$, this candidate is outside. The off-diagonal entries in $\widehat V$ matter: the region uses the estimated joint covariance, not just the two marginal standard errors.

Why Studentization Helps With Scale

Result: Studentized coordinate statistics are invariant in absolute value to a change of units in each coordinate, provided the estimator and its standard error are transformed consistently. The quadratic statistic is invariant under any invertible linear change of coordinates when its covariance estimate is transformed consistently.

To verify the coordinate statement, replace $\theta_j$ by $\phi_j=a_j\theta_j+b_j$, where $a_j\neq0$ and $b_j$ is constant. Then $\hat{\phi}_j-\phi_j=a_j(\hat{\theta}_j-\theta_j)$, while the transformed standard error is $|a_j|\widehat{\operatorname{se}}_j$. Therefore $$ \left|\frac{\hat{\phi}_j-\phi_j}{|a_j|\widehat{\operatorname{se}}_j}\right| =\left|\frac{\hat{\theta}_j-\theta_j}{\widehat{\operatorname{se}}_j}\right|. $$ The same identity holds for the bootstrap replicate. Thus the absolute studentized coordinate and the maximum of these absolute statistics do not depend on whether, for example, a measurement is recorded in meters or centimeters.

For the quadratic statement, write a change of coordinates as $\phi=A\theta+b$, where $A$ is invertible. The transformed covariance estimate is $A\widehat V A^{\mathsf T}$, whose inverse is $A^{-\mathsf T}\widehat V^{-1}A^{-1}$. Substituting the transformed difference $A(\hat{\theta}-\theta)$ into the quadratic form cancels $A$ and $A^{-1}$, leaving the original value of $Q$. This establishes invariance, not automatic validity: the bootstrap still has to approximate the sampling distribution, and the covariance estimates must be well behaved.

Calibration and Coverage: A Result With Conditions

The following result states conditions under which the quadratic bootstrap region has asymptotically correct coverage. Write $P^{*}$ for probability conditional on the observed data. Conditional convergence in distribution in probability means that the conditional distribution functions converge in probability at every continuity point of the limiting distribution.

Theorem: Suppose $p$ is fixed, $\widehat V$ is positive definite with probability tending to one, and, for a positive-definite matrix $V$, $$ \sqrt n(\hat{\theta}-\theta)\xrightarrow{d}N_p(0,V), \qquad \widehat V\xrightarrow{p}V. $$ Suppose also that the empirical bootstrap satisfies $$ \sqrt n(\hat{\theta}^{*}-\hat{\theta})\xrightarrow{d}N_p(0,V) \quad\text{conditionally in probability}, \qquad \widehat V^{*}\xrightarrow{p^{*}}V \quad\text{conditionally in probability}. $$ If the bootstrap statistic $Q^{*}$ is defined when $\widehat V^{*}$ is invertible, and $c^{*}_{1-\alpha}$ is its conditional $1-\alpha$ quantile, then $$ P\{\theta\in\mathcal C_{1-\alpha}\}\longrightarrow 1-\alpha $$ for $0<\alpha<1$.

Proof. By the assumed sampling convergence and $\widehat V\xrightarrow{p}V$, continuity of matrix inversion at a positive-definite matrix and Slutsky’s theorem give $$ Q\xrightarrow{d} Z^{\mathsf T}V^{-1}Z,\qquad Z\sim N_p(0,V). $$ Writing $Z=V^{1/2}U$ with $U\sim N_p(0,I_p)$, the limit is $U^{\mathsf T}U$, which has a chi-squared distribution with $p$ degrees of freedom. This distribution is continuous.

The conditional bootstrap convergence and conditional consistency of $\widehat V^{*}$ give the same chi-squared limit for $Q^{*}$, conditionally in probability. For a continuous limiting distribution, convergence of the conditional distribution at its quantile implies $c^{*}_{1-\alpha}\xrightarrow{p}\chi^2_{p,1-\alpha}$, where $\chi^2_{p,1-\alpha}$ is the $1-\alpha$ quantile of that chi-squared distribution.

The event that the true $\theta$ is in the region is exactly $Q\leq c^{*}_{1-\alpha}$. Since $Q$ converges to a continuous distribution and the random cutoff converges in probability to its $1-\alpha$ quantile, the probability of this event converges to $1-\alpha$. Therefore the stated coverage holds. $\square$

The theorem is an asymptotic result, not a guarantee of exact coverage for a finite sample. Its assumptions require a regular, fixed-dimensional limit and a valid conditional bootstrap approximation. They do not cover every estimator or every data structure. In particular, a studentized bootstrap does not by itself repair non-smoothness, non-identifiability, dependence handled by an unsuitable resampling unit, or unstable covariance estimation.

Testing a Vector Null Hypothesis

For the point null hypothesis $H_0:\theta=\theta_0$, calculate the observed studentized statistic using $\hat{\theta}-\theta_0$. For a quadratic test, this is $$ Q_0=n(\hat{\theta}-\theta_0)^{\mathsf T}\widehat V^{-1}(\hat{\theta}-\theta_0). $$ Compare $Q_0$ with the bootstrap quantile of $Q^{*}$, which is centered at $\hat{\theta}$. Equivalently, estimate a bootstrap p-value by counting bootstrap statistics at least as large as $Q_0$. For a maximum test, use $M_0=\max_j|(\hat{\theta}_j-\theta_{0j})/\widehat{\operatorname{se}}_j|$ and compare it with the bootstrap distribution of $M^{*}$.

Worked Example: A Simultaneous Bootstrap Test

A study tests $H_0:\theta=(\theta_1,\theta_2)^{\mathsf T}=(0,0)^{\mathsf T}$. The observed estimates are $(1.0,0.12)$, with standard errors $(0.5,0.10)$. The observed maximum statistic is therefore $$ M_0=\max\left\{\left|\frac{1.0}{0.5}\right|,\left|\frac{0.12}{0.10}\right|\right\} =\max\{2.0,1.2\}=2.0. $$

Among $B=999$ bootstrap replicates, 24 maximum statistics are at least $2.0$. Using the finite-resample estimate $$ \hat p=\frac{1+\#\{b:M_b^{*}\geq M_0\}}{B+1}, $$ gives $\hat p=(1+24)/(999+1)=0.025$. At significance level $0.05$, this procedure rejects the point null. The maximum statistic calibrates the search across both coordinates; comparing each coordinate separately to a marginal 5% cutoff would not provide the same simultaneous test.

Common Pitfall and Practical Checks

A common mistake is to calculate bootstrap standard errors but then use an unstudentized bootstrap distribution of raw errors as though it were a bootstrap-t distribution. These are different procedures: the bootstrap-t statistic divides each replicate’s error by that replicate’s estimated standard error. Another mistake is to use a separate 95% interval for every coordinate and claim 95% coverage for the whole vector. Marginal coverage does not establish simultaneous coverage. Use a joint statistic, such as the maximum or the quadratic form, when the inferential claim concerns the vector as a whole.

Before interpreting a result, check that the resampling units match the sampling design, that standard errors are recomputed in every resample, and that the chosen joint statistic matches the intended claim. Also inspect whether the bootstrap statistics and covariance estimates are stable. Studentization improves scale comparability and can improve calibration, but it is not a substitute for a suitable model, a suitable resampling scheme, or adequate information in the data.

Check Your Understanding

Use the definitions and results above to answer these questions.

Why is the bootstrap-t statistic centered at $\hat{\theta}$ rather than at the unknown $\theta$?
Given bootstrap-t quantiles $q^{*}_{0.025}=-1.8$ and $q^{*}_{0.975}=2.1$, write the 95% interval formula for an estimate $\hat{\theta}_j$ with standard error $s_j$.
Why do separate coordinate-wise 95% intervals not automatically give a 95% joint confidence region?
What role does the off-diagonal covariance play in a quadratic studentized statistic?
Name one condition in the theorem that is not guaranteed merely by using a bootstrap.