Finite-Sample Calibration of Robust Location Tests

Why Finite-Sample Calibration Matters

A robust one-sample location test can have a well-justified asymptotic reference distribution and still make unreliable decisions at the sample size actually observed. This distinction matters when a nominal 5% test is used to support a clinical or scientific conclusion: a test that rejects too often under the null has a false-positive rate above its stated level, while an overly conservative test loses opportunities to detect a real shift.

The usual asymptotic argument describes the statistic as the sample size grows. It does not guarantee accurate tail probabilities at a particular finite $n$. Robust estimators may have non-negligible small-sample bias, and their estimated covariance can be variable or unstable. Procedures involving data-dependent subsets or weights can also have irregular finite-sample distributions. Calibration asks a specific question: under the null model, what reference distribution should be used for this test statistic at this sample size?

The Robust Wald Statistic and Its Large-Sample Reference

Definition: Let $X_1,\ldots,X_n$ be independent observations in $\mathbb{R}^p$, with common location parameter $\mu$. Let $\hat{\mu}_n$ estimate $\mu$, and let $\hat{V}_n$ estimate the asymptotic covariance matrix of $\sqrt{n}(\hat{\mu}_n-\mu)$. For testing $H_0:\mu=\mu_0$, the robust Wald statistic is $$ W_n=n(\hat{\mu}_n-\mu_0)^\mathsf{T}\hat{V}_n^{-1}(\hat{\mu}_n-\mu_0), $$ provided $\hat{V}_n$ is invertible. The same statistic can be written as the squared Mahalanobis length of $\sqrt n(\hat{\mu}_n-\mu_0)$, measured using $\hat V_n$.

A standard large-sample calibration is available under regularity conditions. In particular, suppose $H_0$ holds, $\sqrt n(\hat{\mu}_n-\mu_0)\xrightarrow{d}N(0,V)$ for positive-definite $V$, and $\hat V_n\xrightarrow{p}V$. Then $W_n\xrightarrow{d}\chi^2_p$. This gives an asymptotic level-$\alpha$ test that rejects for $W_n>\chi^2_{p,1-\alpha}$, the $1-\alpha$ quantile of the chi-square distribution with $p$ degrees of freedom.

The justification is Slutsky’s theorem and the continuous mapping theorem. Write $Z_n=\sqrt n(\hat{\mu}_n-\mu_0)$. Under the null, $Z_n\xrightarrow{d}Z\sim N(0,V)$, while $\hat V_n^{-1}\xrightarrow{p}V^{-1}$, because matrix inversion is continuous at the nonsingular matrix $V$. Therefore $$ W_n=Z_n^\mathsf{T}\hat V_n^{-1}Z_n\xrightarrow{d}Z^\mathsf{T}V^{-1}Z. $$ If $V^{-1/2}$ is the symmetric inverse square root of $V$, then $V^{-1/2}Z\sim N(0,I_p)$, and $Z^\mathsf{T}V^{-1}Z$ is the sum of squares of $p$ independent standard normal variables. It consequently has a $\chi^2_p$ distribution. This proves the stated limit, but it does not establish that the finite-$n$ distribution of $W_n$ is chi-square. $\square$

Worked Example: A Finite-Sample Decision Can Differ

Suppose a robust Wald test has $p=2$, $n=20$, and observed statistic $W_n=6.4$. Its asymptotic 5% critical value is $\chi^2_{2,0.95}=5.991$, so the asymptotic rule rejects. For two degrees of freedom, the chi-square upper-tail probability at 6.4 is $\exp(-6.4/2)=\exp(-3.2)\approx0.0408$.

Now suppose a null simulation for this particular procedure and sample size gives a 95th percentile of 7.10, and 79 of 999 simulated statistics are at least 6.4. These are illustrative simulation results, not universal reference values. The simulated-tail estimate with the usual plus-one adjustment is $(79+1)/(999+1)=0.080$. The finite-sample calibration does not reject at 5%: 6.4 is below 7.10, and the simulated tail estimate exceeds 0.05. The example shows how the same observed statistic can lead to different decisions under different calibrations.

Calibrating by Null Simulation

A direct way to assess a robust test at finite $n$ is to generate data under the null and apply the test exactly as it will be applied to the observed data. For each simulated sample, recompute the robust location estimate, any scale or covariance estimates, any consistency corrections, and the final statistic. Reusing an estimate from the observed data or simulating only the final statistic’s limiting approximation does not reproduce the finite-sample procedure.

Null simulation: For a specified null distribution $F_0$, generate $B$ independent samples of size $n$ from $F_0$, calculate the statistic $W_n^{(b)}$ for each sample using the complete analysis procedure, and use the simulated distribution as a reference. An empirical upper-tail probability for an observed statistic $w$ is $$ \hat p_{\mathrm{MC}}=\frac{1+\sum_{b=1}^{B} I(W_n^{(b)}\geq w)}{B+1}. $$ The plus-one adjustment prevents a zero reported probability and gives a valid finite-sample Monte Carlo test when the observed and simulated statistics are generated under the same fully specified null distribution.

Simulation has to represent the null model relevant to the test. If the null distribution is fully specified, the procedure above can provide a finite-sample test. If the null distribution has unknown nuisance parameters, plugging in estimates and simulating from the fitted model generally gives an approximation, not an exact finite-sample test. Exactness in the presence of nuisance parameters requires, for example, a valid pivotal or invariance argument that removes their effect. The fact that an estimator is robust does not by itself remove nuisance parameters or guarantee a pivotal reference distribution.

A further practical issue is Monte Carlo uncertainty. With $B$ simulations, the estimated tail probability changes in increments of $1/(B+1)$. Near a decision threshold, more simulations can reduce numerical uncertainty, but they cannot repair a misspecified null model or an incorrectly implemented analysis pipeline.

Worked Example: A Monte Carlo Test at the 5% Level

Suppose $B=999$ null samples are generated from a fully specified null model. The same robust fitting algorithm and test-statistic calculation are run on each sample and on the observed data. If 18 simulated statistics are at least as large as the observed statistic, then $$ \hat p_{\mathrm{MC}}=\frac{1+18}{999+1}=0.019. $$ Because $0.019<0.05$, the Monte Carlo test rejects at the 5% level. The smallest possible adjusted value with 999 simulations is $1/1000=0.001$, so this simulation size does not support reporting arbitrarily small tail probabilities.

This conclusion has a finite-sample justification when the observed statistic and the 999 simulated statistics are exchangeable under the null—that is, they have the same distribution and are treated symmetrically. If the statistic has a continuous distribution, ties occur with probability zero, and the rank of the observed statistic among the 1,000 values is uniform under the null. Counting simulated statistics at least as large as the observed one therefore produces a valid rank-based test. If ties can occur, counting ties against rejection gives a conservative test; randomized tie-breaking can restore exact size. If simulations instead use a plug-in nuisance estimate, the exchangeability argument does not apply automatically.

A Finite-Sample Spatial-Sign Reference Distribution

A useful example of finite-sample calibration is a spatial-sign statistic. For observations not equal to the null location, define their spatial signs by $$ U_i=\frac{X_i-\mu_0}{\lVert X_i-\mu_0\rVert},\qquad i=1,\ldots,n, $$ where $\lVert\cdot\rVert$ is Euclidean length. The statistic below measures the length of the sum of these directions: $$ Q_n=\left\lVert\sum_{i=1}^n U_i\right\rVert^2,\qquad A_n=\frac{p}{n}Q_n. $$ The signs use directions rather than distances, so a single very large observation cannot dominate the sum merely because its distance from $\mu_0$ is large. The reference distribution discussed here requires a specific model condition: under $H_0$, the observations are independent and spherically symmetric about $\mu_0$, with a continuous distribution assigning probability zero to $X_i=\mu_0$.

Result: Under the stated spherical null, each $U_i$ is uniform on the unit sphere in $\mathbb{R}^p$, and the signs are independent. Thus the finite-sample null distribution of $Q_n$ (and of $A_n$) depends only on $n$ and $p$, not on the radial distribution. It can be obtained from that distribution or simulated by generating independent uniform directions.

To see why the distribution is independent of the radial part, write $X_i-\mu_0=R_iU_i$, where $R_i=\lVert X_i-\mu_0\rVert>0$. Spherical symmetry means the distribution is unchanged by every orthogonal transformation. Consequently, conditional on any radius in the support, the direction has the rotation-invariant uniform distribution on the sphere. Since the observations are independent, their directions are independent as well. In the statistic $Q_n$, each radius disappears: $U_i$ depends only on direction. Hence the joint distribution of the signs, and therefore the distribution of $Q_n$, is the same for every spherical radial distribution satisfying the stated conditions. $\square$

This finite-sample property is model-specific. Spherical symmetry is stronger than merely having a location and scatter matrix. In particular, one should not use this reference distribution for an arbitrary nonspherical distribution without a justified transformation or model argument.

Worked Example: The Exact Reference for Two Signs in the Plane

Take $p=2$ and $n=2$. Under the spherical null, the angle $\delta$ between the two independent signs is uniform on $[0,\pi]$. This follows because the difference between two independent uniform directions, reduced to the smaller angle between them, is uniform on that interval. The squared length of their sum is $$ Q_2=\lVert U_1+U_2\rVert^2 = \lVert U_1\rVert^2+\lVert U_2\rVert^2+2U_1^\mathsf{T}U_2 =2+2\cos\delta =4\cos^2(\delta/2). $$ Here $\lVert U_1\rVert=\lVert U_2\rVert=1$ and $U_1^\mathsf{T}U_2=\cos\delta$.

For $0\leq c\leq4$, the event $Q_2>c$ is equivalent to $\delta<2\arccos\sqrt{c/4}$. Since $\delta$ is uniform on an interval of length $\pi$, $$ P(Q_2>c)=\frac{2}{\pi}\arccos\sqrt{\frac{c}{4}}. $$ Setting this probability to $0.05$ gives $$ \frac{2}{\pi}\arccos\sqrt{\frac{c}{4}}=0.05,\qquad c=4\cos^2(0.025\pi)=4\cos^2(\pi/40)\approx3.9754. $$ Because $A_2=(p/n)Q_2=Q_2$ in this example, the exact upper 5% critical value for $A_2$ is also approximately 3.9754.

By contrast, its large-sample approximation uses $\chi^2_2$, whose 95th percentile is 5.991. But $Q_2\leq4$ for every pair of unit vectors, so a rule that rejects only above 5.991 can never reject in this setting. This is an extreme small-sample failure of the asymptotic critical value, not a contradiction: the chi-square result is a limit as $n$ grows, whereas this calculation gives the exact distribution at $n=2$.

Common Pitfalls and Practical Checks

Treating an asymptotic reference as exact. A chi-square limit supports large-sample calibration under its conditions; it does not determine the finite-sample tail probability.
Simulating a different procedure. Recompute every data-dependent step, including robust fitting, scale estimation, and corrections, for each simulated sample.
Calling plug-in simulation exact. When nuisance parameters are estimated, the resulting simulated reference is generally approximate unless a separate argument establishes exactness.
Ignoring the null model. A spherical sign reference is not automatically valid for nonspherical observations. Robustness to outliers does not mean robustness to every distributional assumption.
Overinterpreting a simulated tail estimate. Simulation error, especially near the decision threshold, is separate from uncertainty due to the statistical model and the data.

Finite-sample calibration matters because it aligns a test’s actual null behavior with the error rate stated in its conclusion. When a valid finite-sample reference is unavailable, null simulation can reveal how far the asymptotic approximation is from the behavior of the implemented procedure. The interpretation of that simulation depends on the null model and on whether nuisance parameters have been handled exactly or approximately.

Check Your Understanding

Use the definitions and conditions in this tutorial to answer the following.

What assumptions justify the asymptotic chi-square reference for a robust Wald statistic?
Why must each null-simulated sample go through the complete robust analysis procedure?
When is the plus-one Monte Carlo test finite-sample valid, and what changes when nuisance parameters are estimated?
For $p=2,n=2$, why can the asymptotic 5% chi-square rule never reject the spatial-sign test?
Why does the finite-sample spatial-sign reference above require spherical symmetry?