Studentized Permutation Tests Under Unequal Covariances

Why Permuting Labels Can Mislead

A label-permutation test repeatedly reallocates the observed vectors to groups and compares the observed statistic with its values after reallocation. If the groups have the same distribution, labels are exchangeable, and this procedure gives an exact test under the randomization distribution. But a common scientific null is weaker: the groups have the same mean while their covariances may differ. Under that null, the observations need not be exchangeable. A permutation test based on an unstudentized difference can therefore use the wrong reference distribution.

Studentization addresses this mismatch by scaling the difference in sample means by an estimate of its covariance. The key is to recompute that scale for every permutation. With fixed dimension and suitable moment and nonsingularity conditions, the observed and permuted studentized statistics have the same limiting chi-square distribution even when the group covariances are unequal. This is an asymptotic result, not a claim of exact finite-sample validity under the equal-means null.

Setup and the Studentized Statistic

Suppose $X_{11},\ldots,X_{1n_1}$ are independent and identically distributed $p$-dimensional vectors with mean $\mu_1$ and covariance $\Sigma_1$, and $X_{21},\ldots,X_{2n_2}$ are independent and identically distributed vectors with mean $\mu_2$ and covariance $\Sigma_2$. The two groups are independent. Write $\bar X_g$ and $S_g$ for group $g$'s sample mean and unbiased sample covariance matrix, and let $n=n_1+n_2$.

Definition: The weak equal-means null hypothesis is $H_0:\mu_1=\mu_2$; it places no equality restriction on $\Sigma_1$ and $\Sigma_2$. The studentized Wald statistic for this null is $$ W=(\bar X_1-\bar X_2)^\top \left(\frac{S_1}{n_1}+\frac{S_2}{n_2}\right)^{-1} (\bar X_1-\bar X_2), $$ provided the estimated covariance matrix in parentheses is invertible.

The matrix inside the inverse estimates the covariance of $\bar X_1-\bar X_2$, since independence gives $\operatorname{Cov}(\bar X_1-\bar X_2)=\Sigma_1/n_1+\Sigma_2/n_2$. The statistic is a squared, covariance-scaled distance between the two sample means. A permutation replicate is formed by reallocating the pooled observations to groups of sizes $n_1$ and $n_2$, then recomputing both sample means, both sample covariances, and $W$. Keeping the original covariance estimate fixed while permuting only the numerator is not this studentized procedure.

What Goes Wrong Without Studentization

Assume $n_1/n\to\lambda$, where $0<\lambda<1$, and consider the null $\mu_1=\mu_2$. The multivariate central limit theorem gives $\sqrt n(\bar X_1-\bar X_2)\xrightarrow{d}N(0,\Sigma_1/\lambda+\Sigma_2/(1-\lambda))$. A naive statistic based on the unscaled squared Euclidean distance therefore reflects this covariance. In contrast, after random reassignment, both permuted groups are drawn from the same pooled empirical distribution. Its limiting covariance is the mixture $\Sigma_*=\lambda\Sigma_1+(1-\lambda)\Sigma_2$. Thus the permutation distribution of the unstudentized difference reflects $\Sigma_*/[\lambda(1-\lambda)]$, which need not equal the actual covariance of the scaled difference.

Worked Example: Unequal Covariances Change the Permutation Scale

Let $p=2$, $\lambda=1/4$, and suppose the groups have equal means but $\Sigma_1=\operatorname{diag}(9,1)$ and $\Sigma_2=\operatorname{diag}(1,4)$. For the actual sampling distribution, the limiting covariance of $\sqrt n(\bar X_1-\bar X_2)$ is

\frac{\Sigma_1}{\lambda}+\frac{\Sigma_2}{1-\lambda} =4\begin{pmatrix}9&0\\0&1\end{pmatrix} +\frac43\begin{pmatrix}1&0\\0&4\end{pmatrix} =\begin{pmatrix}112/3&0\\0&28/3\end{pmatrix}.

For a random reassignment, the pooled covariance tends to

\Sigma_*=\tfrac14\Sigma_1+\tfrac34\Sigma_2 =\begin{pmatrix}3&0\\0&13/4\end{pmatrix}.

Consequently the limiting covariance of the permuted $\sqrt n$-scaled mean difference is

\frac{\Sigma_*}{\lambda(1-\lambda)} =\frac{16}{3}\begin{pmatrix}3&0\\0&13/4\end{pmatrix} =\begin{pmatrix}16&0\\0&52/3\end{pmatrix}.

These matrices differ in both coordinates. In particular, an unstudentized squared length does not have the same limiting distribution under the actual sampling and permutation schemes. The discrepancy is not repaired by merely taking more permutations: more permutations estimate the wrong reference distribution more accurately.

Asymptotic Validity of the Studentized Procedure

Here is a fixed-dimension result. Assume $p$ does not vary with $n$, $n_1/n\to\lambda\in(0,1)$, and both group distributions have finite $2+\delta$ moments for some $\delta>0$. Assume $\Sigma_1$ and $\Sigma_2$ are positive definite. These conditions ensure a nondegenerate limiting covariance and allow the sample covariances to estimate their population counterparts. The inverse in $W$ then exists with probability tending to one.

Theorem: Under these conditions and $H_0:\mu_1=\mu_2$, $W\xrightarrow{d}\chi^2_p$. Further, let $W^\pi$ be the statistic recomputed after a uniformly chosen allocation of the pooled observations to groups of sizes $n_1,n_2$. Its conditional permutation distribution converges in probability to $\chi^2_p$. Thus the upper-tail permutation test based on $W$ has asymptotic level $\alpha$ for $0<\alpha<1$, under the weak equal-means null.

Proof. Set $D=\bar X_1-\bar X_2$ and $V=\Sigma_1/n_1+\Sigma_2/n_2$. Under the null, the independent-sample multivariate central limit theorem gives $V^{-1/2}D\xrightarrow{d}N(0,I_p)$. By the law of large numbers, $S_g\xrightarrow{p}\Sigma_g$, so $\hat V=S_1/n_1+S_2/n_2$ satisfies $V^{-1/2}\hat V V^{-1/2}\xrightarrow{p}I_p$. Slutsky's theorem then gives

W=(V^{-1/2}D)^\top (V^{-1/2}\hat V V^{-1/2})^{-1} (V^{-1/2}D)\xrightarrow{d}\chi^2_p.

For the permutation claim, write $S_{\mathrm{pool}}$ for the unbiased covariance of all $n$ pooled observations. The group proportions and the law of large numbers imply $S_{\mathrm{pool}}\xrightarrow{p}\Sigma_*$, where $\Sigma_*=\lambda\Sigma_1+(1-\lambda)\Sigma_2$. Under a random allocation, each permuted group has asymptotic covariance $\Sigma_*$, and the conditional covariance of the difference of permuted means is asymptotically $(1/n_1+1/n_2)\Sigma_*$. The finite-population central limit theorem for random partitions applies conditionally in probability: for a centered finite population with covariance tending to a positive-definite matrix and maximum squared norm divided by $n$ tending to zero, the appropriately covariance-standardized difference between two groups whose proportions stay away from zero and one converges to $N(0,I_p)$. Here the moment assumption ensures the maximum-norm condition in probability: for any $\epsilon>0$, the probability that $\max_i\|X_i-\bar X_{\mathrm{pool}}\|>\epsilon\sqrt n$ is bounded by a constant times $n^{-\delta/2}$, using the finite $2+\delta$ moments and a union bound. The random-partition sample covariances both converge conditionally in probability to $\Sigma_*$, so the recomputed studentizer estimates that same conditional covariance. Applying Slutsky's theorem conditionally yields $W^\pi\xrightarrow{d}\chi^2_p$ in probability. Since the chi-square distribution is continuous, its upper-tail critical quantiles converge to those of $\chi^2_p$; combined with the observed-statistic limit, this gives asymptotic rejection probability $\alpha$. $\square$

The argument depends on matching the studentizer to the covariance in each scheme. In the original sample, it estimates $\Sigma_1/n_1+\Sigma_2/n_2$. In a permutation replicate, it estimates the covariance generated by the pooled mixture. After studentization, both statistics have the same standard normal quadratic-form limit.

Computing the Statistic and Its Permutations

Worked Example: Calculating a Two-Dimensional Wald Statistic

Suppose $n_1=20$, $n_2=30$, the observed mean difference is $\bar X_1-\bar X_2=(1,0.5)^\top$, and the sample covariance matrices are $S_1=\operatorname{diag}(4,1)$ and $S_2=\operatorname{diag}(1,4)$. The estimated covariance of the difference is

\hat V=\frac{S_1}{20}+\frac{S_2}{30} =\begin{pmatrix}1/5+1/30&0\\0&1/20+4/30\end{pmatrix} =\begin{pmatrix}7/30&0\\0&11/60\end{pmatrix}.

Its inverse is $\operatorname{diag}(30/7,60/11)$. Therefore

W=1^2\frac{30}{7}+(0.5)^2\frac{60}{11} =\frac{30}{7}+\frac{15}{11} =\frac{435}{77}\approx5.65.

For an asymptotic chi-square reference with two degrees of freedom, the 5% upper critical value is approximately 5.99, so this value does not exceed that cutoff. A permutation test instead compares 5.65 with the empirical distribution of the recomputed $W^\pi$ values; it need not use the chi-square cutoff in a finite sample.

Worked Example: Recomputing a Permutation Statistic

Consider one-dimensional observations with original groups $\{0,2\}$ and $\{1,5\}$, each of size two. For a group of two values, the unbiased sample variance is half the squared difference between its observations. The original means are 1 and 3, and the sample variances are 2 and 8. Thus

W=\frac{(1-3)^2}{2/2+8/2}=\frac45=0.8.

There are $\binom42=6$ allocations for the first group. For allocation $\{0,1\}$ versus $\{2,5\}$, the means are $0.5,3.5$, the sample variances are $0.5,4.5$, and $W^\pi=(-3)^2/(0.5/2+4.5/2)=3.6$. For allocation $\{0,5\}$ versus $\{1,2\}$, the means are $2.5,1.5$, the variances are $12.5,0.5$, and $W^\pi=1^2/(12.5/2+0.5/2)=2/13$. The complementary allocations give the same values because exchanging the group labels changes the sign of the mean difference but not its squared, studentized value. The six statistics are therefore $0.8,0.8,3.6,3.6,2/13,2/13$.

The inclusive permutation tail proportion for the observed value is $4/6$, since four of the six statistics are at least 0.8. This small example illustrates the mechanics, not the asymptotic unequal-covariance guarantee: with only two observations per group, the conditions of the theorem and its large-sample approximation are not established.

Interpretation, Scope, and a Common Pitfall

Studentized permutation tests are useful when the inferential target is equality of mean vectors and unequal group covariances are plausible. They preserve the observed group sizes, use the data to estimate the appropriate scale in every allocation, and can offer a permutation-based calibration without requiring the groups to have identical distributions. The asymptotic statement is for fixed $p$; it does not by itself justify growing dimension, singular covariance estimates, dependent observations, or small-sample exactness under unequal distributions. The sampling units and any design restrictions must also be respected when reallocating labels.

Common mistake: “The null says the means are equal, so the labels are exchangeable and every permutation test is exact.” Equal means do not imply equal distributions. With unequal covariances, exchangeability generally fails, and an unstudentized permutation reference can have the wrong scale. The correction is to distinguish exact randomization validity under exchangeability from asymptotic validity under the weaker equal-means null, and to recompute the covariance studentizer in every permutation.

A second practical error is to permute the observations but retain the original groups' covariance estimates. That uses a scale tied to the original allocation rather than to the permuted groups. For the studentized procedure described here, each allocation produces its own two sample means and covariance matrices before its Wald statistic is calculated.

Check Your Understanding

Use the definitions and assumptions above to answer the following questions.

Why does equality of the two mean vectors not, by itself, make the pooled observations exchangeable?
In the two-group setting, what covariance does the studentizer estimate for the observed difference in sample means?
Why can the unstudentized permutation distribution have a different limiting covariance from the sampling distribution under the weak null?
What quantities must be recomputed for each permutation replicate in the studentized procedure?
Name two assumptions or settings that fall outside the theorem as stated.