Studentized Max-T Tests Under Unequal Variances

Why the Scale of a Test Statistic Matters

A maximum-statistic procedure compares the largest test statistic in a family with a resampled reference distribution. This comparison is meaningful only if the statistics are on a suitably comparable scale. In genomic analyses, different features can have very different variability: a raw difference in average expression of 2 units may be striking for a stable feature but unremarkable for a highly variable one.

Studentization addresses this scale problem by dividing an estimated effect by an estimate of its standard error. It does not, by itself, make an invalid resampling scheme valid. In particular, unequal variances can make group labels nonexchangeable under a weak equality-of-means null. The justification for the resampling distribution must still come from the design or from an appropriate asymptotic argument.

For a two-group comparison of independent observations, a common estimate of the difference in means for feature \(j\) is \(\hat{\delta}_j=\bar{x}_{1j}-\bar{x}_{0j}\). Under unequal variances, its Welch-type standard error is

Here \(n_1,n_0\) are the group sample sizes and \(s_{1j},s_{0j}\) are the within-group sample standard deviations for feature \(j\). This formula assumes independent observations across subjects and a valid sample variance within each group; it does not assume the two population variances are equal. For paired, blocked, clustered, or covariate-adjusted designs, the standard error and resampling scheme must instead respect that design.

What the Resampled Maximum Represents

Suppose a resampling scheme produces \(B\) permitted resamples. For each resample \(b\), compute all feature-specific statistics using the same procedure as for the observed data, then retain their maximum. For studentized max-T, this means recomputing both the estimated effect and its standard error in each resample:

For feature \(j\), a single-step adjusted resampling p-value compares its observed \(|T_j|\) with this distribution of maxima. With a Monte Carlo resampling procedure, a common estimate is

The added one avoids reporting a zero p-value when no sampled maximum exceeds the observed statistic. The reference distribution uses the joint resampled outcomes across features, so it retains their dependence to the extent that the resampling scheme preserves it. A max-T adjustment is a familywise procedure, but the validity of its error-rate guarantee depends on the null and assumptions supporting that reference distribution.

Why Studentization Helps Under Unequal Variances

The central benefit is that the standard error puts feature-specific effects in units of their sampling uncertainty. Under independent sampling, finite positive variances, and adequate sample sizes, the central limit theorem gives an approximately normal sampling distribution for a difference in means. Dividing by a consistent estimate of its standard deviation makes the statistic's null scale approximately one, even when the underlying feature variances differ.

A raw difference has a null variance that depends directly on both group variances and sample sizes. Thus, even if all null differences are zero, features with large variances tend to generate larger raw fluctuations and can dominate a raw maximum. Studentizing can reduce this scale imbalance, making the maximum a more coherent measure of the most extreme standardized evidence.

A Scale Calculation for the Permutation Problem

The distinction between a sharp null and a weak null is essential. A sharp null specifies no treatment effect for every unit, so each unit's outcome would be the same under every allowed assignment. A weak equality-of-means null says only that the two population means are equal; it permits individual effects or differing outcome distributions that cancel in the mean. Randomized group assignments alone do not make a test exact for this weaker null.

The issue with raw permutation statistics can be seen in a two-sample setting. Let \(N=n_1+n_0\), and suppose the sample-size fraction in group 1 approaches \(\lambda=n_1/N\), with \(0<\lambda<1\). Under independent groups with population variances \(\sigma_1^2,\sigma_0^2\), the variance of the raw difference in means is

A label permutation draws both groups from the pooled observations. Its pooled variance approaches \(\lambda\sigma_1^2+(1-\lambda)\sigma_0^2\), so the variance of a permuted raw difference approaches

These expressions need not agree. For example, their variance factors after multiplying by \(N\) are respectively \(\sigma_1^2/\lambda+\sigma_0^2/(1-\lambda)\) and \(\sigma_1^2/(1-\lambda)+\sigma_0^2/\lambda\). They coincide when group sizes are asymptotically equal or the variances are equal, but not in general. The raw permutation distribution can therefore be on the wrong scale under a weak null with unequal variances.

Studentization uses a group-specific standard error, rather than a pooled equal-variance standard error. In a two-group mean test, the Welch statistic is asymptotically standard normal under the weak null when the groups are independent, both sample sizes grow with their ratio bounded away from zero and one, the variances are finite and positive, and suitable moment conditions hold. A studentized permutation test can likewise have asymptotic validity for this single contrast under suitable conditions, despite not being an exact test for the weak null.

This univariate argument is not automatically a proof for a maximum across many correlated features. For a family max-T test, the joint distribution of the vector of studentized statistics must also be represented adequately. Studentization of each coordinate corrects its marginal scale; it does not necessarily correct a mismatch in cross-feature dependence under heteroskedasticity. The broad high-dimensional case can require stronger conditions and methods designed to preserve or estimate the relevant joint structure.

Exactness, Diagnostics, and Alternatives

Several checks help identify problems before interpreting adjusted p-values:

• Compare estimated standard errors. A wide range across features signals that raw effects may have strongly unequal null scales.
• Inspect group variances and sample sizes. Extreme imbalance combined with unequal variances is a warning that raw label permutation may be poorly calibrated under a weak null.
• Check the resampling unit. Permuting individual measurements when observations are paired, blocked, or clustered breaks the design structure and can distort both variances and dependence.
• Inspect studentized resample statistics. Heavy tails, unstable standard errors, or frequent zero denominators indicate that normal-scale intuition may be unreliable.
• Check joint behavior, not only marginal scales. Correlated features can determine the tail of the maximum. Matching each feature's variance does not prove that the dependence in the resampled vector is appropriate.

A raw max statistic may be reasonable when feature effects genuinely share a common, interpretable scale and the resampling distribution is valid for the null. Studentized max-T is usually more comparable when standard errors differ, but it can be unstable for features with tiny sample variances and it changes the emphasis from absolute effect size to effect relative to uncertainty. Under nonexchangeable errors, a heteroskedasticity-aware procedure such as a wild bootstrap may be preferable; it too requires assumptions and diagnostics. As covered in the earlier material on strong familywise control and subset pivotality, improving statistic scale is distinct from establishing strong familywise error control. Studentization alone does not establish subset pivotality or any other condition needed for that guarantee.