Common Calibration and Scaling Mistakes in Mean Testing

Why Calibration Errors Matter

A high-dimensional mean test can use a familiar statistic and still have the wrong null rejection rate. The problem may be a mismatch between the variance used to scale a coordinate and its true variance, a centering term omitted from an extreme-value threshold, dependence ignored by a reference distribution, or a permutation that does not preserve the null law. These errors are consequential because a maximum over many coordinates can magnify small scale discrepancies.

The aim here is diagnostic: identify the assumptions that make a calibration correct, and see what fails when one of them is broken. Unless stated otherwise, a test is calibrated under a null hypothesis of zero mean. For an independent sample $X_1,\ldots,X_n$ of $p$-dimensional vectors, write $\bar X_j=n^{-1}\sum_{i=1}^n X_{ij}$ for coordinate $j$. If its variance $\sigma_j^2$ is known, the standardized mean is $Z_j=\sqrt n\,\bar X_j/\sigma_j$. A maximum test examines $\max_{1\le j\le p}|Z_j|$, or its square.

Definition: A test is calibrated at level $\alpha$ under a null distribution if its rejection probability under that distribution is at most $\alpha$ (and often converges to $\alpha$ in an asymptotic calibration). A calibration is valid only for the statistic and null law to which its reference distribution applies.

Variance Scale Must Match the Statistic

Studentization divides an estimated mean by an estimated standard error. Its purpose is to put coordinates on comparable scales, but the numerator and denominator must use the same variance convention. In particular, variance is the square of standard deviation: dividing by $\hat\sigma_j^2$ when the numerator is on a standard-deviation scale is not a harmless change of notation.

A useful way to see the effect is to suppose, for the moment, that the estimated standard deviation is asymptotically a fixed multiple of the truth: $\hat\sigma_j/\sigma_j\to c$, with $c>0$. Then the reported standardized coordinate behaves like $Z_j/c$, not $Z_j$. For a maximum, a constant scale error changes the leading order of the maximum; it does not disappear merely because the sample size grows.

Worked Example: A Standard Deviation Reported on the Wrong Scale

Suppose under the null that $X_{ij}$ are independent $N(0,4)$ variables across observations and coordinates. The true standard deviation is $\sigma=2$, so $Z_j=\sqrt n\,\bar X_j/2\sim N(0,1)$. Imagine that an analyst mistakenly uses $1$ as the standard deviation. The reported statistic is then $T_j=\sqrt n\,\bar X_j=2Z_j$.

For a fixed coordinate, $T_j\sim N(0,4)$, not $N(0,1)$. For the maximum $T_{\max}=\max_j|T_j|$, the event $T_{\max}>\sqrt{2\log p}$ is the same as $\max_j|Z_j|>\sqrt{2\log p}/2$. The latter threshold is only half the usual leading-order threshold for $p$ independent standard normal coordinates. In fact, the maximum of those coordinates grows on the order of $\sqrt{2\log p}$, so this wrongly scaled test rejects with probability tending to one.

If instead the analyst divides by $4$, the variance, then $T_j=Z_j/2$. The threshold is effectively twice as large on the correct standard-normal scale, and rejection probability tends to zero. Both errors arise from mixing up standard deviation and variance; they produce opposite distortions.

Result: For an independent standard-normal maximum $M_p=\max_{1\le j\le p}|Z_j|$, the centered squared maximum $M_p^2-2\log p+\log\log p$ has a nondegenerate limit. If every coordinate is instead multiplied by a fixed positive constant $a\ne1$, its squared maximum has leading order $2a^2\log p$, so the standard calibration no longer applies.

In practice, consistency of a scale estimate for each fixed coordinate need not by itself guarantee a correct maximum calibration when $p$ grows. To preserve the centered squared-maximum limit, a relative scale error that is common across coordinates must typically be small enough that its effect on a quantity of order $\log p$ vanishes. For example, if $\hat\sigma_j/\sigma_j=1+\delta_p$ uniformly and $\delta_p\log p\to0$, then the scale perturbation in a squared maximum is negligible at the centered order. This is a diagnostic condition, not a universal substitute for the assumptions of a particular test.

Extreme-Value Centering: The Missing Logarithm

For independent standard normal coordinates, using $2\log p$ as the squared threshold gets the leading growth right but misses a second-order term. That term matters for a fixed target significance level. The next result derives the appropriate centering and also shows why a fixed-argument limit cannot simply be evaluated at a threshold that changes with $p$.

Theorem: Let $Z_1,\ldots,Z_p$ be independent $N(0,1)$ variables and $M_p=\max_j|Z_j|$. For each fixed real $x$, $$ P\left(M_p^2-2\log p+\log\log p\le x\right) \longrightarrow \exp\left(-\frac{1}{\sqrt{\pi}}e^{-x/2}\right). $$

Proof. Set $u_p(x)=\sqrt{2\log p-\log\log p+x}$, which is positive for all sufficiently large $p$ when $x$ is fixed. If $\bar\Phi(u)=P(Z_1>u)$, independence gives $$ P(M_p\le u_p(x))=\{1-2\bar\Phi(u_p(x))\}^p. $$ The standard normal tail satisfies $\bar\Phi(u)\sim \phi(u)/u$ as $u\to\infty$, where $\phi(u)=(2\pi)^{-1/2}e^{-u^2/2}$. Therefore $$ p\,2\bar\Phi(u_p(x)) \sim p\frac{\sqrt{2/\pi}}{u_p(x)} \exp\left\{-\frac{u_p(x)^2}{2}\right\}. $$ Here $e^{-u_p(x)^2/2}=p^{-1}(\log p)^{1/2}e^{-x/2}$, and $u_p(x)\sim\sqrt{2\log p}$. Hence $$ p\,2\bar\Phi(u_p(x))\longrightarrow \frac{1}{\sqrt{\pi}}e^{-x/2}. $$ Since $2\bar\Phi(u_p(x))\to0$, the limit $(1-a_p)^p\to e^{-\lambda}$ when $pa_p\to\lambda$ yields the stated result. $\square$

Worked Example: Why the Leading Threshold Is Not Enough

Let $c$ be fixed and compare the threshold $2\log p+c$ for $M_p^2$ with the correctly centered threshold. The first threshold corresponds to $M_p>\sqrt{2\log p+c}$. A direct union bound and the normal tail approximation give $$ P(M_p^2>2\log p+c) \le pP\left(|Z_1|>\sqrt{2\log p+c}\right) \sim \frac{e^{-c/2}}{\sqrt{\pi\log p}} \longrightarrow 0. $$ Thus a fixed $c$ added to $2\log p$ does not yield a nonzero limiting rejection probability.

For a nondegenerate limit, use $2\log p-\log\log p+x$. At nominal level $\alpha$, set the limiting non-rejection probability equal to $1-\alpha$: $$ \exp\left(-\frac{1}{\sqrt{\pi}}e^{-x/2}\right)=1-\alpha. $$ Solving gives $x=-2\log\{\sqrt{\pi}[-\log(1-\alpha)]\}$. This is an asymptotic threshold under the theorem's independent standard-normal assumptions, not a universal maximum-test cutoff.

The direct tail argument above is important. Convergence of a centered statistic's distribution at each fixed argument does not, by itself, establish a probability limit at an argument that moves with $p$. The union bound supplies the needed control for the threshold $2\log p+c$ directly.

Dependence Changes the Reference Law

An independent-coordinate extreme-value calibration uses the joint distribution, not just the marginal distribution of each coordinate. If every coordinate is marginally standard normal but the coordinates are dependent, the probability of at least one extreme need not equal the independent-coordinate probability. One cannot diagnose this issue by checking only that each coordinate has mean zero and variance one.

Worked Example: Perfect Dependence Reduces the Effective Maximum

Let $Z\sim N(0,1)$ and set $Z_1=\cdots=Z_p=Z$. Each coordinate is standard normal, but $M_p=\max_j|Z_j|=|Z|$ for every $p$. At the independent-coordinate threshold $u_p(x)=\sqrt{2\log p-\log\log p+x}$, the rejection probability is $$ P(M_p>u_p(x))=P(|Z|>u_p(x))\longrightarrow0, $$ because $u_p(x)\to\infty$. Under independence, the theorem instead gives a positive limiting rejection probability $1-\exp\{-\pi^{-1/2}e^{-x/2}\}$. The same marginal distributions have led to different maximum calibrations because the joint dependence differs.

This example is deliberately extreme, but the diagnostic is general: a dependence assumption used to derive a reference law must be checked or replaced by a method that accounts for the actual dependence. Dependence corrections for maximum tests address this issue under specified dependence structures. The perfect-dependence example does not imply that all dependence makes a test conservative, or that one can simply substitute an informal “effective number of coordinates.” The direction and size of the calibration error depend on the joint law and the statistic.

Permutation Validity Requires Invariance

A permutation distribution is exact only when the allowed permutations leave the null distribution unchanged, conditional on the information held fixed by the procedure. Equality of means alone is generally not enough. In a two-sample problem, freely permuting group labels requires exchangeability of the pooled observations under the null; unequal distributions can violate that condition even when the population means agree.

Worked Example: Equal Means Do Not Justify Label Permutation

Suppose the first group consists of independent $N(0,1)$ observations and the second of independent $N(0,9)$ observations. Both population means are zero, so the null hypothesis of equal means holds. But an observation's distribution depends on its group: for example, the probability that its absolute value exceeds $3$ is different in the two groups.

Consequently, the joint distribution of the labelled sample is not invariant to swapping a first-group observation with a second-group observation. A label-permutation reference distribution treats such swaps as equally plausible under the null, which is not justified by this model. A test may have a useful approximation under additional conditions, but its exact randomization validity does not follow from equal means. The remedy is to use a calibration justified for the variance and sampling structure, or to establish the extra assumptions that make the chosen permutation valid.

The same logic applies to permutations of coordinates, signs, or blocks: state exactly what is being permuted and identify the null invariance that makes those rearrangements valid. For example, shuffling coordinate labels assumes a suitable exchangeability across coordinates; it is not justified merely because all coordinate means are zero. Likewise, sign-flipping observations in a one-sample test requires a symmetry or randomization argument appropriate to the data, not just a zero mean.

A Practical Calibration Audit

When a high-dimensional mean test gives an unexpectedly large or small rejection rate, separate the possible failures rather than changing the threshold by trial and error. A concise audit asks four questions.

Check the scale.
Write the statistic in terms of the true standard deviation, then compare it with the denominator actually used. Verify whether each estimated scale is on a standard-deviation or variance scale.

Check the centering.
For a maximum, identify the exact null limit and its centering and scaling. Do not infer a probability at a moving threshold from convergence at fixed arguments alone.

Check dependence.
Determine whether the reference law assumes independent coordinates, a particular correlation structure, or another dependence condition. Marginal standardization does not establish joint independence.

Check permutation invariance.
Specify the objects permuted and show that the null law is invariant under those permutations, either exactly or under a stated approximation.

These checks matter in simulation as well as in analysis of observed data. A simulated size close to its target under one covariance structure or one scale convention does not validate a different structure or convention. Conversely, a size distortion can be informative: if it changes sharply when the variance scale or dependence structure is corrected, that points to calibration rather than to a new signal in the data.

Key takeaway: A valid mean-test calibration is a joint claim about the statistic, its scale, its null dependence, and the reference procedure. The independent Gaussian maximum has a specific second-order centering, and neither a familiar threshold nor a permutation scheme remains valid after its assumptions are changed.

Check Your Understanding

Use the assumptions in each question explicitly.

If every coordinate statistic is multiplied by $3$, what happens to its variance and to the leading order of its maximum?
Why does $2\log p+c$, for fixed $c$, give a vanishing exceedance probability for the squared independent standard-normal maximum?
In the perfect-dependence example, why does the independent-coordinate extreme-value limit fail even though all marginal distributions are standard normal?
What invariance condition is needed to justify permuting two group labels, and why does equality of means alone not supply it?
Suppose a relative scale error is $\delta_p$ in a squared maximum of leading order $\log p$. Why can a scale error tending to zero still affect the centered calibration if it vanishes too slowly?