Why Compare Tests Under the Same Calibration?
A statistic can look large because it is measured on a large numerical scale, not because it supplies stronger evidence. A meaningful comparison therefore fixes the null rejection probability first, then asks how often each test rejects under a specified alternative. That probability is the test’s power. In high dimensions, the answer depends on the shape of the mean shift: a few large coordinates, many small ones, or a direction aligned with the covariance structure can favor different statistics.
Use the one-sample Gaussian model \(X_1,\ldots,X_n\) independent with \(X_i\sim N_p(\mu,\Sigma)\), where \(\Sigma\) is known and positive definite. The null is \(H_0:\mu=0\). Write \(Z=\sqrt n\,\bar X\), so \(Z\sim N_p(\eta,\Sigma)\) with \(\eta=\sqrt n\,\mu\). In the first two examples, \(\Sigma=I_p\); in the third, coordinates are correlated. All tests are calibrated to have null rejection probability \(\alpha=0.05\).
Three Statistics and Their Calibrations
For identity covariance, the quadratic statistic is \(Q=Z^\mathsf{T}Z\), and under the null it has a central chi-square distribution with \(p\) degrees of freedom. Under the alternative it has a noncentral chi-square distribution with \(p\) degrees of freedom and noncentrality \(\lambda=\eta^\mathsf{T}\eta\). The maximum statistic is \(M=\max_j|Z_j|\). Under the identity-covariance null its coordinates are independent, so \(P_0(M\le c)=[2\Phi(c)-1]^p\). Thus its level-\(\alpha\) cutoff satisfies \([2\Phi(c_\alpha)-1]^p=1-\alpha\).
A ridge statistic uses a positive regularization parameter \(\gamma\): \(R_\gamma=Z^\mathsf{T}(\Sigma+\gamma I_p)^{-1}Z\). Unlike the ordinary Hotelling quadratic form, this does not fully whiten directions with small covariance eigenvalues. Its null distribution is generally a weighted sum of chi-square variables, so its cutoff must be calibrated from that distribution or by a valid null resampling procedure. The cutoff cannot simply be borrowed from a chi-square distribution without checking the weights.
Worked Comparisons With Independent Coordinates
For \(p=2\), the 5% quadratic cutoff is the 95th percentile of \(\chi^2_2\), namely \(q=-2\log(0.05)=5.9915\). The maximum cutoff solves \([2\Phi(c)-1]^2=0.95\), giving \(\Phi(c)=(1+\sqrt{0.95})/2=0.98734\) and \(c\approx2.236\). Both critical values are fixed under the null; only the alternative mean changes in the comparisons below.
Worked Example: A Sparse Shift
Let \(\eta=(2.8,0)^\mathsf{T}\). The quadratic noncentrality is \(\lambda=2.8^2=7.84\), so the quadratic-test power is \(P\{\chi^2_2(7.84)>5.9915\}\). One directly checkable evaluation uses the Poisson-mixture identity \(P\{\chi^2_2(\lambda)>q\}=\sum_{k=0}^{\infty}e^{-\lambda/2}(\lambda/2)^k/k!\;P\{\chi^2_{2+2k}>q\}\).
Here \(\lambda/2=3.92\), and for \(t=q/2=2.9957\), the central tail in term \(k\) is \(e^{-t}\sum_{j=0}^k t^j/j!=0.05\sum_{j=0}^k t^j/j!\). For example, the first three central tails are \(0.0500\), \(0.1998\), and \(0.4241\). Weighting the successive tails by the Poisson probabilities with mean 3.92 gives a total of approximately \(0.708\). This is a reproducible series evaluation, not a comparison of uncalibrated statistic values.
For the maximum test, the probability that the shifted coordinate does not cross either boundary is \(q_s=\Phi(c-2.8)-\Phi(-c-2.8)\approx\Phi(-0.564)-\Phi(-5.036)\approx0.2865\). The unshifted coordinate stays within the boundary with probability \(2\Phi(c)-1=\sqrt{0.95}\approx0.9747\). Independence therefore gives \(P_\eta(M>c)=1-q_s\sqrt{0.95}\approx1-(0.2865)(0.9747)=0.721\). At this particular sparse shift, maximum-test power is about \(0.721\), slightly above the quadratic-test power of about \(0.708\).
Worked Example: A Dense Shift With the Same Euclidean Strength
Now let \(\eta=(2.8/\sqrt2,2.8/\sqrt2)^\mathsf{T}\). Its squared length is \(2(2.8/\sqrt2)^2=7.84\), exactly the same as in the sparse example. By the quadratic result above, its calibrated power is again approximately \(0.708\): under identity covariance, quadratic power depends on the mean only through its squared length.
Each coordinate has mean \(a=2.8/\sqrt2\approx1.980\). Its no-exceedance probability is \(q_d=\Phi(c-a)-\Phi(-c-a)\approx\Phi(0.256)-\Phi(-4.216)\approx0.601\). The two coordinates are independent, so the probability that neither exceeds the maximum cutoff is \(q_d^2\approx0.361\). The maximum-test power is therefore \(1-q_d^2\approx0.639\). For these equal-strength alternatives, the quadratic test has greater power for the dense shift (\(0.708\) versus \(0.639\)), while the maximum test has a small advantage for the sparse shift (\(0.721\) versus \(0.708\)). The difference is modest here; the example illustrates a direction of sensitivity, not a universal ranking.
Correlated Shifts and Ridge Weighting
When \(\Sigma\ne I_p\), coordinatewise independence and equal coordinate scales no longer hold. A quadratic statistic based on \(Z^\mathsf{T}\Sigma^{-1}Z\) fully whitens the covariance, whereas ridge regularization replaces \(\Sigma^{-1}\) by \((\Sigma+\gamma I_p)^{-1}\). The latter changes the relative emphasis on covariance eigen-directions. Calibration must account for that change.
Worked Example: A Shift Along a Correlated Direction
Take \(p=2\), \(\Sigma=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\), and an alternative \(\eta=(\sqrt n\,\delta,\sqrt n\,\delta)^\mathsf{T}\). The eigenvectors are the common direction \((1,1)^\mathsf{T}/\sqrt2\), with eigenvalue \(1.8\), and the contrast direction \((1,-1)^\mathsf{T}/\sqrt2\), with eigenvalue \(0.2\). The mean has common-direction coordinate \(\sqrt{2n}\delta\) and zero contrast coordinate.
For the fully whitened quadratic \(Q_\Sigma=Z^\mathsf{T}\Sigma^{-1}Z\), the null law is \(\chi^2_2\), while the alternative noncentrality is \((\sqrt{2n}\delta)^2/1.8=2n\delta^2/1.8\). For \(\gamma=0.2\), the ridge statistic has eigen-direction representation \(R_{0.2}=0.5\,Y_+^2+2.5\,Y_-^2\), where independently \(Y_+\sim N(\sqrt{2n}\delta,1.8)\) and \(Y_-\sim N(0,0.2)\). Under the null this is \(0.9\chi^2_1+0.5\chi^2_1\), with independent terms. Its critical value is the 95th percentile of that weighted sum, not the 95th percentile of \(\chi^2_2\).
The ridge test’s power is precisely \(P\{0.5Y_+^2+2.5Y_-^2>r_{0.05}\}\), using the alternative distributions just given and the null-calibrated weighted-sum cutoff \(r_{0.05}\). This probability can be evaluated by one-dimensional numerical integration after conditioning on \(Y_-\), or by a calibrated simulation. The formula makes clear what must be compared: the calibrated power, not the raw value of \(R_{0.2}\). A maximum test in the original coordinates likewise needs a critical region calibrated under the correlated null; the independent-coordinate cutoff from the previous examples is not valid here.
How to Choose and Compare
The comparison should begin with a scientifically defensible description of the alternative. A quadratic statistic accumulates squared evidence across coordinates, which can suit many moderate shifts. A maximum statistic responds to the most extreme coordinate and can suit a few strong shifts. A ridge statistic interpolates across covariance directions: directions with different eigenvalues receive different effective weights. No one of these facts alone establishes which test has greater power for a given problem. That requires specifying the alternative and comparing rejection probabilities after each test has been calibrated to the same size.
Specify the covariance assumptions and how the null distribution will be obtained.
Distinguish a few large effects, many modest effects, and shifts aligned with particular covariance eigen-directions.
Use its own null distribution or a valid common resampling scheme to achieve the target size.
Evaluate exact probabilities where available, or use simulation with uncertainty reported for each estimated power.
Common Calibration Pitfall
A frequent error is to compare the numerical sizes of \(Q\), \(M\), and \(R_\gamma\), or to use the same numerical cutoff for all three. Their scales and null distributions differ. For example, even with identity covariance, the quadratic statistic has a chi-square null distribution while the maximum statistic has a distribution determined by the maximum of absolute normal variables. A ridge statistic generally has yet another null distribution. The correction is to set each test’s rejection threshold from its own null law, then compare the resulting powers under the same specified alternatives.
A second caution is that “sparse favors maximum” and “dense favors quadratic” are useful design intuitions, not pointwise theorems that settle every comparison. Signal strength, dimension, correlation, and calibration all matter. When the alternative is uncertain, an adaptive combination may be considered, but its calibration must include the adaptation; selecting whichever statistic looks largest and then using its unadjusted cutoff inflates size.
Check Your Understanding
Use the model and calculations above to answer these questions.
- Why does the identity-covariance quadratic-test power agree for two mean vectors with the same squared Euclidean length?
- For independent standard normal coordinates, write the probability that a maximum test rejects under a general mean vector \(\eta\).
- Why is the ridge statistic in the correlated example not calibrated by the ordinary \(\chi^2_2\) cutoff?
- What feature of the alternative is the maximum statistic designed to emphasize?
- Explain why comparing observed statistic values without calibrating their null distributions does not compare evidence at a common significance level.