Adaptive Combination of Dense and Sparse Tests

Why Combine Mean Tests?

A high-dimensional mean shift can be spread across many coordinates or concentrated in only a few. A quadratic statistic accumulates evidence across coordinates, while a maximum statistic is sensitive to an unusually large coordinate. Higher criticism searches across thresholds for an excess of small coordinatewise p-values, making it useful when the number and size of the nonzero means are unknown. No one of these summaries is uniformly best across all signal patterns.

An adaptive test can reject when any member of this collection rejects. Its key design problem is calibration: testing each component at the full nominal level and then taking their union inflates the probability of a false rejection. A simple solution is to allocate the allowed error across the components. This gives a valid test without requiring the statistics to be independent.

Model and Component Statistics

Use the Gaussian sequence model \(Z_j\sim N(\theta_j,1)\), independently for \(j=1,\ldots,d\). The null hypothesis is \(H_0:\theta_1=\cdots=\theta_d=0\). The vector \(Z\) can represent standardized coordinatewise estimates; the model makes the null distributions and the combination rule explicit. The focus here is how to combine tests, not how to construct these estimates.

Definition: The quadratic statistic, maximum statistic, and higher-criticism statistic are, respectively,

Q_d=\frac{\sum_{j=1}^{d}(Z_j^2-1)}{\sqrt{2d}},\qquad M_d=\max_{1\leq j\leq d}|Z_j|,\qquad HC_d=\max_{u\in\mathcal U_d} \left[\frac{N_d(u)-du}{\sqrt{du(1-u)}}\right]_+ .

Here \([x]_+=\max(x,0)\), \(p_j=2\{1-\Phi(|Z_j|)\}\) is the two-sided null p-value, \(N_d(u)=\sum_{j=1}^d I(p_j\leq u)\), and \(\mathcal U_d\) is a prespecified finite grid of thresholds in \((0,1)\). Under \(H_0\), the \(p_j\) are independent Uniform\((0,1)\) variables. The grid must be fixed before observing the data; in a sparse-signal application it commonly includes small thresholds.

Under \(H_0\), \(\sum_j Z_j^2\) has a chi-square distribution with \(d\) degrees of freedom, so \(Q_d\) has a centered and scaled chi-square null distribution. Also, \(P_0(M_d\leq c)=\{2\Phi(c)-1\}^d\) for \(c\geq0\). For each fixed \(u\), \(N_d(u)\) has a Binomial\((d,u)\) distribution under the null. The maximum over the grid makes the null distribution of \(HC_d\) depend on the grid and on the dependence among its threshold counts. Its critical value must calibrate the maximum itself, not just one threshold.

Worked Example: A Dense, Moderate Shift

Take \(d=100\) and suppose the observed vector has \(Z_j=1.2\) for every coordinate. The quadratic statistic is \(Q_{100}=100(1.2^2-1)/\sqrt{200}=44/14.142\approx3.11\). The maximum is \(M_{100}=1.2\). Each two-sided p-value is approximately \(2\{1-\Phi(1.2)\}=0.230\). If the higher-criticism grid uses thresholds no larger than \(0.10\), then \(N_{100}(u)=0\) at every grid point and \(HC_{100}=0\).

For illustration, suppose the component critical values are \(2.70\) for \(Q_{100}\), \(3.80\) for \(M_{100}\), and \(4.00\) for \(HC_{100}\). Only the quadratic statistic exceeds its cutoff, so the combined test rejects. The example shows how moderate evidence distributed across many coordinates can accumulate in the quadratic statistic even when no coordinate is extreme.

Worked Example: One Extreme Coordinate

Let \(d=100\), with \(Z_1=4\) and \(Z_2=\cdots=Z_{100}=0\). Then \(Q_{100}=(16-100)/\sqrt{200}\approx-5.94\), while \(M_{100}=4\). The first coordinate has two-sided p-value approximately \(0.000063\); the other 99 p-values equal 1. If the HC grid starts at \(u=0.01\), then at \(u=0.01\), \(N_{100}(u)=1\), and the standardized excess is \(10(0.01-0.01)/\sqrt{0.01(0.99)}=0\).

At larger grid thresholds, the standardized excess is negative and is truncated to zero; at grid thresholds below \(0.000063\), the count is zero and the positive-part statistic is also zero. Thus \(HC_{100}=0\) for this grid. With the illustrative cutoffs above, the maximum statistic alone rejects. A maximum test can therefore contribute power where the quadratic statistic does not.

Worked Example: A Threshold Detects Several Signals

Again let \(d=100\), but now suppose ten coordinates equal \(2.5\) and the remaining 90 equal zero. The quadratic statistic is \(Q_{100}=\{10(2.5^2)-100\}/\sqrt{200}=-37.5/14.142\approx-2.65\), and \(M_{100}=2.5\). Each of the ten nonzero coordinates has two-sided p-value approximately \(0.01242\). If \(u=0.02\) is in the HC grid, then \(N_{100}(0.02)=10\), giving

\frac{10-100(0.02)}{\sqrt{100(0.02)(0.98)}} =\frac{8}{1.4}\approx5.71.

The higher-criticism statistic is at least \(5.71\), so it exceeds the illustrative cutoff \(4.00\). Neither \(Q_{100}\) nor \(M_{100}\) exceeds its illustrative cutoff. Here an intermediate threshold reveals an excess of moderately small p-values that is not captured by the other two statistics.

Calibrating the Combined Test

Let \(T_Q=Q_d\), \(T_M=M_d\), and \(T_H=HC_d\). For a chosen overall level \(\alpha\), select nonrandomized critical values \(c_Q,c_M,c_H\) so that under the null \(P_0(T_k>c_k)\leq\alpha/3\) for each \(k\in\{Q,M,H\}\). The critical values may be obtained from the corresponding null distributions. For the higher-criticism statistic, this means calibrating the maximum over the prespecified grid. Using conservative upper-tail quantiles is valid even when the null distribution has atoms.

Definition: The Bonferroni combined test rejects \(H_0\) if at least one of \(Q_d>c_Q\), \(M_d>c_M\), or \(HC_d>c_H\) holds, where each component has null rejection probability at most \(\alpha/3\).

The following result gives the size guarantee. It requires no independence among the three statistics.

Theorem: If each component rejection event has null probability at most \(\alpha/3\), then the Bonferroni combined test has size at most \(\alpha\).

Proof. Write \(A_Q=\{Q_d>c_Q\}\), \(A_M=\{M_d>c_M\}\), and \(A_H=\{HC_d>c_H\}\). The combined rejection event is their union. By the union bound,

P_0(A_Q\cup A_M\cup A_H) \leq P_0(A_Q)+P_0(A_M)+P_0(A_H) \leq \frac{\alpha}{3}+\frac{\alpha}{3}+\frac{\alpha}{3} =\alpha.

No assumption about the joint distribution of the statistics was used. In particular, their dependence does not invalidate the size control. \(\square\)

The illustrative cutoffs in the examples show how to apply a combined rule once the cutoffs are calibrated; they are not universal values. Actual critical values depend on \(d\), the chosen HC grid, and the null calibration method. More generally, one can allocate levels \(\alpha_Q,\alpha_M,\alpha_H\) with \(\alpha_Q+\alpha_M+\alpha_H\leq\alpha\), which can be useful if some component is given greater priority.

Power Across Signal Densities

For a mean vector \(\theta\), let \(\beta_k(\theta)=P_\theta(T_k>c_k)\) be the power of component \(k\), using its allocated critical value. The combined rejection event contains each individual rejection event. Consequently,

P_\theta(\text{combined test rejects}) =P_\theta(A_Q\cup A_M\cup A_H) \geq \max\{\beta_Q(\theta),\beta_M(\theta),\beta_H(\theta)\}.

This inequality is exact for every alternative; it does not require an asymptotic approximation. It says that the combined test is at least as powerful as each component at its allocated level. It does not say that it is as powerful as each component tested separately at the full level \(\alpha\).

The components are suited to different patterns. When many coordinates carry small shifts, the accumulated squared signal can make \(Q_d\) effective. When only a very small number of coordinates are unusually large, \(M_d\) can respond to the most extreme one. When the number of affected coordinates is unknown and signals yield a detectable excess of small p-values at some threshold, \(HC_d\) can find that threshold. These are broad roles, not guarantees that one statistic has power against every alternative in its named regime.

Power takeaway: If, along a sequence of alternatives, any one component test has power tending to one at its allocated level, then the combined test also has power tending to one. This follows directly from the power inequality above.

For example, if \(P_{\theta}(A_H)\to1\), then \(P_{\theta}(\text{combined test rejects})\geq P_{\theta}(A_H)\to1\). The same argument applies to the quadratic and maximum components. In asymptotic comparisons, splitting a fixed \(\alpha\) among a fixed number of components often changes critical values by constants rather than changing a detection rate. That observation is not a universal theorem about every model: the component statistics, their calibration, and the alternative sequence all matter.

Why the Calibration Choice Matters

A combined test trades some component-level size for insurance against uncertainty about signal density. If the signal pattern is known in advance, a test tailored to that pattern may have greater power at the same overall level. The benefit of combination is protection against choosing the wrong summary, not a guarantee of optimality.

Using the full level three times. Testing all three components at level \(\alpha\) and rejecting if any rejects can produce a null rejection probability larger than \(\alpha\). The union bound controls the total by allocating levels whose sum is at most \(\alpha\).
Calibrating one HC threshold instead of the HC maximum. A threshold count at a single \(u\) is not the same statistic as the maximum over \(\mathcal U_d\). The cutoff must account for the search across the entire prespecified grid.
Changing the grid after seeing the data. Choosing thresholds in response to observed p-values adds another search. Unless that search is included in the calibration, the stated size guarantee need not hold.

A transparent analysis reports the component statistics, their allocated levels, the grid used for higher criticism, and the rule used to obtain each critical value. These details make the overall error guarantee interpretable and allow the role of each component to be assessed.

Check Your Understanding

Use the definitions and size argument to answer the following questions.

Why does calibrating three component tests at level \(\alpha/3\) control the size of their union even when their statistics are dependent?
In the Gaussian sequence model under the null, what is the distribution of \(N_d(u)\) for one fixed threshold \(u\)?
Which component is most directly responsive to one unusually large absolute coordinate, and why?
If the HC threshold grid is changed after examining the data, what part of the calibration may no longer be valid?
Suppose the maximum test has power tending to one along a sequence of alternatives at its allocated level. What can be concluded about the power of the combined test?