Higher Criticism for Unknown Sparsity

Why Scan the Small p-Values?

A maximum-coordinate test asks whether one coordinate is exceptionally large. A threshold-count test asks whether there are unusually many coordinates exceeding a chosen cutoff. The difficulty is that a useful cutoff depends on the unknown fraction of nonzero means: a very small fraction may produce only a few extreme p-values, while a larger fraction may produce a moderate excess of less-extreme p-values. Higher criticism (HC) addresses this by scanning a range of thresholds rather than selecting one in advance.

The scan is built from coordinatewise p-values. Its key quantity is not just the number below a threshold, but the excess over the null expectation measured in null standard-deviation units. This makes the statistic sensitive to departures spread across a range of rarity levels.

Definition from a Null Count

Suppose $p_1,\ldots,p_n$ are independent, continuous Uniform$(0,1)$ variables under a global null hypothesis. For a threshold $t$, let $N_n(t)$ be the number of p-values no greater than $t$. Under this null, $N_n(t)$ has a Binomial$(n,t)$ distribution, so its expectation is $nt$ and its variance is $nt(1-t)$.

Definition: The higher-criticism scan over the conventional range $[1/n,1/2]$ is $$ \operatorname{HC}_n = \sup_{1/n\le t\le 1/2} \frac{N_n(t)-nt}{\sqrt{nt(1-t)}} = \sup_{1/n\le t\le 1/2} \frac{\sqrt n\{N_n(t)/n-t\}}{\sqrt{t(1-t)}}. $$ The threshold is eligible exactly when it lies in $[1/n,1/2]$. There is no additional eligibility condition on the observed count $N_n(t)$.

The numerator measures the observed excess of small p-values; the denominator is the standard deviation of that count under the global null. A large positive score indicates an excess that is large relative to ordinary null fluctuation. The lower cutoff avoids scanning thresholds below the scale of the smallest typical null p-values, while the upper cutoff focuses attention on small p-values.

Write the ordered values as $p_{(1)}\le\cdots\le p_{(n)}$. With no ties, on an interval where $N_n(t)=k$, the score is $(k-nt)/\sqrt{nt(1-t)}$, which decreases as $t$ increases for $t<1/2$. Thus the scan can be evaluated at its left endpoints: at $1/n$, and at observed ordered p-values in the scan range. In particular, the familiar order-statistic scores are

\frac{\sqrt n\{i/n-p_{(i)}\}}{\sqrt{p_{(i)}(1-p_{(i)})}}, \qquad 1\le i\le n,\quad 1/n\le p_{(i)}\le 1/2.

If some ordered p-values fall below $1/n$, the scan still includes $t=1/n$; it does not include those smaller observed values as thresholds. This endpoint detail is one reason to define the scan in terms of $t$ before using its order-statistic representation.

Worked Example: Computing an Ordered-Value Score

Suppose $n=10$, and the first two ordered p-values are $p_{(1)}=0.01$ and $p_{(2)}=0.03$. Both are in the scan range. At $t=p_{(2)}$, the count is $N_{10}(t)=2$, so the score is

\frac{2-10(0.03)}{\sqrt{10(0.03)(0.97)}} = \frac{1.7}{\sqrt{0.291}} \approx \frac{1.7}{0.5394} \approx 3.15.

For comparison, the score at $t=p_{(1)}=0.01$ is

\frac{1-10(0.01)}{\sqrt{10(0.01)(0.99)}} = \frac{0.9}{\sqrt{0.099}} \approx \frac{0.9}{0.3146} \approx 2.86.

The second threshold has the larger standardized excess. The scan takes the largest score over all eligible thresholds, not necessarily the score for the smallest p-value.

Valid P-Values and the Sparse Gaussian Model

The null distribution used to form p-values must be correctly specified. For independent $Z_j\sim N(0,1)$ under the global null, the one-sided p-values $p_j=1-\Phi(Z_j)$ are independent Uniform$(0,1)$: for $0<u<1$,

P(p_j\le u) = P\{Z_j\ge \Phi^{-1}(1-u)\} = u.

Continuity alone is not enough if the p-value transformation uses the wrong null distribution. For example, applying $1-\Phi(Z_j)$ to a variable that is not standard normal under the null need not produce a uniform p-value.

To describe power, consider the following independent-coordinate model, with a known positive signal direction:

Model: The coordinates $Z_1,\ldots,Z_n$ are independent. Under the null, $Z_j\sim N(0,1)$. Under an alternative, $m_n$ coordinates have distribution $N(\mu_n,1)$ and the remaining $n-m_n$ coordinates have distribution $N(0,1)$, where $$ m_n\sim n^{1-\beta},\qquad \mu_n=\sqrt{2r\log n},\qquad 1/2<\beta<1,\quad r>0. $$ Use $p_j=1-\Phi(Z_j)$ for every coordinate.

The proportion of signals is approximately $n^{-\beta}$, which tends to zero. The parameter $\beta$ describes sparsity, while $r$ describes signal strength. In practice the signal fraction is unknown; the HC scan does not require it as an input.

Worked Example: A Threshold Under a Positive Mean Shift

For a standard-normal-null p-value threshold $t=0.05$, the corresponding $Z$-threshold is approximately $1.645$, since $1-\Phi(1.645)\approx0.05$. If a signal coordinate has $Z\sim N(2,1)$, then its probability of producing a p-value below $0.05$ is

P(p_j\le0.05\mid \text{signal}) = P(Z_j\ge1.645) = P\{N(0,1)\ge -0.355\} = \Phi(0.355) \approx0.639.

Thus a signal coordinate crosses this threshold with probability about $0.639$, compared with probability $0.05$ for a null coordinate. The HC statistic asks whether the aggregate excess across coordinates is large, and searches for the threshold where that excess is most pronounced.

How a Threshold Produces Power

For a threshold $t$, let $q_n(t)$ be the probability that a signal p-value is no greater than $t$. In the model above, the expected count under the alternative is

E[N_n(t)] = (n-m_n)t+m_nq_n(t) = nt+m_n\{q_n(t)-t\}.

The mean score at this threshold is therefore

\frac{m_n\{q_n(t)-t\}}{\sqrt{nt(1-t)}}.

This expression shows the adaptation problem directly: the most informative $t$ depends on both the signal fraction and the strength of the signals. HC scans the thresholds instead of committing to one.

Theorem: Fix $1/2<\beta<1$, $r>0$, and $s\in(0,1)$. In the independent Gaussian model, at $t_n=n^{-s}$, define $$ g(s,r)=\{\max(\sqrt{s}-\sqrt r,0)\}^{2}, \qquad E(s;\beta,r)=\frac12-\beta+\frac{s}{2}-g(s,r). $$ If $E(s;\beta,r)>0$, then the HC score at $t_n$ diverges to $+\infty$ in probability. Consequently, $\operatorname{HC}_n\xrightarrow{p}+\infty$. The condition $t_n\in[1/n,1/2]$ holds for all sufficiently large $n$.

Proof. First, $n^{-s}\le1/2$ eventually, and $n^{-s}\ge n^{-1}$ because $s<1$. Thus the threshold belongs to the scan. Let $z_n=\Phi^{-1}(1-t_n)$. The normal tail relation $\log\{1-\Phi(x)\}=-x^2/2+o(x^2)$ as $x\to\infty$ gives

\frac{z_n}{\sqrt{2\log n}}\longrightarrow\sqrt{s}.

For a signal coordinate, $q_n(t_n)=1-\Phi(z_n-\mu_n)$. If $s>r$, then $z_n-\mu_n$ is positive and its ratio to $\sqrt{2\log n}$ tends to $\sqrt{s}-\sqrt r$. Applying the same normal-tail relation gives $\log q_n(t_n)/\log n\to-(\sqrt{s}-\sqrt r)^2$. If $s\le r$, $q_n(t_n)$ has logarithmic exponent zero: when $s<r$ it tends to one, and when $s=r$ its logarithm divided by $\log n$ tends to zero. Together these cases give

q_n(t_n)=n^{-g(s,r)+o(1)}.

Because $g(s,r)<s$ for every $r>0$, $q_n(t_n)/t_n\to\infty$, so $q_n(t_n)-t_n\sim q_n(t_n)$. Also $m_n=n^{1-\beta+o(1)}$. Since $1-t_n\to1$, the expected HC score at $t_n$ is

\frac{m_n\{q_n(t_n)-t_n\}}{\sqrt{nt_n(1-t_n)}} = n^{E(s;\beta,r)+o(1)}.

It remains to check random fluctuation. The count is a sum of independent Bernoulli variables, so its variance is at most its expectation, which is at most $nt_n+m_nq_n(t_n)$. Dividing this variance by the square of the mean excess $D_n^2=\{m_n(q_n(t_n)-t_n)\}^2$, the contribution from $nt_n$ tends to zero because $D_n/\sqrt{nt_n}\to\infty$. The contribution from $m_nq_n(t_n)$ tends to zero because $m_nq_n(t_n)\to\infty$: indeed, $E(s;\beta,r)>0$ implies $1-\beta-g(s,r)>0$. Chebyshev's inequality now shows that the count's deviation from its expectation is negligible relative to $D_n$. The standardized score therefore diverges in probability. Since the scan includes $t_n$, its supremum diverges as well. $\square$

Worked Example: A Sparse Alternative Above the Boundary

Take $\beta=0.60$ and $r=0.15$, and choose $s=4r=0.60$. Since $s>r$,

g(0.60,0.15) = (\sqrt{0.60}-\sqrt{0.15})^2 \approx(0.7746-0.3873)^2 =0.15.

The score exponent is

E(0.60;0.60,0.15) = \frac12-0.60+\frac{0.60}{2}-0.15 =0.05>0.

The selected threshold is $t_n=n^{-0.60}$, which is in the scan range for large $n$. The theorem shows that the HC score at this threshold grows as $n^{0.05+o(1)}$, so the full scan diverges in probability. The threshold was identified from the model for this calculation; the HC statistic itself scans without being given $\beta$ or $r$.

Optimizing the Threshold and the Sparse Boundary

For $1/2<\beta<1$, the supremum of $E(s;\beta,r)$ over $0<s<1$ determines the largest attainable exponent; when it occurs at the excluded endpoint $s=1$, it is approached as $s\to1$ (and the scan itself includes $t=1/n$). When $s\le r$, $g(s,r)=0$, so $E=1/2-\beta+s/2$, which increases with $s$. When $s>r$,

E(s;\beta,r) = \frac12-\beta-\frac{s}{2}-r+2\sqrt{sr}.

Its derivative with respect to $s$ is $-1/2+\sqrt{r/s}$, so its unconstrained maximum is at $s=4r$. If $r<1/4$, this threshold is in the scan's asymptotic range and the maximum exponent is $1/2-\beta+r$. It is positive when $r>\beta-1/2$. If $r=1/4$, the same value is the supremum, approached as $s\to1$. If $r>1/4$, the exponent increases up to the upper endpoint $s=1$; its limiting value there is positive precisely when $2\sqrt r-r>\beta$, which, for $r<1$, is equivalent to $r>(1-\sqrt{1-\beta})^2$. For $r\ge1$, choosing $s<1$ sufficiently close to one gives a positive exponent as well.

Takeaway: The resulting sparse detection boundary is $$ \rho^*(\beta)= \begin{cases} \beta-\tfrac12,&1/2<\beta\le3/4,\\ (1-\sqrt{1-\beta})^2,&3/4<\beta<1. \end{cases} $$ For fixed $\beta$ and $r>\rho^*(\beta)$, some threshold has a positive power exponent, and the HC scan is consistent in the sense that its statistic diverges in probability. This is an above-boundary power statement; the exponent argument alone does not establish a claim of power below the boundary or settle behavior exactly on it.

The change in formula at $\beta=3/4$ comes from whether the maximizing threshold $s=4r$ remains below one. Less sparse alternatives can be detected through a threshold with many moderately small p-values. For more extreme sparsity, the useful threshold approaches the smallest scale included in the scan.

Calibration, Scope, and a Common Mistake

Under the independent Uniform$(0,1)$ global null, the count at any fixed threshold has the binomial mean and variance used to construct HC. The maximum over thresholds is not itself a single standard normal variable: the scores at different thresholds are dependent. A test therefore needs a critical value calibrated for the scan, such as a null quantile or a suitable empirical-process calibration. The statistic's definition and its power argument do not by themselves prescribe one universal finite-sample critical value.

The power result also relies on independent coordinates and the stated Gaussian model. Dependence or misspecified marginal null distributions can alter the behavior of the p-value counts, and thus require suitable calibration or a different analysis. If signal directions are unknown, the one-sided p-values used here are not automatically appropriate; p-values must match the testing direction and null model.

Mistake: “A threshold is eligible only if its count is at most $n/2$.” That condition is not part of the definition. Eligibility requires only $1/n\le t\le1/2$; $N_n(t)$ may exceed $n/2$.
Mistake: “Continuous null marginals make $1-\Phi(Z_j)$ uniform.” That transformation is uniform when $Z_j$ has the standard normal null distribution. For another continuous null distribution, use its correctly specified null distribution function to form the p-value.
Mistake: “HC is just the smallest p-value.” HC standardizes an excess count and maximizes over thresholds. A collection of moderately small p-values can produce a larger score than a single extreme value.

Higher criticism is useful when sparse signals are plausible but their fraction is not known: its threshold scan can respond at the scale where the excess signal count is most informative. Its adaptivity is not a substitute for valid marginal p-values or appropriate null calibration; those are essential parts of the procedure.

Check Your Understanding

Try these questions using the definition and model above.

Under independent standard normal null coordinates, why is $1-\Phi(Z_j)$ Uniform$(0,1)$? What assumption would fail if the null distribution of $Z_j$ were instead $N(1,1)$?
For $n=100$, is $t=0.02$ in the scan range? If $N_n(t)=70$, does that make the threshold ineligible?
In the sparse Gaussian model, what do $\beta$ and $r$ represent, and why does a threshold $t_n=n^{-s}$ belong to the scan for $0<s<1$ once $n$ is large enough?
For $\beta=0.60$ and $r=0.15$, verify the exponent at $s=0.60$. What does its positive sign imply about the HC statistic?
Why is the HC scan not calibrated by treating its maximum as a single fixed-threshold normal score?