Estimating The Null Proportion For Storey’s Method

Why Estimate the True-Null Proportion?

In a family of genomic tests, some null hypotheses may be false, so the proportion of true nulls can be less than one. The Benjamini–Hochberg procedure uses the full family size when setting its rejection thresholds. Storey’s method instead begins by estimating how many hypotheses are true nulls. Its basic idea is that p-values near the upper end of the scale are more likely to come from true null hypotheses than from alternatives.

The estimate is based on the fraction of observed p-values above a chosen threshold, called $\lambda$. It is a data-based estimate, not a count of known true nulls: the observed p-values do not label which hypotheses are null. The threshold determines how the tail is used, and the estimate can be affected by both random variation and alternative p-values in that tail.

Definition and Interpretation

Suppose a family contains $m$ tests with p-values $p_1,\ldots,p_m$. Let $m_0$ be the number of true null hypotheses and define the true-null proportion as $\pi_0=m_0/m$. For a selected tuning parameter $0\leq\lambda<1$, count p-values strictly greater than $\lambda$.

Definition: Storey’s tail estimator of the true-null proportion at $\lambda$ is $$ \widehat{\pi}_0(\lambda)=\frac{\#\{i:p_i>\lambda\}}{m(1-\lambda)}. $$ A commonly used bounded version is $\min\{1,\widehat{\pi}_0(\lambda)\}$, since a proportion cannot exceed one. The unbounded ratio is useful for understanding the estimator’s sampling behavior; report clearly whether it has been capped.

The denominator $m(1-\lambda)$ is the number of p-values that would be expected above $\lambda$ if all $m$ tests were true nulls with continuous uniform p-values. The numerator is the number actually observed there. Thus the ratio compares the observed upper-tail count with its all-null reference count. It estimates a proportion; it does not identify which tests are true nulls.

Worked Example: Calculating the Tail Estimate

Suppose a hypothetical genomic analysis has $m=12$ tests and reports the following p-values: $0.04, 0.09, 0.16, 0.22, 0.31, 0.43, 0.55, 0.62, 0.71, 0.83, 0.91, 0.97$. Use $\lambda=0.60$.

The p-values strictly greater than $0.60$ are $0.62, 0.71, 0.83, 0.91,$ and $0.97$, so the tail count is 5. The all-null reference count is $m(1-\lambda)=12(0.40)=4.8$. Therefore,

\widehat{\pi}_0(0.60)=\frac{5}{12(1-0.60)}=\frac{5}{4.8}=1.042.

The raw estimate is about 1.04. Since a true-null proportion cannot exceed one, the bounded estimate is 1. This does not establish that all 12 hypotheses are true nulls. It means the observed tail count is slightly larger than the all-null reference count, which can happen through sampling variation or departures from the estimator’s assumptions.

What the Estimator Targets

The estimator’s behavior is clearest by considering its expectation. Write $I_i(\lambda)=1$ when $p_i>\lambda$, and $I_i(\lambda)=0$ otherwise. The tail count is $\sum_{i=1}^m I_i(\lambda)$. Suppose first that each true-null p-value is exactly uniform on $[0,1]$. Then a true-null p-value exceeds $\lambda$ with probability $1-\lambda$. For false nulls, let $\beta_i(\lambda)=P(p_i>\lambda\mid H_i\text{ is false})$, the probability that test $i$'s p-value falls in the upper tail under its alternative.

Taking the expectation of the raw estimator and using linearity of expectation gives the following result. This expectation calculation does not require independence among tests.

\begin{aligned} E\{\widehat{\pi}_0(\lambda)\} &=\frac{1}{m(1-\lambda)}\sum_{i=1}^{m}P(p_i>\lambda)\\ &=\frac{m_0(1-\lambda)+\sum_{i:H_i\text{ false}}\beta_i(\lambda)} {m(1-\lambda)}\\ &=\pi_0+\frac{1}{m(1-\lambda)} \sum_{i:H_i\text{ false}}\beta_i(\lambda). \end{aligned}

The second term is nonnegative. If some alternative p-values fall above $\lambda$, they contribute to the observed tail count, and the raw estimator’s expectation exceeds $\pi_0$. This explains why the estimate is often conservative in the sense of being upwardly biased under exact null calibration: alternatives can be counted as if they were nulls. The amount of this bias depends on the alternatives and on $\lambda$; it is not automatically zero just because $\lambda$ is large.

Key result: With exactly uniform true-null p-values, the raw estimator’s expectation is $\pi_0$ plus a nonnegative contribution from false-null p-values in the upper tail. It estimates $\pi_0$ well only when that contribution is small and the observed tail count is not too variable.

If null p-values are valid but conservative rather than exactly uniform, then $P(p_i\leq t\mid H_i\text{ true})\leq t$ for $0\leq t\leq1$. Consequently, $P(p_i>\lambda\mid H_i\text{ true})\geq1-\lambda$. In that case the true-null contribution to the expected tail count is at least the continuous-uniform reference count. The estimator may then be more upwardly biased. When p-values are discrete, the strict $>\lambda$ rule matters: the actual tail probability depends on the null distribution and on the threshold’s position relative to attainable p-values.

Worked Examples: Scale, Capping, and Threshold Sensitivity

Worked Example: A Bounded Estimate

In a hypothetical family of $m=10$ tests, suppose the p-values are $0.02, 0.13, 0.22, 0.51, 0.58, 0.64, 0.72, 0.81, 0.90,$ and $0.97$. At $\lambda=0.50$, seven p-values are strictly greater than the threshold.

The denominator is $10(1-0.50)=5$, so

\widehat{\pi}_0(0.50)=\frac{7}{5}=1.40.

The bounded estimate is $\min(1,1.40)=1$. The raw ratio exceeds one because the observed tail count exceeds the all-null reference count, not because a proportion of 1.40 is possible. Capping makes the reported proportion interpretable, but it does not remove the sampling variation or reveal the actual value of $\pi_0$.

Worked Example: Comparing Two Thresholds

Consider a hypothetical set of $m=100$ p-values. Suppose 54 are greater than $0.40$, and 17 are greater than $0.80$. These counts are nested: every p-value above $0.80$ is also above $0.40$.

At $\lambda=0.40$, the estimate is

\widehat{\pi}_0(0.40)=\frac{54}{100(0.60)} =\frac{54}{60}=0.900.

At $\lambda=0.80$, it is

\widehat{\pi}_0(0.80)=\frac{17}{100(0.20)} =\frac{17}{20}=0.850.

The estimates differ by $0.050$, even though they use the same family of tests. The higher threshold has a smaller denominator and uses fewer tail observations, so its estimate can be more sensitive to small changes in the count. Neither value can be declared correct from these calculations alone; the true $\pi_0$ is unknown, and each threshold makes a different bias–variability tradeoff.

Assumptions, Variability, and Diagnostics

The calculation itself only requires a p-value list and a specified $\lambda$. Its statistical interpretation, however, depends on the null p-value distribution. Exact uniformity under true nulls supplies the reference probability $1-\lambda$. If p-values are miscalibrated, or are conservative because they are discrete, the reference may not match their actual null tail probability. The estimator also describes a specified family of $m$ tests; changing which hypotheses are included changes the proportion being estimated.

Independence is not needed for the expectation calculation, but dependence can affect the variability of the tail count. For illustration, if the indicators $I_i(\lambda)$ are independent, then

\operatorname{Var}\{\widehat{\pi}_0(\lambda)\} =\frac{\sum_{i=1}^{m}q_i(1-q_i)} {m^2(1-\lambda)^2}, \qquad q_i=P(p_i>\lambda).

The denominator shows why thresholds close to one can be unstable: $1-\lambda$ is small, and relatively few observations are expected in the tail. Under dependence, covariances among the indicators also enter the variance, so the independent formula no longer applies. A similar estimate at nearby thresholds is a useful descriptive diagnostic, but apparent stability does not prove that the estimate is unbiased or that its assumptions hold.

Check the family definition, the p-value calibration under true nulls, the number of observations above the chosen threshold, and whether the estimate changes sharply across plausible thresholds. Sparse tails, estimates above one, or large changes across thresholds are warnings to interpret the estimate cautiously. A value above one is not a valid proportion; it indicates that the raw ratio has exceeded the all-null reference and should not be reported as an uncapped proportion.

How It Relates to Other FDR Choices

A common conservative choice in multiple testing is to use $\pi_0=1$, meaning that all tests are treated as if they could be true nulls. Storey’s estimator attempts to use evidence in the p-value tail to allow for $\pi_0<1$. If the estimate is below one, it can support less conservative downstream FDR calculations, but this tutorial concerns the estimate itself: an estimated $\pi_0$ is not a guarantee of error control, nor is it a classification of individual hypotheses.

The main alternative is not another tail threshold but a different modeling choice: retain $\pi_0=1$ rather than estimate it. That choice avoids uncertainty from estimating the null proportion but may be conservative when many tests are non-null. Storey’s estimate can be more adaptive, yet it adds sensitivity to null calibration, tail contamination, and threshold choice. In either approach, the hypotheses must belong to the intended family, and the p-values must be appropriate for the tests performed.

Common Mistakes and Limitations

Interpreting the estimate as a known count. The tail count is observed, but which tests are true nulls is not. The ratio estimates a proportion and does not identify null hypotheses.
Forgetting the denominator. The estimator is the tail fraction divided by $1-\lambda$, not simply the fraction of p-values above $\lambda$.
Treating an estimate above one as possible. The raw ratio can exceed one; use the bounded version when reporting a proportion and explain what the raw value indicates.
Assuming a high threshold removes all alternatives. Alternative p-values can still exceed $\lambda$, adding upward bias. A high threshold can also leave very few observations and increase variability.
Assuming independence is necessary for the expectation formula. Linearity of expectation does not require independence. Dependence does matter for variability and for broader guarantees of downstream procedures.
Ignoring discrete or conservative null p-values. The $1-\lambda$ reference relies on continuous uniform null p-values. With conservative null p-values, the true-null upper-tail probability can be larger.

Key takeaway: Storey’s estimator divides the observed fraction of p-values above $\lambda$ by the corresponding continuous-uniform null tail probability, $1-\lambda$. Its interpretation depends on null calibration, and its accuracy balances alternative contamination against tail-count variability.

Check Your Understanding

Work each problem before reading the solutions below.

A hypothetical family has 40 p-values, of which 12 are strictly greater than $\lambda=0.70$. Calculate the raw and bounded estimates.
Why can false-null p-values cause the raw estimate’s expectation to exceed $\pi_0$ under exact uniform null p-values?
Suppose every test is a true null and its p-value is continuous uniform. At $\lambda=0.90$, what is the expected tail count for $m=200$, and why can this threshold produce a variable estimate?
For a valid but conservative discrete null p-value, explain why using $1-\lambda$ as the null tail probability can make the estimate conservative in the upward direction.

Solutions. (1) The raw estimate is $12/[40(1-0.70)]=12/12=1.000$; the bounded estimate is also 1. (2) A false-null p-value can still exceed $\lambda$. Its positive upper-tail probability adds to the tail count, while the denominator’s reference count includes only the uniform-null contribution; the expectation therefore includes a nonnegative extra term. (3) The expected count is $200(1-0.90)=20$. The threshold leaves a relatively small tail, and random changes in that count are divided by $200(0.10)=20$, so the estimate can fluctuate substantially. (4) Superuniformity gives $P(p\leq\lambda\mid H_0)\leq\lambda$, hence $P(p>\lambda\mid H_0)\geq1-\lambda$. The null tail can therefore exceed the reference used in the denominator, increasing the expected estimate.