Storey Q-Values and the Null Fraction

Why Estimate the Null Fraction?

A genome-wide association analysis may test many variants, while only a small share are expected to be associated with the phenotype. Procedures that control the false discovery rate (FDR) can use this fact: if the number of true null hypotheses is below the total number of tests, a calculation that treats all tests as null can be conservative. Storey’s method estimates the share of true null hypotheses and uses it to produce q-values, which summarize estimated false-discovery rates across possible significance thresholds.

The estimate is not a count of which individual tests are truly null. It is a single estimated fraction used to calibrate a collection of tests. Its accuracy depends on the p-value distribution and on a tuning choice, so q-values should be read as estimated error summaries rather than as known probabilities that particular findings are false.

Definition: For a family of $m$ hypotheses, let $m_0$ be the number of true null hypotheses and let $\pi_0=m_0/m$ be the null fraction. For a tuning value $0\leq\lambda<1$, Storey’s tail estimator is $$ \hat{\pi}_0(\lambda)=\min\left\{1,\frac{\#\{i:p_i>\lambda\}}{m(1-\lambda)}\right\}. $$ It estimates the null fraction by comparing the observed share of p-values above $\lambda$ with the share expected there under a uniform null distribution.

The cap at 1 enforces the fact that a fraction cannot exceed 1. The tuning value $\lambda$ determines which part of the p-value distribution is treated as informative about the nulls. When null p-values are uniform, the probability that a null p-value exceeds $\lambda$ is $1-\lambda$. Non-null p-values are often concentrated nearer zero, so their contribution to the upper tail may be smaller, though it need not be zero.

Estimating the Null Fraction

The estimator involves a bias-variance tradeoff. A relatively low $\lambda$ leaves more p-values in the tail count, which can make the estimate less variable, but some non-null p-values may also exceed $\lambda$, pushing the estimate upward. A high $\lambda$ reduces that contamination if alternatives tend to have small p-values, but uses fewer observations and can make the estimate unstable. The cap at 1 prevents an estimate above 1; it does not make the estimate unbiased.

Worked Example: Calculating a Tail Estimate

Suppose a hypothetical association analysis reports $m=10$ p-values: 0.002, 0.015, 0.030, 0.080, 0.110, 0.220, 0.410, 0.580, 0.730, and 0.940. Use $\lambda=0.50$.

There are three p-values greater than 0.50: 0.580, 0.730, and 0.940. The denominator is $m(1-\lambda)=10(0.50)=5$. Therefore,

\hat{\pi}_0(0.50)=\min\left\{1,\frac{3}{5}\right\}=0.600.

The estimate suggests that 60.0% of the tested hypotheses are null. It does not establish that exactly six of the ten hypotheses are null; it is an estimate of the fraction in the testing family. Nor does it identify which hypotheses are null.

A simple expectation calculation explains the estimator’s motivation and an important limitation. Write $m_1=m-m_0$ for the number of non-null hypotheses. If null p-values are exactly uniform, the expected number of null p-values above $\lambda$ is $m_0(1-\lambda)$. The expected number of non-null p-values above $\lambda$ is nonnegative. Thus the uncapped tail estimator has expectation at least $\pi_0$:

E\left[\frac{\#\{i:p_i>\lambda\}}{m(1-\lambda)}\right] = \frac{m_0(1-\lambda)+\sum_{i\text{ non-null}}P(p_i>\lambda)} {m(1-\lambda)} \geq \frac{m_0}{m}=\pi_0.

This statement applies to the uncapped estimator. It does not generally apply to the capped estimator $\hat{\pi}_0(\lambda)$: capping can lower its expectation. For example, with one true null test and $\lambda=0.50$, a uniform p-value gives an uncapped estimate of 0 when $p\leq0.50$ and 2 when $p>0.50$. Its expectation is 1. After capping at 1, the estimate is 0 or 1 with equal probability, so its expectation is 0.5, below the true null fraction of 1. Non-null tail contributions can increase the estimate, but the capped estimator’s expectation need not be at least $\pi_0$.

The calculation uses marginal tail probabilities; independence among tests is not needed for that expectation identity. Dependence does, however, affect the variability of the tail count and can matter for the performance or guarantees of a multiple-testing procedure. A tail estimate by itself does not establish FDR control under an arbitrary dependence structure.

From an Estimated Null Fraction to Q-Values

Let $R(t)=\#\{i:p_i\leq t\}$, the number of tests rejected by a p-value threshold $t$. The FDR is the expected proportion of rejected tests that are false discoveries, with the proportion defined as zero when there are no rejections. Storey’s plug-in estimate of the FDR at a threshold $t$ with $R(t)>0$ is

\widehat{\mathrm{FDR}}(t) = \min\left\{1,\frac{\hat{\pi}_0 m t}{R(t)}\right\}.

The numerator estimates the number of true-null p-values at or below $t$, using the null fraction and the uniform-null probability $t$. The denominator is the observed number of rejections. This ratio is an estimated FDR, not generally the exact FDR: in particular, a ratio of estimated counts is not automatically the expectation of the random false-discovery proportion.

Definition: The Storey q-value for a test with p-value $p_i$ is the smallest estimated FDR attained by a p-value threshold that includes that test: $$ q_i=\min_{t\geq p_i,\ R(t)>0}\widehat{\mathrm{FDR}}(t). $$ Thus, a q-value is a minimum estimated FDR over thresholds at which the test would be included among the discoveries.

For distinct ordered p-values $p_{(1)}\leq\cdots\leq p_{(m)}$, the threshold $p_{(j)}$ includes $j$ tests. The estimated FDR at that threshold is $\min\{1,\hat{\pi}_0 m p_{(j)}/j\}$. Taking the smallest such value over thresholds that include rank $i$ gives

q_{(i)} = \min\left\{1,\ \min_{j\geq i}\frac{\hat{\pi}_0 m p_{(j)}}{j}\right\}.

The minimum over $j\geq i$ makes q-values nondecreasing with p-value rank. With tied p-values, use the threshold definition: $R(t)$ is the number of p-values at or below the threshold, and tests tied at that p-value receive the same q-value.

Worked Example: Calculating Q-Values

Return to the hypothetical ten-p-value family above, and use the estimate $\hat{\pi}_0=0.600$. The p-values are already ordered. For rank $j$, the uncapped ratio is $0.600(10)p_{(j)}/j=6p_{(j)}/j$.

For the first three ranks, the ratios are $6(0.002)/1=0.0120$, $6(0.015)/2=0.0450$, and $6(0.030)/3=0.0600$. For the remaining ranks they are $6(0.080)/4=0.120$, $6(0.110)/5=0.132$, $6(0.220)/6=0.220$, $6(0.410)/7=0.3514$, $6(0.580)/8=0.435$, $6(0.730)/9=0.4867$, and $6(0.940)/10=0.564$. These ratios are increasing, so the minimum over each rank and all later ranks is the ratio at that rank.

Rank	p-value	Storey q-value
1	0.002	0.0120
2	0.015	0.0450
3	0.030	0.0600
4	0.080	0.120
5	0.110	0.132
6	0.220	0.220
7	0.410	0.351
8	0.580	0.435
9	0.730	0.487
10	0.940	0.564

At a target estimated FDR of 0.05, the first two tests qualify because their q-values are at most 0.05; the third does not. The conclusion is that this procedure selects two tests at that estimated FDR level, under the method’s assumptions. It is not a claim that either selected test has a 4.5% or 1.2% probability of being false.

How the Null-Fraction Estimate Changes the Assessment

Holding the p-values and $m$ fixed, each uncapped threshold ratio is proportional to $\hat{\pi}_0$. A larger estimated null fraction therefore tends to yield larger q-values and fewer discoveries at a fixed target FDR; a smaller estimate tends to yield smaller q-values. The effect is direct, but the validity of the resulting assessment still depends on the quality of the p-values, the null-fraction estimate, and the applicable assumptions for the procedure.

Worked Example: Comparing Two Null-Fraction Estimates

For the same hypothetical family of ten tests, compare the earlier estimate $\hat{\pi}_0=0.600$ with a more conservative value of $\hat{\pi}_0=1$. This comparison isolates how the null-fraction input changes the q-values; it does not assert that both values are equally well supported by the data.

At rank 2, where $p_{(2)}=0.015$, the ratio using $\hat{\pi}_0=0.600$ is $0.600(10)(0.015)/2=0.0450$. With $\hat{\pi}_0=1$, it is $1(10)(0.015)/2=0.0750$. At rank 3, the corresponding ratios are $0.0600$ and $0.100$. Taking the minimum over later ranks leaves these values unchanged for these two ranks, so the q-values increase from 0.0450 to 0.0750 and from 0.0600 to 0.100, respectively.

At a target estimated FDR of 0.05, the estimate 0.600 yields two discoveries in this example, while the estimate 1 yields only the first. A smaller null-fraction estimate can therefore make the assessment less conservative. It should not be selected simply because it produces more discoveries: the choice of $\lambda$, stability of the tail estimate, and calibration of the p-values matter.

Interpretation and Common Pitfalls

Storey q-values are useful because they provide a threshold-based summary: a test’s q-value is the lowest estimated FDR among thresholds that would include it. They allow researchers to examine discoveries at a chosen estimated FDR level without treating each raw p-value as if it were already adjusted for the full testing family. In GWAS, where many variants are tested, accounting for the estimated null share can materially affect which associations meet a specified FDR target.

Do not read a q-value as an individual probability. A q-value of 0.05 does not mean there is a 5% probability that this particular variant is null. It summarizes an estimated false-discovery proportion across a threshold-defined set.
Do not assume the tail estimate is exact. The choice of $\lambda$, tail-count variability, and non-null p-values in the tail all affect $\hat{\pi}_0$. Estimates from different $\lambda$ values can vary.
Do not confuse estimated FDR with guaranteed FDR control. The plug-in estimate is not by itself a proof of a finite-sample guarantee. Dependence among association tests and the conditions of the chosen method must be considered.
Do not claim the capped estimator is conservative in expectation. The uncapped estimator has expectation at least $\pi_0$ under valid null p-values; capping it at 1 can reduce its expectation below $\pi_0$.

In practice, implementations may evaluate the tail estimator over several candidate $\lambda$ values and smooth or otherwise stabilize the resulting estimates. Such procedures address sensitivity to a single tail cutoff, but they do not remove the need to check calibration and interpret the resulting q-values as estimates.

Key takeaway: Storey’s method estimates the null fraction from the upper tail of the p-value distribution, then scales threshold-based FDR estimates by that fraction. Lower estimated null fractions generally reduce q-values, but a q-value is an estimated set-level error summary—not a probability that an individual association is false or an automatic guarantee of FDR control.

Check Your Understanding

Try these questions using the definitions and calculations above.

For $m=20$, $\lambda=0.50$, and 7 p-values greater than $\lambda$, calculate Storey’s capped null-fraction estimate.
Why can non-null p-values in the upper tail increase the uncapped tail estimate?
For $m=100$, $\hat{\pi}_0=0.80$, $p_{(5)}=0.002$, and distinct p-values, calculate the estimated FDR at threshold $p_{(5)}$.
In your own words, distinguish a q-value from the probability that one particular association is false.
Why can increasing $\lambda$ reduce contamination from alternatives but increase variability in the tail estimate?