From an FDR Threshold to a Test’s q-Value
A false discovery rate procedure selects a set of hypotheses, and its error rate concerns that set as a whole. A Storey q-value gives a related summary for an individual test: it is the smallest estimated false discovery rate attainable by using a p-value cutoff that would still include that test. The minimization matters. The estimated false discovery rate at the test’s own p-value is only one candidate; a less stringent cutoff may yield a smaller estimate.
This definition is useful when comparing tests or describing a discovery list. It does not say that a particular null hypothesis has a given probability of being true. Instead, it associates each observed p-value with the best (smallest) estimated FDR among eligible threshold-based rejection sets.
The introductory material on estimating the null proportion explains how \(\hat{\pi}_0\) can be obtained. Here it is treated as an input to the q-value calculation. In the formula, \(\hat{\pi}_0m\) estimates the number of true null hypotheses, and multiplying by \(t\) estimates how many of their p-values fall at or below \(t\), under the usual uniform-null model. Dividing by the observed number of rejections \(R(t)\) gives a plug-in estimate of the false-discovery fraction for that threshold. The cap at 1 keeps the reported estimate within the range of a proportion.
This is an estimate, not an exact identity for the realized fraction of false discoveries. Its reliability depends on the p-values and on the null-proportion estimate. In particular, a q-value is not the probability that test \(i\) is null given its data, nor is it by itself a guarantee that a selected list has a specified FDR under every dependence pattern.
Why the Minimum Is Over Eligible Cutoffs
A cutoff \(t\) rejects every test with \(p_j\le t\). If \(t\ge p_i\), test \(i\) is included in that rejection set. Raising the cutoff can add other tests, changing both the estimated number of false discoveries and the total number of rejections. As a result, the estimated FDR can go up or down across observed p-value cutoffs. The q-value records the smallest estimate among the cutoffs that include test \(i\).
For a finite p-value list, this minimum can be checked at the distinct observed p-values. To see why, sort the distinct values as \(c_1<\cdots<c_K\). For a cutoff between \(c_j\) and \(c_{j+1}\), the rejection count is constant: \(R(t)=R(c_j)\). The numerator \(\hat{\pi}_0mt\) does not decrease as \(t\) increases. Thus the estimated FDR cannot be smaller inside that interval than at its left endpoint \(c_j\). Above the largest observed p-value, the rejection count remains \(m\), while the numerator again does not decrease. Consequently, for test \(i\), it suffices to check every distinct observed p-value \(c_j\ge p_i\). This argument also handles ties: the entire tied group enters \(R(t)\) together.
The minimization also implies a useful ordering property. If \(p_i\le p_k\), then the eligible cutoffs for test \(k\) are a subset of those for test \(i\). Minimizing over the larger set cannot give a larger minimum, so \(q_i\le q_k\). Therefore q-values are nondecreasing as p-values increase, with equal p-values receiving equal q-values.
Worked Example: Checking Every Eligible Cutoff
Suppose eight tests have sorted p-values \(0.004, 0.012, 0.018, 0.040, 0.110, 0.240, 0.550,\) and \(0.900\), and \(\hat{\pi}_0=0.75\). Calculate the q-value for the test with p-value \(0.018\). The candidate cutoffs must include every distinct observed p-value from \(0.018\) onward.
Here \(\hat{\pi}_0m=0.75(8)=6\). The full cutoff table is:
| Cutoff \(t\) | \(R(t)\) | \(\widehat{\mathrm{FDR}}(t)=6t/R(t)\) |
|---|---|---|
| 0.004 | 1 | 0.0240 |
| 0.012 | 2 | 0.0360 |
| 0.018 | 3 | 0.0360 |
| 0.040 | 4 | 0.0600 |
| 0.110 | 5 | 0.132 |
| 0.240 | 6 | 0.240 |
| 0.550 | 7 | 0.4714 |
| 0.900 | 8 | 0.675 |
For the test with \(p_i=0.018\), the eligible estimated FDRs are \(0.0360, 0.0600, 0.132, 0.240, 0.4714,\) and \(0.675\). Their minimum is \(0.0360\), so \(q_i=0.0360\). The test’s own cutoff attains the minimum here, but that cannot be assumed without checking the other eligible cutoffs.
In context, this q-value is the smallest estimated FDR among the threshold-based rejection sets that include this test. It is not a 3.6% probability that this particular test is a false discovery.
A Later Cutoff Can Give a Smaller Estimate
The estimated FDR at a test’s own p-value need not be the minimum. A later cutoff raises the estimated number of null p-values below the cutoff, but it can also add enough discoveries to make the estimated fraction smaller. The next example shows why using only the test’s own cutoff can give the wrong q-value.
Worked Example: The Minimum Occurs at a Later Cutoff
Consider six sorted p-values \(0.010, 0.020, 0.030, 0.200, 0.210,\) and \(0.900\), with \(\hat{\pi}_0=0.50\). Find the q-value for the test with p-value \(0.200\).
The multiplier is \(\hat{\pi}_0m=0.50(6)=3\). The full set of observed cutoffs and their estimates is:
| Cutoff \(t\) | \(R(t)\) | \(\widehat{\mathrm{FDR}}(t)=3t/R(t)\) |
|---|---|---|
| 0.010 | 1 | 0.0300 |
| 0.020 | 2 | 0.0300 |
| 0.030 | 3 | 0.0300 |
| 0.200 | 4 | 0.150 |
| 0.210 | 5 | 0.126 |
| 0.900 | 6 | 0.450 |
The test’s own cutoff gives \(3(0.200)/4=0.150\). But the eligible cutoffs are \(0.200, 0.210,\) and \(0.900\), whose estimated FDRs are \(0.150, 0.126,\) and \(0.450\). The minimum is \(0.126\), attained at cutoff \(0.210\), so the test’s q-value is \(0.126\). Reporting \(0.150\) would report the estimate at one cutoff, not the q-value defined by the minimum.
The Role of Ties and of \(\hat{\pi}_0\)
When p-values are tied, every test in a tied group has the same eligible cutoffs and the same q-value. The rejection count jumps by the size of the group at its shared p-value; it is not appropriate to include only part of the tied group in \(R(t)\). The following example calculates the q-values using the complete set of distinct cutoffs.
Worked Example: Tied P-Values and Shared Q-Values
Suppose seven tests have p-values \(0.020, 0.020, 0.050, 0.050, 0.050, 0.400,\) and \(0.800\), with \(\hat{\pi}_0=0.60\). Then \(\hat{\pi}_0m=0.60(7)=4.2\). The distinct observed cutoffs are \(0.020, 0.050, 0.400,\) and \(0.800\).
| Cutoff \(t\) | \(R(t)\) | \(\widehat{\mathrm{FDR}}(t)=4.2t/R(t)\) |
|---|---|---|
| 0.020 | 2 | 0.0420 |
| 0.050 | 5 | 0.0420 |
| 0.400 | 6 | 0.280 |
| 0.800 | 7 | 0.480 |
For either test with p-value \(0.020\), the eligible estimates are \(0.0420, 0.0420, 0.280,\) and \(0.480\), giving \(q=0.0420\). Each of the three tests with p-value \(0.050\) also has \(q=0.0420\), since its eligible estimates begin at the \(0.050\) cutoff. The test at \(0.400\) has \(q=0.280\), and the one at \(0.800\) has \(q=0.480\). The ties are included as groups, so the rejection counts are 2 and 5 at the first two cutoffs, respectively.
Holding the p-values fixed, a larger \(\hat{\pi}_0\) generally raises the estimated FDRs because it increases the estimated number of true nulls. Before the cap at 1, multiplying \(\hat{\pi}_0\) by a constant multiplies every cutoff estimate by that constant. Thus the null-proportion estimate affects the scale of q-values; it is not a cosmetic adjustment.
Interpretation, Comparisons, and Diagnostics
A Storey q-value is a threshold-based summary. For example, a q-value of \(0.08\) means the minimum estimated FDR over cutoffs that include that test is 8%. It does not mean there is an 8% chance that the test’s null hypothesis is true. Nor does it guarantee that the actual false discovery proportion in any particular selected list will be 8% or less.
The definition resembles the familiar idea of adjusted p-values: both summarize a test relative to a family of possible rejection thresholds. The Storey estimate uses \(\hat{\pi}_0m\) in its numerator, whereas the Benjamini–Hochberg adjusted-p-value calculation uses \(m\). Thus the estimated null proportion is an explicit part of Storey q-values. They are not interchangeable with unadjusted p-values, and their interpretation is not a posterior probability for an individual hypothesis.
Useful checks include confirming that \(R(t)\) counts all p-values at or below each cutoff, that every eligible distinct cutoff has been considered, and that q-values do not decrease as p-values increase. A violation of the ordering property often signals an incomplete minimization or a tie-handling error. Values should also be consistent with the stated \(\hat{\pi}_0\); for instance, changing that estimate while leaving the p-values fixed should change the estimated FDR scale.
The procedure has limitations. The estimated FDR depends on the quality of \(\hat{\pi}_0\), on valid null p-values, and on the setting in which the tests are dependent or independent. The formula alone does not establish a formal FDR guarantee under arbitrary dependence. It is therefore important to report the null-proportion estimate and the interpretation as an estimated, threshold-based error rate, rather than presenting q-values as certain error probabilities for individual findings.
A Common Mistake
A common error is to calculate \(\widehat{\mathrm{FDR}}(p_i)\) and call that number the q-value. That is only the estimated FDR at one eligible cutoff. For instance, in the second worked example, this shortcut gives \(0.150\) for the test at \(0.200\), while the minimum across all eligible cutoffs is \(0.126\). The correction is to check the complete set of distinct observed p-values at least as large as \(p_i\), then take the minimum of their estimated FDRs. A few selected cutoffs do not establish the q-value unless they include all candidates needed to verify the minimum.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- Four tests have sorted p-values \(0.020, 0.060, 0.100,\) and \(0.500\), and \(\hat{\pi}_0=0.50\). Calculate all observed-cutoff estimates and the q-value for each test.
- Why is the estimated FDR at a test’s own p-value not necessarily its q-value?
- If the p-values and \(m\) stay fixed but \(\hat{\pi}_0\) increases, what happens to the uncapped cutoff estimates? Explain.
- Three tests have p-values \(0.03, 0.03,\) and \(0.20\). Why must the rejection count at cutoff \(0.03\) be 2 rather than 1?
Solutions. (1) Here \(\hat{\pi}_0m=0.50(4)=2\). At the four cutoffs, the estimates are \(2(0.020)/1=0.0400\), \(2(0.060)/2=0.0600\), \(2(0.100)/3=0.0667\), and \(2(0.500)/4=0.250\). Taking eligible minima gives q-values \(0.0400, 0.0600, 0.0667,\) and \(0.250\), respectively. (2) A later cutoff can have a smaller estimate because it can add discoveries as well as increase the estimated null count; the q-value is the minimum across all eligible cutoffs. (3) Each uncapped estimate is proportional to \(\hat{\pi}_0\), so increasing it raises every estimate by the same multiplicative factor, with the p-values and rejection counts unchanged. (4) By definition \(R(t)=\#\{j:p_j\le t\}\), so both tests tied at \(0.03\) are included at that cutoff.