Why Work Through the Ranks?
A genomic analysis may test thousands of hypotheses, but the Benjamini–Hochberg (BH) procedure makes its decisions by comparing ordered p-values with a simple sequence of rank-specific thresholds. Working through a small example carefully reveals two details that are easy to miss: the selected rank is the largest rank that meets its threshold, and the procedure rejects every hypothesis up to that rank, even if some earlier ranks do not meet their own thresholds.
The calculation itself is a rule for turning a chosen family of p-values into rejections. Its interpretation as a false discovery rate procedure also depends on assumptions about the p-values. The arithmetic examples below are hypothetical; they illustrate the rule, not findings from a real genomic study.
Conditions and the Calculation Rule
The family must be defined before interpreting the result: \(m\) is the number of hypotheses included in that family, not the number of p-values that happen to look promising. Each test must produce a p-value under the same analysis plan. For the usual BH false discovery rate guarantee, p-values for true null hypotheses must be valid (their null distributions are super-uniform), and a sufficient dependence condition is mutual independence of all p-values. Independence among only the true-null p-values, without an appropriate relationship to the other p-values, is not enough. Other sufficient dependence conditions exist, but are not needed to perform the hand calculation.
The step-up rule can be expressed as a short sequence of operations. Sorting is essential because the thresholds increase with rank; keeping test labels alongside the values is essential because the scientific conclusion concerns hypotheses, not anonymous numbers.
Sort all \(m\) values from smallest to largest, and record the hypothesis corresponding to each rank.
At rank \(r\), calculate \(r\alpha/m\), for \(r=1,\ldots,m\).
Compare \(p_{(r)}\) with \(r\alpha/m\) at every rank. Choose the largest rank \(k\) for which the p-value is no greater than its threshold.
Reject every hypothesis at ranks \(1\) through \(k\), then translate those ranks back to the original hypothesis labels.
A brief justification clarifies why the rule is step-up rather than a collection of separate rank-by-rank tests. By definition, the selected \(k\) is the maximum of the set of ranks satisfying \(p_{(r)}\leq r\alpha/m\). The decision rule then uses that maximum as a cutoff and rejects the first \(k\) ordered hypotheses. It does not require every rank \(r\leq k\) to satisfy its own inequality. This distinction is part of the definition of BH and matters when explaining a result.
Worked Example: Six Rejections From Eight Hypothetical Tests
Suppose eight genomic hypotheses have the following hypothetical p-values. The labels \(G_1,\ldots,G_8\) stand for distinct tests; the values are deliberately listed in their original, unsorted order.
| Hypothesis | G₁ | G₂ | G₃ | G₄ | G₅ | G₆ | G₇ | G₈ |
|---|---|---|---|---|---|---|---|---|
| P-value | 0.021 | 0.200 | 0.004 | 0.047 | 0.001 | 0.032 | 0.013 | 0.009 |
Use \(\alpha=0.05\). There are \(m=8\) tests, so the rank-\(r\) threshold is \(r(0.05)/8\). Sorting the values and retaining their labels gives:
| Rank \(r\) | Hypothesis | Ordered p-value \(p_{(r)}\) | Threshold \(r(0.05)/8\) | Qualifies? |
|---|---|---|---|---|
| 1 | G₅ | 0.001 | 0.00625 | Yes |
| 2 | G₃ | 0.004 | 0.01250 | Yes |
| 3 | G₈ | 0.009 | 0.01875 | Yes |
| 4 | G₇ | 0.013 | 0.02500 | Yes |
| 5 | G₁ | 0.021 | 0.03125 | Yes |
| 6 | G₆ | 0.032 | 0.03750 | Yes |
| 7 | G₄ | 0.047 | 0.04375 | No |
| 8 | G₂ | 0.200 | 0.05000 | No |
For example, the rank-6 threshold is \(6(0.05)/8=0.0375\), and \(0.032\leq0.0375\). The rank-7 comparison fails because \(0.047>7(0.05)/8=0.04375\); rank 8 also fails because \(0.200>0.0500\). Thus the largest qualifying rank is \(k=6\). Reject the hypotheses at ranks 1 through 6: \(G_5,G_3,G_8,G_7,G_1,\) and \(G_6\). Do not reject \(G_4\) or \(G_2\) by this procedure at the specified level.
Two Further Checks on the Rule
Worked Example: No Rank Qualifies
Consider five hypothetical tests with ordered p-values \(0.012, 0.030, 0.080, 0.110,\) and \(0.400\), using \(\alpha=0.05\). Since \(m=5\), the thresholds are \(r(0.05)/5\).
| Rank | P-value | Threshold | Comparison |
|---|---|---|---|
| 1 | 0.012 | 0.010 | 0.012 > 0.010 |
| 2 | 0.030 | 0.020 | 0.030 > 0.020 |
| 3 | 0.080 | 0.030 | 0.080 > 0.030 |
| 4 | 0.110 | 0.040 | 0.110 > 0.040 |
| 5 | 0.400 | 0.050 | 0.400 > 0.050 |
No rank satisfies its comparison, so \(k=0\) and the procedure makes no rejections. The smallest p-value is not automatically a rejection: it must be compared with the rank-1 threshold, which here is \(0.05/5=0.010\), smaller than \(0.012\).
Worked Example: The Largest Qualifying Rank Determines the Cutoff
Now take five hypothetical ordered p-values \(0.015,0.018,0.024,0.070,\) and \(0.400\), again with \(\alpha=0.05\). The thresholds are \(0.010,0.020,0.030,0.040,\) and \(0.050\), respectively.
At rank 1, \(0.015>0.010\), so that rank does not qualify. At rank 2, \(0.018\leq0.020\), and at rank 3, \(0.024\leq0.030\). At rank 4, \(0.070>0.040\), and at rank 5, \(0.400>0.050\). The largest qualifying rank is therefore \(k=3\).
BH rejects the first three hypotheses, with p-values \(0.015,0.018,\) and \(0.024\). In particular, the rank-1 hypothesis is rejected even though its own rank-1 comparison failed. That is not an exception or a correction to the calculation; it follows directly from the step-up rule, which sets the rejection cutoff using the largest qualifying rank.
How to Check, Compare, and Interpret the Result
A useful audit is to verify that (1) exactly \(m\) values were sorted, (2) the threshold at rank \(r\) is \(r\alpha/m\), (3) comparisons use “less than or equal to,” and (4) the largest qualifying rank was selected. Then confirm that every rejection is among ranks 1 through \(k\), and that the original labels have been restored correctly. Sorting mistakes or losing labels can produce an arithmetically plausible but scientifically misassigned list of discoveries.
BH and Bonferroni address different error-rate goals. Bonferroni uses the common threshold \(\alpha/m\) for each test to control the familywise error rate, the probability of at least one false rejection. BH uses increasing rank-specific thresholds to control the false discovery rate under appropriate conditions. In the eight-test example, the Bonferroni threshold is \(0.05/8=0.00625\), so only \(0.001\) and \(0.004\) fall at or below it; BH rejects six. This contrast illustrates why the procedures can yield different lists, not that one is universally preferable. The appropriate procedure depends on the scientific objective and assumptions.
The target \(\alpha\) is not a statement that exactly an \(\alpha\) fraction of this particular list will be wrong, nor does the calculation identify which rejected hypotheses are false. The BH rule concerns an expected error-rate guarantee across repeated applications under its assumptions. If genomic tests have dependence, do not infer that the guarantee follows merely because true-null p-values are mutually independent: a sufficient condition is mutual independence of the full p-value collection, with valid true-null p-values, or another suitable dependence condition. The numerical step-up rule can still be computed, but a guarantee requires appropriate assumptions.
A frequent mistake is to compare each p-value with its own threshold and reject only the ranks that pass individually. In the five-test example above, that approach would reject ranks 2 and 3 but omit rank 1. The correction is to find the largest qualifying rank first, then reject all ranks up to that cutoff. A second mistake is to stop at the first failed comparison; later ranks must still be checked because a later rank can qualify. Finally, \(\alpha\) must be fixed as the intended target level, not chosen after inspecting which value would yield a desired rejection count.
Check Your Understanding
Try each calculation before reading the solutions below.
- For \(m=4\), \(\alpha=0.10\), and ordered p-values \(0.008,0.025,0.060,0.200\), calculate the thresholds, find \(k\), and state the rejections.
- For \(m=3\), \(\alpha=0.05\), and ordered p-values \(0.020,0.030,0.040\), does BH reject any hypotheses?
- In one sentence, explain why failing the rank-1 comparison does not necessarily prevent rejection of the rank-1 hypothesis.
- Is independence among true-null p-values alone sufficient to claim the usual BH FDR guarantee? State a sufficient condition accurately.
Solutions. (1) The thresholds are \(0.10/4=0.025\), \(2(0.10)/4=0.050\), \(3(0.10)/4=0.075\), and \(4(0.10)/4=0.100\). The first three comparisons qualify: \(0.008\leq0.025\), \(0.025\leq0.050\), and \(0.060\leq0.075\); the fourth fails because \(0.200>0.100\). Thus \(k=3\), and the first three hypotheses are rejected. (2) The thresholds are \(0.0167,0.0333,\) and \(0.0500\). Rank 3 qualifies because \(0.040\leq0.0500\), so \(k=3\) and all three hypotheses are rejected. (3) BH rejects all ranks through the largest qualifying rank, not just ranks that pass their own comparisons; a later qualifying rank can therefore set a cutoff that includes rank 1. (4) No. A sufficient condition is valid p-values under the true nulls and mutual independence of all p-values; other suitable dependence conditions can also support a guarantee.