What a Genomic Discovery List Represents
A list of genes passing an FDR procedure is the result of a specified analysis, not a self-interpreting catalogue of confirmed effects. To read it correctly, identify the testing family (all hypotheses included in the adjustment), the procedure and target used, and the rule that produced the list. These details matter because changing the family or procedure can change the discoveries, even when the p-values for individual tests do not change.
Suppose each test concerns a null hypothesis that a particular gene has no association with a phenotype. The family might consist of all genes tested for one phenotype under a prespecified model. It is not necessarily the genes that were later highlighted, or only the genes with small p-values. The family should be defined by the analysis plan and the actual set of tests entering the adjustment.
An FDR target such as \(0.05\) is a bound on this expectation under the procedure’s conditions. It is not a statement that the particular list has exactly 5% false discoveries, that each listed gene has a 5% probability of being null, or that any individual result is correct with probability \(0.95\). The realized \(V\) is generally unknown, so the FDP of a particular list is not known just from the target.
Reading a Benjamini–Hochberg List
For the Benjamini–Hochberg (BH) procedure, order the \(m\) p-values as \(p_{(1)}\leq\cdots\leq p_{(m)}\). At target \(\alpha\), find the largest rank \(k\) satisfying \(p_{(k)}\leq k\alpha/m\). If such a rank exists, reject the hypotheses corresponding to \(p_{(1)},\ldots,p_{(k)}\). The observed p-value \(p_{(k)}\) is often reported as the BH cutoff, but the decision comes from the rank-based rule, not from comparing every p-value with \(\alpha\).
Under independent tests with valid, uniform null p-values, BH controls FDR at no more than \(\alpha\). It also has this guarantee under positive regression dependence on a subset (PRDS) for the true-null test statistics. Dependence among genomic tests, including dependence induced by linkage disequilibrium, therefore needs to be considered in choosing and interpreting a procedure. Under arbitrary dependence, the Benjamini–Yekutieli (BY) procedure gives a more conservative control guarantee when the null p-values are valid.
The critical value \(k\alpha/m\) at the selected rank is not generally the same as the largest rejected p-value \(p_{(k)}\). Nor does it follow that every p-value below \(\alpha\) is rejected: the number of tests and the rank of a p-value determine the threshold. A report should identify the actual procedure and family, rather than describe the results only as “significant at 0.05.”
Worked Example: Finding the BH Cutoff in a Small Family
Consider 12 hypothetical gene tests for one phenotype. Their sorted p-values are \(0.0004, 0.0015, 0.0030, 0.0060, 0.0090, 0.0120, 0.0210, 0.0340, 0.0410, 0.0550, 0.2000,\) and \(0.6000\). Apply BH at \(\alpha=0.05\), treating these 12 tests as the complete family.
At rank \(i\), the BH critical value is \(i(0.05)/12\). At rank 7 it is \(7(0.05)/12=0.02917\), and \(p_{(7)}=0.0210\leq0.02917\). At rank 8 it is \(8(0.05)/12=0.03333\), but \(p_{(8)}=0.0340>0.03333\). Rank 9 also fails because \(0.0410>9(0.05)/12=0.03750\); the larger ranks fail as well. Thus the largest passing rank is \(k=7\).
The rejection list contains the seven genes with p-values at most \(0.0210\). The largest rejected p-value, \(0.0210\), is the observed cutoff; the rank-7 critical value is \(0.02917\). Under the relevant BH assumptions, the procedure controls the expected false discovery proportion for this family at no more than 5%. It does not tell us how many of these seven genes are false discoveries in this particular analysis.
What to Report Alongside the List
A concise report should let another analyst reconstruct what was adjusted and understand the scope of the claim. State the phenotype and model or test defining the hypotheses, the family size \(m\), the FDR procedure, the chosen target, the number of rejections, and the cutoff or rank rule. If the family was restricted—for example, to a prespecified set of genes—say so. Also report how the family was formed, especially if filtering or selection occurred before adjustment.
For BH, a useful report gives both the observed largest rejected p-value \(p_{(k)}\) and the selected rank \(k\), or enough information to reproduce the rule. For q-value reporting, specify the q-value method and target used to select discoveries. A q-value is a minimum estimated FDR level at which a test would be called significant under that method; it is not a posterior probability that the null hypothesis for that gene is true. Storey q-values and BH-adjusted results can differ because they use different estimation or adjustment details, so the procedure’s name is part of the result.
Worked Example: Writing a Reproducible Genomic Result
Suppose a hypothetical expression analysis tests 20,000 genes for association with one phenotype. The analysts apply BH at \(\alpha=0.05\), and the largest passing rank is \(k=31\), with \(p_{(31)}=0.000074\). The next value is \(p_{(32)}=0.000082\). Assume all later ranks also fail their BH comparisons.
The rank-31 critical value is \(31(0.05)/20{,}000=0.0000775\), so rank 31 passes because \(0.000074\leq0.0000775\). At rank 32 the critical value is \(32(0.05)/20{,}000=0.0000800\), and \(0.000082>0.0000800\), so rank 32 fails. The BH rejection list therefore contains 31 genes.
A suitable report is: “For the 20,000 gene-level tests of association with the specified phenotype, BH at a 0.05 FDR target rejected 31 hypotheses; the largest rejected p-value was \(0.000074\) at rank 31 (critical value \(0.0000775\)).” The report should additionally describe the model and the test family in enough detail to identify which hypotheses were included. The 0.05 target describes the procedure’s expected error-rate control under its assumptions; it is not evidence that precisely one or two of these 31 genes are false.
Dependence, Alternatives, and Diagnostics
The choice of procedure affects what the list can support. Under independence or PRDS conditions, BH is often used for genomic families. If arbitrary dependence must be covered and valid null p-values are available, BY uses the more stringent threshold \(i\alpha/(m c_m)\), where \(c_m=\sum_{j=1}^m1/j\). Because \(c_m\geq1\), each BY critical value is no larger than its BH counterpart, so BY’s rejection set cannot be larger than BH’s for the same p-values and target. This conservatism can reduce discoveries, but it does not repair invalid p-values or an incorrectly specified testing family.
Dependence diagnostics can describe patterns in the p-values and test statistics, but a plot alone does not prove that PRDS holds. A sound report distinguishes what was assessed from what is guaranteed: for example, it can state that BH was used and describe the dependence rationale, without claiming that an empirical diagnostic establishes the theorem’s conditions. If the dependence structure makes the BH guarantee uncertain, explain that limitation or use a procedure with a guarantee suited to the stated dependence assumptions.
Worked Example: Comparing BH and BY Under a Dependence Concern
For four hypothetical tests, suppose the sorted p-values are \(0.001, 0.012, 0.020,\) and \(0.200\), with target \(\alpha=0.05\). Compare BH with BY, where \(c_4=1+1/2+1/3+1/4=25/12\approx2.0833\).
The BH critical values are \(0.0125, 0.0250, 0.0375,\) and \(0.0500\). The first three p-values pass their respective comparisons, while \(0.200>0.0500\), so BH rejects three hypotheses. For BY, the rank-specific critical values are \(i(0.05)/(4(2.0833))\), approximately \(0.006000, 0.01200, 0.01800,\) and \(0.02400\). The first two pass; rank 3 fails because \(0.020>0.01800\), and rank 4 fails. BY rejects two hypotheses.
This example shows how a dependence-robust procedure can yield a shorter list. It does not establish that BY is always preferable: its suitability depends on the inferential goal and the assumptions about p-value validity and dependence. The family remains all four tests under either procedure.
Several checks help identify a list that may be misleading. Confirm that \(m\) matches the tests actually adjusted, that the p-values correspond to the stated hypotheses, and that the sorted p-values satisfy the reported rank rule. Check whether any filtering or model selection used the same data and could affect p-value validity. Inspect dependence as part of the analysis, but do not infer a formal guarantee solely from a reassuring-looking histogram or correlation summary. Finally, distinguish an association signal from a causal effect: rejection of a gene-level null does not by itself establish a biological mechanism, direction of causation, replication, or clinical importance.
Common Mistakes and Limits on Individual Claims
- Wrong: “At a 5% FDR target, each listed gene has a 5% chance of being a false discovery.” Correction: the target concerns the expected false discovery proportion over repeated applications of the procedure, under its conditions. It is not a gene-specific probability.
- Wrong: “The procedure found exactly 5% false genes in this list.” Correction: the realized number \(V\) is usually unknown. FDR control is not a guarantee of the exact FDP in each dataset.
- Wrong: “All p-values below 0.05 are discoveries.” Correction: BH uses rank-dependent critical values; the rejection set depends on \(m\) and the ordered p-values.
- Wrong: “The list is reproducible because we report the target.” Correction: also report the family, procedure, and enough information about the model and selection to identify the tests.
A discovery list is therefore best understood as a set of rejections produced by a specified multiple-testing procedure. Its FDR target describes a repeated-analysis error rate under stated assumptions; it does not replace gene-specific validation, effect-size estimation, or biological interpretation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study tests 8,000 genes for one phenotype and reports BH at 0.05. What else should be stated to make the rejection list interpretable?
- In the 12-test example, why is the BH cutoff not \(0.05\), and what are the observed cutoff and rank-7 critical value?
- Does an FDR target of 0.05 mean that each rejected gene has a 95% probability of being associated with the phenotype? Explain.
- For the four-test example, how many hypotheses does BY reject, and why is its list no larger than BH’s?
Solutions. (1) State which hypotheses and tests define the family, the model or analysis defining them, the number of rejections, the procedure and target, and the cutoff or rank rule; disclose relevant filtering or selection. (2) BH’s threshold depends on rank and family size. At rank 7, the critical value is \(7(0.05)/12=0.02917\); the observed cutoff is \(p_{(7)}=0.0210\). (3) No. The 0.05 target bounds the expected false discovery proportion over repetitions under the procedure’s assumptions; it is not an individual gene’s probability of being associated or null. (4) BY rejects two. Its critical values divide the BH thresholds by \(c_4=25/12>1\), so they are no larger than BH’s, and a p-value passing a BY rank comparison also passes the corresponding BH comparison.