Two Different Requirements Behind FDR Guarantees
In a genomic analysis, thousands of hypotheses may be tested, and their p-values can be dependent because tests share samples, genomic regions, or other sources of variation. Two properties that are easy to conflate are marginal validity and independence. Validity concerns the distribution of each p-value when its own null hypothesis is true. Independence concerns the joint distribution of p-values across tests. One property does not imply the other.
This distinction matters for the Benjamini–Hochberg (BH) procedure. Its standard false discovery rate guarantee under independence requires valid p-values for true nulls as well as a suitable independence condition. Validity by itself is not enough for that particular guarantee under arbitrary dependence. Conversely, independence does not repair p-values that are anti-conservative under their nulls.
An exactly uniform null p-value satisfies \(P(P_i\leq t)=t\). A discrete or conservative test may instead satisfy a strict inequality for some thresholds. Super-uniformity allows both. For a composite null hypothesis, validity is required at each parameter value covered by that null, or under the specified null sampling distribution; it is not enough that the p-value looks approximately uniform under one convenient setting.
Write \(m\) for the total number of hypotheses and \(m_0\) for the number of true null hypotheses. If a multiple-testing procedure makes \(R\) rejections and \(V\) of those are false rejections of true nulls, its false discovery rate is \( \mathrm{FDR}=E[V/(R\vee1)] \), where \(R\vee1=\max(R,1)\). This is an expectation over repeated data sets under the assumed model. It is not the probability that a particular rejection is false.
What Marginal Validity Does—and Does Not—Say
Super-uniformity controls the chance that a particular true-null p-value falls below any fixed threshold. It does not specify how often several true-null p-values fall below thresholds together, nor how the p-values are arranged relative to one another. Those joint features can affect a procedure whose rejection thresholds depend on the ordered list of all p-values.
For BH at target level \(q\), sort the observed p-values as \(P_{(1)}\leq\cdots\leq P_{(m)}\). The procedure rejects through the largest rank \(k\) satisfying \(P_{(k)}\leq qk/m\), if such a rank exists. The threshold at rank \(k\) therefore depends on the full collection of p-values. A guarantee for this adaptive rule needs assumptions about the joint distribution as well as about each true-null marginal.
Worked Example: A Valid Marginal Is Not an Independence Statement
Suppose one true-null p-value has distribution \(P_1=0.2\) with probability \(0.2\) and \(P_1=1\) with probability \(0.8\). Check whether it is super-uniform.
For \(0\leq t<0.2\), \(P(P_1\leq t)=0\leq t\). For \(0.2\leq t<1\), \(P(P_1\leq t)=0.2\leq t\). At \(t=1\), the probability is \(1\). Thus \(P_1\) is super-uniform, though it is not exactly uniform.
This calculation establishes only a property of \(P_1\)'s marginal distribution. To make a statement about independence, we would need at least one other p-value and information about their joint distribution. For example, two p-values could each have this valid distribution but always take the same value; they would then be dependent.
Why Independence Enters the BH Bound
Under mutual independence of the p-values and super-uniformity for every true null, the standard BH guarantee is \(\mathrm{FDR}\leq (m_0/m)q\). In particular, the FDR is at most \(q\). Independence is used in the argument to separate a true-null p-value from the rejection count determined when that p-value is set to zero. Marginal validity then bounds the probability that this p-value crosses a specified threshold.
Here is the key argument. For a true-null index \(i\), let \(R_i\) be the number of BH rejections when \(P_i\) is replaced by zero and the other p-values are left unchanged. The BH ordering rule gives the following leave-one-out relation: if hypothesis \(i\) is rejected and the observed rejection count is \(r\), then \(R_i=r\) and \(P_i\leq qr/m\). Conversely, if \(R_i=r\) and \(P_i\leq qr/m\), hypothesis \(i\) is rejected with \(r\) rejections. Intuitively, replacing a p-value already among the rejections by zero does not change the rejection count; the other p-values determine the count at which it can enter.
Since \(R=0\) contributes no false rejections, write the expected false discovery proportion as a sum over true nulls. For one such \(i\), independence of \(P_i\) from the other p-values makes \(R_i\) independent of \(P_i\). Conditioning on the possible values \(r\geq1\) of \(R_i\) gives
The first inequality uses independence and super-uniformity: \(P(P_i\leq qr/m)\leq qr/m\). The last inequality follows because the probabilities \(P(R_i=r)\) over \(r=1,\ldots,m\) sum to at most one. Summing the bound \(q/m\) over the \(m_0\) true nulls yields
This proof shows the separate roles of the assumptions: super-uniformity supplies the threshold bound, while independence permits it to be used after conditioning on the rejection count generated by the other p-values. The result does not say that independence is the only possible route to BH control. The earlier material on positive regression dependence gives another sufficient joint condition. Nor does the proof say that every dependent configuration causes BH to fail; it says that marginal validity alone does not establish this independence-based guarantee.
Worked Example: Valid Marginals but BH FDR Above Its Target
Consider two true null hypotheses and BH at \(q=0.10\). Let \(U\sim\mathrm{Uniform}(0,1)\), and define the p-values by
The intervals mapped by \(P_2\) are respectively \([0.10,0.15)\), \([0.05,0.10)\), \([0,0.05)\), and \([0.15,1]\). They partition \([0,1]\), and each is obtained by a length-preserving translation from its corresponding interval for \(U\). Thus \(P_2\), like \(P_1\), is uniform and hence super-uniform. But \(P_1\) and \(P_2\) are dependent because both are determined by \(U\).
For \(m=2\), BH rejects if the smaller p-value is at most \(q/2=0.05\), or if both p-values are at most \(q=0.10\). If \(0\leq U<0.05\), \(P_1\leq0.05\), so there is a rejection. If \(0.05\leq U<0.10\), both p-values equal \(U\), so both are at most \(0.10\). If \(0.10\leq U<0.15\), \(P_2=U-0.10<0.05\), so there is a rejection. For \(U\geq0.15\), neither rejection condition holds. The rejection probability is therefore \(0.05+0.05+0.05=0.15\).
All hypotheses are true, so every rejection is false and the false discovery proportion is one whenever there is a rejection. Hence \(\mathrm{FDR}=0.15\), which exceeds the nominal \(q=0.10\). The marginal p-values are valid; it is the independence condition that fails. This example illustrates why validity alone cannot support the usual BH guarantee.
Independence Cannot Fix Invalid P-Values
The converse distinction is equally important. Suppose every true-null p-value is identically zero. Constant random variables are mutually independent, but they are not super-uniform: for any \(t>0\), \(P(P_i\leq t)=1>t\). Under BH, all such hypotheses are rejected, giving FDR \(1\) when all nulls are true. Independence does not protect against a test that produces anti-conservative p-values.
Worked Example: Checking an Anti-Conservative Null P-Value
Suppose a test claimed to have a true null produces \(P_i=0.01\) with probability \(0.10\), and \(P_i=1\) otherwise. At \(t=0.01\),
The p-value is not super-uniform. If several tests use independent copies of this mechanism, the p-values are independent but still invalid. Any multiple-testing guarantee that requires valid true-null p-values is inapplicable. The remedy is to examine the test calibration and null model, not to rely on independence.
What to Check, and What the Guarantee Covers
These assumptions concern different parts of an analysis and call for different questions. A null p-value histogram may help reveal some forms of marginal miscalibration, but it cannot establish independence: dependence can be strong even when each marginal distribution is exactly uniform. Likewise, observing correlation among p-values does not by itself prove that the BH guarantee fails; dependence conditions and dependence-robust procedures require their own reasoning.
| Property | Question it answers | What it supports |
|---|---|---|
| Super-uniformity | For a true null, is \(P(P_i\leq t)\leq t\) at every threshold \(t\)? | Controls each true-null marginal tail probability. |
| Independence | Does the joint distribution factor across tests, or does each true-null p-value behave independently of the others? | Supports the standard BH proof under independence. |
| PRDS or another dependence condition | Does the joint dependence satisfy a condition that supports a known guarantee? | Can support BH guarantees without mutual independence. |
| Arbitrary-dependence protection | Is a procedure designed to control FDR without a special dependence assumption? | BY uses a more conservative threshold with a harmonic-factor adjustment. |
The standard BY guarantee under arbitrary dependence still requires valid true-null p-values. Its adjustment addresses dependence, not invalid marginal calibration. These guarantees are also model-based: violations of the sampling assumptions used to compute p-values can undermine super-uniformity. In genomic applications, the number of tests may be large, but neither a large \(m\) nor a plausible-looking histogram substitutes for the assumptions required by a procedure.
Common Mistakes
- “Every null p-value is valid, so BH is guaranteed under any dependence.” Super-uniformity is a marginal condition. The standard independence proof also uses a joint-distribution condition; for general dependence, use a guarantee appropriate to the dependence assumptions.
- “The p-values are independent, so the procedure is safe.” Independence does not imply valid null p-values. First check calibration under each true null.
- “The p-values look uniform, so they must be independent.” Marginal distributions do not determine a joint distribution. Uniform marginals can be strongly dependent.
- “Dependence means BH must fail.” Dependence alone does not establish failure. Some dependent settings satisfy sufficient conditions for BH control; guarantees depend on the actual condition, not simply the presence of dependence.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A true-null p-value satisfies \(P(P_i\leq t)\leq t\) for every \(t\). What property has been established, and what has not?
- With two all-true nulls, BH uses \(q=0.08\). If the p-values are dependent but each is exactly uniform, does marginal validity alone establish the standard independence-based FDR guarantee?
- If independent p-values are identically zero under true nulls, which requirement fails?
- Under independence and valid p-values, what upper bound does the standard BH result give when \(m=100\), \(m_0=60\), and \(q=0.05\)?
Solutions: 1. Super-uniformity, or marginal validity, has been established for that true null. Nothing about independence or the joint distribution has been established. 2. No. Exact uniformity gives marginal validity, but dependence means the stated independence condition is absent; another suitable dependence guarantee would need separate justification. 3. Super-uniformity fails: at any \(t>0\), \(P(P_i\leq t)=1>t\). Independence does not repair this. 4. The bound is \((m_0/m)q=(60/100)(0.05)=0.030\), so the FDR is at most \(0.030\), or 3.0%, under the stated assumptions.