Benjamini-Hochberg Testing Under Independence

Why Use a Step-Up Rule?

A genome-wide association analysis may test thousands or millions of variants. Controlling the chance of making even one false rejection can be very strict when the testing family is large. The Benjamini–Hochberg (BH) procedure instead targets the false discovery rate: the expected fraction of rejected hypotheses that are false. It can allow more discoveries than procedures designed to control the probability of any false rejection, while still providing a formal error-rate guarantee under stated assumptions.

The rule uses the entire ordered list of p-values. It compares each p-value with a threshold that increases with its rank, then rejects through the largest rank that meets its threshold. This is called a step-up procedure because the comparisons proceed from the smallest p-value toward the largest, and a qualifying rank determines a block of rejections.

Definition: Consider a family of $m$ hypotheses. Let $R$ be the number rejected and $V$ the number of rejected hypotheses whose null hypotheses are true. The false discovery proportion is $V/\max(R,1)$, so it is defined as zero when there are no rejections. The false discovery rate is its expectation: $$ \mathrm{FDR}=E\left[\frac{V}{\max(R,1)}\right]. $$

The Benjamini–Hochberg Rule

Choose a target level $q$, usually between 0 and 1, and let $p_1,\ldots,p_m$ be the p-values for the hypotheses in the family. Sort them in nondecreasing order, writing $p_{(1)}\le \cdots \le p_{(m)}$. The threshold at rank $i$ is $iq/m$. The procedure finds the largest rank that meets its threshold, not simply the first rank that fails.

Definition: The Benjamini–Hochberg procedure at target level $q$ sets $$ k=\max\left\{i\in\{1,\ldots,m\}:p_{(i)}\le \frac{iq}{m}\right\}. $$ If this set is nonempty, reject the hypotheses corresponding to $p_{(1)},\ldots,p_{(k)}$. If it is empty, make no rejections. The thresholds $iq/m$ are the BH rejection thresholds.

A p-value that fails its own rank threshold is not automatically a stopping point. A larger rank has a larger threshold, so it is possible for a later p-value to qualify even if an earlier one did not. Conversely, once the largest qualifying rank $k$ is identified, every p-value at a smaller rank is rejected, whether or not it was separately compared with the threshold at $k$.

Worked Example: Finding the Rejection Set

Suppose a hypothetical association analysis tests $m=8$ hypotheses and has sorted p-values $0.001, 0.008, 0.013, 0.021, 0.049, 0.110, 0.320,$ and $0.700$. Apply BH at $q=0.05$.

The rank-$i$ threshold is $i(0.05)/8$. In order, the thresholds are $0.00625, 0.01250, 0.01875, 0.02500, 0.03125, 0.03750, 0.04375,$ and $0.05000$.

At ranks 1 through 4, the comparisons are $0.001\le0.00625$, $0.008\le0.01250$, $0.013\le0.01875$, and $0.021\le0.02500$, all of which qualify. At rank 5, $0.049>0.03125$; ranks 6 through 8 also fail their thresholds. Thus the largest qualifying rank is $k=4$, and BH rejects the four hypotheses with p-values $0.001, 0.008, 0.013,$ and $0.021$. The result is a rejection set under a procedure targeting FDR 0.05; it does not assert that exactly 5% of these four findings are false.

Adjusted P-Values

A BH-adjusted p-value summarizes the smallest target level $q$ at which a hypothesis would be included in the BH rejection set. For ordered p-values, the raw rank-specific quantities $mp_{(j)}/j$ need not increase with rank. The adjustment takes a minimum over this value and all later ranks, and caps the result at 1. This makes the adjusted values nondecreasing in the ordered p-values.

Definition: For the hypothesis whose p-value has rank $i$, its BH-adjusted p-value is $$ \widetilde p_{(i)}=\min\left(1,\min_{j\ge i}\frac{m p_{(j)}}{j}\right). $$ To calculate them, work from the largest rank toward the smallest, taking the running minimum of $mp_{(j)}/j$, then cap values at 1. Rejecting at target $q$ using these adjusted p-values is equivalent to the BH step-up rule: reject precisely those with $\widetilde p_{(i)}\le q$.

Worked Example: Calculating Adjusted P-Values

Use the eight hypothetical p-values from the previous example. The rank-specific quantities $mp_{(i)}/i$, calculated before taking running minima, are $0.00800, 0.0320, 0.0347, 0.0420, 0.0784, 0.1467, 0.3657,$ and $0.700$.

Starting at rank 8, the running minimum is $0.700$. At rank 7, the minimum of $0.3657$ and $0.700$ is $0.3657$; at rank 6 it is $0.1467$; at rank 5 it is $0.0784$; and at rank 4 it is $0.0420$. Continuing gives $0.0347$ at rank 3, $0.0320$ at rank 2, and $0.00800$ at rank 1. No value exceeds 1, so capping does not change them.

In sorted order, the adjusted p-values are therefore approximately $0.00800, 0.0320, 0.0347, 0.0420, 0.0784, 0.1467, 0.3657,$ and $0.700$. At $q=0.05$, the first four are at most 0.05 and the fifth is not, matching the four rejections from the step-up calculation.

Worked Example: A Later Rank Can Determine the Cutoff

Consider a second hypothetical family with $m=5$, sorted p-values $0.004, 0.018, 0.026, 0.041,$ and $0.600$, and target $q=0.05$. The thresholds at ranks 1 through 5 are $0.010, 0.020, 0.030, 0.040,$ and $0.050$.

The first three comparisons qualify: $0.004\le0.010$, $0.018\le0.020$, and $0.026\le0.030$. Rank 4 fails because $0.041>0.040$, and rank 5 fails because $0.600>0.050$. The largest qualifying rank is 3, so BH rejects the first three hypotheses. The rank-4 failure does not undo the qualifying rank-3 result; nor does it prevent the rule from checking rank 5.

For confirmation, the adjusted p-value at rank 3 is the minimum of $1$, $5(0.026)/3$, $5(0.041)/4$, and $5(0.600)/5$. These quantities are approximately $0.04333, 0.05125,$ and $0.600$, so the adjusted value is $0.04333$, below 0.05. The rank-4 adjusted value is $0.05125$, just above 0.05. Thus the adjusted-value criterion gives the same three rejections.

Why the Independence Guarantee Holds

The standard BH guarantee requires a fixed family of $m$ tests, mutual independence of their p-values, and valid p-values for every true null hypothesis. A valid true-null p-value is super-uniform: for every $t\in[0,1]$, $P(p_i\le t\mid H_i\text{ is true})\le t$. It is uniform when it is an exact p-value obtained from a continuous null test statistic and its exact null distribution; discrete or conservative p-values may be strictly super-uniform. Continuity of a null statistic alone does not ensure a uniform p-value.

The independence condition applies to the p-values across the family, not just to each p-value considered separately. Under these conditions, if $m_0$ of the $m$ null hypotheses are true, BH satisfies the bound below.

Theorem: Under mutually independent p-values, with every true-null p-value super-uniform, the BH procedure at target $q$ satisfies $$ \mathrm{FDR}\le \frac{m_0}{m}q\le q. $$

A short conditional argument explains the bound. Fix one true-null p-value $p_i$ and hold the other p-values fixed. Let $R_i(0)$ denote the number of BH rejections if $p_i$ is replaced by zero. The step-up rank rule has this property: $p_i$ is rejected exactly when $p_i\le qR_i(0)/m$, and on that event the rejection count is $R_i(0)$. To see why, lowering a rejected p-value to zero cannot reduce the qualifying rank; the p-value already met the threshold at the original rejection rank, so the qualifying count stays the same. Conversely, if $p_i\le qR_i(0)/m$, restoring it from zero leaves it below the threshold at rank $R_i(0)$, while it cannot create a qualifying larger rank that was absent with $p_i=0$.

Consequently, conditional on all other p-values, the expected contribution of this true-null hypothesis to $V/\max(R,1)$ is $$ \frac{1}{R_i(0)}P\left(p_i\le\frac{qR_i(0)}{m}\,\middle|\,\text{other p-values}\right) \le \frac{1}{R_i(0)}\frac{qR_i(0)}{m} =\frac{q}{m}. $$ Here $R_i(0)\ge1$ for $q>0$, because setting one p-value to zero makes the first BH threshold qualify. Independence makes the conditional distribution of $p_i$ its true-null distribution, and super-uniformity supplies the inequality. Summing the bound over the $m_0$ true-null hypotheses gives $\mathrm{FDR}\le m_0q/m$. This derivation also shows why validity of each null p-value and independence across tests are distinct requirements.

Interpreting the Guarantee and Avoiding Pitfalls

FDR control is a repeated-procedure, expectation-based statement: across repetitions under the assumptions, the expected proportion of false rejections among all rejections is at most the stated bound. It does not mean that the realized false-discovery proportion in a particular analysis must be at most $q$. Nor does it mean that each rejected hypothesis has probability $q$ of being false. The procedure does not assign a posterior probability that an individual association is null.

In genetic association analyses, tests can be related through linkage disequilibrium, so independence should not be assumed merely because there is one p-value per variant. This tutorial’s guarantee is specifically for independent tests; applying it when that condition is not met requires a justification beyond the independent-test result. The testing family must also be defined before applying the rule: $m$ is the number of hypotheses in the family, and changing that family changes the rank thresholds.

Worked Example: Reading an FDR Target Correctly

Suppose a hypothetical analysis tests $m=100$ independent variant hypotheses, with $m_0=80$ true nulls, and applies BH at $q=0.05$. Under the theorem’s assumptions, the bound is $$ \mathrm{FDR}\le \frac{80}{100}(0.05)=0.040. $$

Thus the expected false discovery proportion is bounded above by 0.040 for this procedure and testing family. This does not say that any particular set of discoveries contains exactly 4% false findings, or that each discovery has a 4% probability of being false. In practice, $m_0$ is usually unknown, so the general guarantee $\mathrm{FDR}\le q=0.05$ is the useful bound that does not require knowing the number of true nulls.

Common mistake: “The p-value at rank 4 misses its threshold, so the procedure stops and rejects only the first three.” The correction is to evaluate every rank and choose the largest qualifying one; a failure at one rank is not a stopping rule. A related mistake is to describe $q=0.05$ as a 5% chance that each reported association is false. The actual guarantee concerns the expected fraction of false rejections over repetitions, under the stated assumptions.

Key takeaway: Sort the p-values, compare $p_{(i)}$ with $iq/m$ at every rank, and reject through the largest qualifying rank. Under independent tests and super-uniform true-null p-values, the expected false discovery proportion is at most $q$, and the BH-adjusted p-values give the equivalent decision rule.

Check Your Understanding

Use the definitions and rule above to answer the following.

For $m=10$ tests at $q=0.05$, what is the BH threshold at rank 6?
Why must the procedure choose the largest qualifying rank rather than stop at the first rank that fails?
In precise terms, what does $\mathrm{FDR}\le q$ say, and what does it not say about an individual rejection?
What condition on a true-null p-value is needed for the independence guarantee, and when is such a p-value uniform rather than merely super-uniform?
For sorted p-values $0.01, 0.03, 0.04$, $m=3$, and $q=0.05$, identify the largest qualifying rank and the rejection set.