BH-Adjusted P-Values From Ordered Tests

From BH Rejection Thresholds to Adjusted Values

The Benjamini–Hochberg (BH) procedure is often applied at a chosen significance level, but an analysis may also need a summary for each test that works across many possible levels. The BH-adjusted p-value provides that summary: it is the smallest BH level at which that test would be rejected. Computing it requires more than multiplying a raw p-value by a rank-based factor. Because BH uses a step-up rule, the rank-scaled quantities must be made monotone with a reverse cumulative minimum.

The calculation is within a specified family of $m$ tests. Write the raw p-values in nondecreasing order as $p_{(1)}\le \cdots \le p_{(m)}$, where $(i)$ identifies the test at rank $i$. The BH critical value at rank $i$, for a chosen level $\alpha$, is $i\alpha/m$. The BH step-up procedure finds the largest rank $k$ satisfying $p_{(k)}\le k\alpha/m$, then rejects the hypotheses at ranks $1,\ldots,k$. The focus here is how to turn those rank-based decisions into adjusted p-values for individual tests.

Definition: For $m$ raw p-values ordered as $p_{(1)}\le\cdots\le p_{(m)}$, the BH-adjusted p-value at rank $i$ is $$ \tilde p_{(i)}=\min\left\{1,\ \min_{j\ge i}\left(\frac{m}{j}p_{(j)}\right)\right\}. $$ The adjusted value for the original test at rank $i$ is $\tilde p_{(i)}$. The inner minimum is taken over ranks $i,i+1,\ldots,m$, from the current rank through the largest rank.

The quantity $m p_{(j)}/j$ is the level at which the rank-$j$ raw p-value first meets its BH critical value: solving $p_{(j)}\le j\alpha/m$ for $\alpha$ gives $\alpha\ge m p_{(j)}/j$. However, a test at rank $i$ can be rejected because a later rank $j\ge i$ meets its critical value. Its adjusted value must therefore consider all such later ranks, not just its own rank-scaled quantity.

Why the Minimum Runs Backward

Define the rank-scaled quantity $a_j=m p_{(j)}/j$. The adjusted value at rank $i$ is the minimum of $a_i,a_{i+1},\ldots,a_m$, capped at one. It can be calculated efficiently by starting at the largest rank and moving toward rank 1: at each step, keep the smaller of the current $a_i$ and the adjusted value already computed for rank $i+1$.

\tilde p_{(m)}=\min(1,a_m),\qquad \tilde p_{(i)}=\min\{1,a_i,\tilde p_{(i+1)}\},\quad i=m-1,\ldots,1.

This is called a reverse cumulative minimum. The adjusted sequence must be nondecreasing with rank: $\tilde p_{(i)}\le\tilde p_{(i+1)}$. To see why, the set of ranks used in the minimum at $i+1$ is a subset of the set used at $i$. The minimum over the larger set cannot exceed the minimum over the subset. The recursion enforces this ordering even when the rank-scaled quantities $a_i$ themselves go up and down.

Worked Example: Computing the Reverse Cumulative Minimum

Suppose a hypothetical genomic analysis tests $m=5$ features in one family. Its ordered raw p-values are $0.010,\ 0.012,\ 0.040,\ 0.045,\ 0.200$. The ordering is nondecreasing, and all five p-values belong to the specified family.

First calculate $a_i=mp_{(i)}/i$:

(a_1,a_2,a_3,a_4,a_5) = \left( \frac{5(0.010)}{1}, \frac{5(0.012)}{2}, \frac{5(0.040)}{3}, \frac{5(0.045)}{4}, \frac{5(0.200)}{5} \right) = (0.0500,\ 0.0300,\ 0.0667,\ 0.0563,\ 0.200).

Now work from rank 5 back to rank 1. At rank 5, the adjusted value is $0.200$. At rank 4, take $\min(0.0563,0.200)=0.0563$. At rank 3, take $\min(0.0667,0.0563)=0.0563$. At rank 2, take $\min(0.0300,0.0563)=0.0300$. At rank 1, take $\min(0.0500,0.0300)=0.0300$. Thus the adjusted values in rank order are

(\tilde p_{(1)},\tilde p_{(2)},\tilde p_{(3)},\tilde p_{(4)},\tilde p_{(5)}) =(0.0300,\ 0.0300,\ 0.0563,\ 0.0563,\ 0.200).

The first adjusted value is $0.0300$, not $0.0500$, because the rank-2 quantity $0.0300$ is smaller and can determine the level at which rank 1 is rejected. The resulting sequence is nondecreasing, whereas the rank-scaled sequence is not.

Justification: Adjusted Values Reproduce BH Decisions

The defining property of an adjusted p-value is that it gives the smallest procedure level at which the test is rejected. For BH, that property follows directly from the step-up rejection rule.

Write the BH condition.
At level $\alpha$, rank $j$ meets its BH critical value exactly when $p_{(j)}\le j\alpha/m$, or equivalently $a_j=mp_{(j)}/j\le\alpha$.

Use the step-up rule.
Rank $i$ is rejected if and only if at least one rank $j\ge i$ meets its critical value. A qualifying rank $j$ makes the BH rejection rank $k$ at least $j$, so rank $i$ is among the rejected ranks.

Express the condition as an adjusted value.
There is a rank $j\ge i$ with $a_j\le\alpha$ exactly when $\min_{j\ge i}a_j\le\alpha$. Since $\alpha\le1$, this is equivalent to $\tilde p_{(i)}\le\alpha$.

Consequently, for any chosen $\alpha$, the BH procedure rejects the test at rank $i$ exactly when its BH-adjusted p-value is at most $\alpha$. This is a statement about the calculation and decision rule; it does not by itself establish an error-rate guarantee. The assumptions needed for such a guarantee are a separate issue.

Worked Example: Mapping Adjusted Values to Tests and Decisions

Consider a hypothetical family of four tests, labelled A, B, C, and D, with raw p-values A: $0.018$, B: $0.006$, C: $0.090$, and D: $0.030$. All four tests are included in the family, so $m=4$. Sorting gives B at rank 1, A at rank 2, D at rank 3, and C at rank 4.

The rank-scaled values are $4(0.006)/1=0.0240$, $4(0.018)/2=0.0360$, $4(0.030)/3=0.0400$, and $4(0.090)/4=0.0900$. These already increase with rank, so their reverse cumulative minima are unchanged. Mapping the results back to the original labels gives B: $0.0240$, A: $0.0360$, D: $0.0400$, and C: $0.0900$.

At $\alpha=0.04$, the adjusted values at or below $0.04$ belong to B, A, and D. To check directly, the ordered raw p-values are $0.006,\ 0.018,\ 0.030,\ 0.090$, and the BH critical values are $0.010,\ 0.020,\ 0.030,\ 0.040$. The first three ranks meet their critical values; rank 4 does not. The BH procedure therefore rejects B, A, and D, in agreement with the adjusted values.

Truncation at One and Practical Checks

An adjusted p-value is reported on the p-value scale, from 0 to 1, so the definition includes a cap at one. A rank-scaled quantity can exceed one: for example, with $m=2$ and ordered p-values $0.700$ and $0.900$, the quantities are $2(0.700)/1=1.400$ and $2(0.900)/2=0.900$. The reverse cumulative minima are $0.900$ at both ranks, and the adjusted values are therefore $0.900,\ 0.900$. In a complete family of valid raw p-values, the last rank-scaled quantity is $mp_{(m)}/m=p_{(m)}\le1$, so the reverse minimum is already at most one. The explicit cap is still part of the standard definition and protects the reported range.

Worked Example: Checking a Calculation With a Large Scaled Value

For a hypothetical family of $m=3$ tests, let the ordered raw p-values be $0.400,\ 0.800,\ 0.950$. The scaled values are

(a_1,a_2,a_3)= \left(\frac{3(0.400)}{1},\frac{3(0.800)}{2},\frac{3(0.950)}{3}\right) =(1.200,\ 1.200,\ 0.950).

Starting at rank 3 gives $\tilde p_{(3)}=0.950$. At rank 2, the reverse minimum is $\min(1.200,0.950)=0.950$; at rank 1 it is $\min(1.200,0.950)=0.950$. Thus all three adjusted values are $0.950$. Although the first two scaled values exceed one, they do not make the adjusted values exceed one or change the minimum here.

For a computational check, confirm that the raw p-values are sorted, that $m$ is the number of tests in the declared family, and that the rank denominator is the rank $i$. Then check that the adjusted values are nondecreasing in rank, lie between 0 and 1, and satisfy the BH decision rule at a selected level. A failure of monotonicity often signals that a forward cumulative minimum was used by mistake.

Interpretation, Comparisons, and Limitations

A BH-adjusted p-value is a threshold summary: it is the smallest BH significance level at which the corresponding test would be rejected, given the same family and its complete set of raw p-values. For example, an adjusted value of $0.030$ means that the test is rejected by BH at levels $\alpha\ge0.030$, and not rejected at levels below $0.030$. It does not mean there is a 3% probability that the null hypothesis is true, nor is it a confidence statement about a particular hypothesis.

The raw p-value describes an individual test before this adjustment. The BH-adjusted value instead incorporates the ranks and the number of tests in the family; changing the family can change the adjusted value even when that test’s raw p-value stays fixed. Unlike a single-step Bonferroni adjustment, the BH calculation uses rank-dependent factors and a reverse cumulative minimum. Holm’s procedure also uses ordered tests but targets familywise error control and has its own adjusted-value calculation; the methods should not be treated as interchangeable. In particular, do not infer an error-rate guarantee merely from the numerical calculation: the validity of BH’s false discovery rate control depends on conditions for the tests and their dependence.

A common mistake is to report $mp_{(i)}/i$ directly as the adjusted value. That can produce values that decrease with rank and fails to account for a later rank that permits an earlier test to be rejected. The correction is to calculate all rank-scaled quantities and take the minimum from rank $m$ backward. Another practical failure is to use only the tests that passed an initial filter as $m$ when the intended family includes additional tests. The family and its size must match the analysis question; omitted tests can change the adjustment.

Key takeaway: Sort the raw p-values, compute $mp_{(i)}/i$, take the reverse cumulative minimum, and cap at one. Then map the adjusted values back to the original tests. Within the same family, a test is rejected by BH at level $\alpha$ exactly when its adjusted p-value is at most $\alpha$.

Check Your Understanding

Work through each question, then compare with the solutions below.

For $m=3$ and ordered raw p-values $0.010,\ 0.030,\ 0.200$, calculate the rank-scaled values and BH-adjusted values.
For ordered raw p-values $0.020,\ 0.025,\ 0.080$ with $m=3$, explain why the rank-1 adjusted value need not equal $mp_{(1)}$.
If a test has BH-adjusted p-value $0.047$, what does that imply about its BH rejection decision at $\alpha=0.05$ and at $\alpha=0.04$?
Name two checks that can reveal an error in a computed list of BH-adjusted p-values.

Solutions. (1) The scaled values are $3(0.010)/1=0.0300$, $3(0.030)/2=0.0450$, and $3(0.200)/3=0.200$. They are already nondecreasing, so the adjusted values are $0.0300,\ 0.0450,\ 0.200$. (2) The rank-1 adjusted value is $\min\{3(0.020)/1,\ 3(0.025)/2,\ 3(0.080)/3\}=\min(0.0600,0.0375,0.0800)=0.0375$, because a later rank supplies a smaller threshold. (3) Since $0.047\le0.05$, BH rejects the test at level $0.05$; since $0.047>0.04$, BH does not reject it at level $0.04$. (4) Check that adjusted values are nondecreasing in rank and between 0 and 1; also verify that comparing them with a chosen $\alpha$ reproduces the BH step-up decisions.