From Ordered P-Values to Holm Decisions
Holm’s procedure turns the p-values from a family of tests into a set of rejection decisions while controlling the familywise error rate. In genomic analyses, a family might consist of tests for several genes or variants that address one defined scientific question. This tutorial focuses on carrying out the calculation: sorting the p-values, finding the first non-rejection, and obtaining an adjusted p-value for every test.
Suppose the family contains \(m\) hypotheses, with valid p-values \(p_1,\ldots,p_m\), and the chosen familywise significance level is \(\alpha\). A p-value is valid for a true null hypothesis if, for every \(u\) between 0 and 1, \(P(p_i\leq u)\leq u\). The tests must be treated as one specified family. Holm’s procedure does not require the test statistics to be independent.
The denominators decrease as the rank increases, so the thresholds become less stringent about the number of tests remaining only in the sense that the threshold increases: the first is \(\alpha/m\), and the last is \(\alpha\). The stopping rule is essential. A later small p-value cannot restart the sequence after a failure; in sorted order, later p-values are at least as large, while their thresholds are also larger, so the formal rule is to stop at the first failure.
Worked Holm Calculations
Worked Example: A Mixed Set of Rejections
Suppose five hypothetical gene-level tests have ordered p-values \(0.001,\ 0.012,\ 0.018,\ 0.041,\ 0.200\). Use \(\alpha=0.05\). The tests are assumed to belong to the same prespecified family, and their p-values are valid under their respective null hypotheses.
The rank-specific thresholds are \(0.05/5=0.0100\), \(0.05/4=0.0125\), \(0.05/3\approx0.01667\), \(0.05/2=0.0250\), and \(0.05/1=0.0500\). At rank 1, \(0.001\leq0.0100\), so reject. At rank 2, \(0.012\leq0.0125\), so reject. At rank 3, \(0.018>0.01667\), so stop. Ranks 3, 4, and 5 are not rejected.
To calculate the adjusted p-values, multiply each ordered p-value by its rank’s denominator and take the cumulative maximum. The products are \(5(0.001)=0.005\), \(4(0.012)=0.048\), \(3(0.018)=0.054\), \(2(0.041)=0.082\), and \(1(0.200)=0.200\). These products are already nondecreasing, so the adjusted p-values are \(0.005,\ 0.048,\ 0.054,\ 0.082,\ 0.200\). Comparing each with \(0.05\) gives the same decisions: the first two tests are rejected.
For example, the adjusted p-value \(0.054\) for the rank-3 test means that the smallest familywise significance level at which Holm would reject that test, for this fixed set of p-values, is \(0.054\). It does not mean there is a 5.4% probability that its null hypothesis is true.
Worked Example: Tied P-Values at the Decision Boundary
Consider six hypothetical tests with ordered p-values \(0.002,\ 0.010,\ 0.010,\ 0.030,\ 0.070,\ 0.200\), and let \(\alpha=0.05\). Assume again that the tests form the intended family and the p-values are valid under true null hypotheses.
The thresholds by rank are \(0.008333,\ 0.0100,\ 0.0125,\ 0.01667,\ 0.0250,\ 0.0500\). The first p-value passes because \(0.002\leq0.008333\). The second passes exactly at its threshold: \(0.010=0.0100\). The third passes because \(0.010\leq0.0125\). The fourth fails because \(0.030>0.01667\), so the procedure stops. The first three tests are rejected; the last three are not.
The denominator-weighted products are \(6(0.002)=0.012\), \(5(0.010)=0.050\), \(4(0.010)=0.040\), \(3(0.030)=0.090\), \(2(0.070)=0.140\), and \(1(0.200)=0.200\). Taking cumulative maxima gives adjusted p-values \(0.012,\ 0.050,\ 0.050,\ 0.090,\ 0.140,\ 0.200\). The tied tests have the same adjusted p-value, \(0.050\), and both are rejected at \(\alpha=0.05\) because equality counts as passing.
A tie can be ordered either way because the tied p-values are equal. Their rank-specific products decrease across the tied ranks, so the cumulative maximum does not decrease; both tied tests receive the same adjusted value and decision.
Why the Adjusted P-Value Formula Works
The formula follows directly from asking how large \(\alpha\) would need to be for Holm to reject a test at rank \(i\). To reach rank \(i\), the procedure must pass every comparison from rank 1 through rank \(i\). At each such rank \(j\), passing requires \(p_{(j)}\leq\alpha/(m-j+1)\), or equivalently \(\alpha\geq(m-j+1)p_{(j)}\). Thus \(\alpha\) must be at least as large as every one of these products.
Since a significance level cannot exceed 1, the Holm-adjusted p-value at rank \(i\) is
This is a cumulative maximum: calculate the product at each rank, keep the largest product seen so far, and cap the result at 1. The cumulative maximum makes adjusted p-values nondecreasing in sorted order. At any chosen level \(\alpha\), reject exactly those hypotheses whose Holm-adjusted p-values are at most \(\alpha\). This agrees with the step-down rule: if a rank fails, its adjusted value exceeds \(\alpha\), and all later cumulative maxima are at least as large.
The sorted calculation must ultimately be mapped back to the original hypotheses. If the rank-1 p-value belongs to one variant and rank 4 belongs to another, report each decision and adjusted value with the corresponding variant, not merely by rank.
Worked Example: Ties After a Non-Rejection
Four hypothetical tests have ordered p-values \(0.009,\ 0.021,\ 0.021,\ 0.500\). Let \(\alpha=0.05\). Assume the p-values are valid and the four tests define the family.
The thresholds are \(0.0125,\ 0.01667,\ 0.0250,\ 0.0500\). Rank 1 passes because \(0.009\leq0.0125\). Rank 2 fails because \(0.021>0.01667\), so only rank 1 is rejected. Although rank 3’s tied value is below its own threshold, \(0.021\leq0.0250\), the procedure has already stopped and rank 3 is not rejected.
The products are \(4(0.009)=0.036\), \(3(0.021)=0.063\), \(2(0.021)=0.042\), and \(1(0.500)=0.500\). Their cumulative maxima are \(0.036,\ 0.063,\ 0.063,\ 0.500\), so these are the adjusted p-values. The tied tests share \(0.063\), which exceeds \(0.05\), and neither is rejected.
Checks, Comparisons, and Limitations
Several checks help catch arithmetic or bookkeeping errors. First, the ordered raw p-values must be nondecreasing. Second, the Holm thresholds must be \( \alpha/m,\alpha/(m-1),\ldots,\alpha\). Third, adjusted p-values in sorted order must be nondecreasing and lie between 0 and 1. Finally, the decision set must be an initial segment of the ordered tests: once a rank is not rejected, no later rank is rejected. If these checks fail, revisit the sorting, denominators, cumulative maximum, or stopping rule.
A common mistake is to calculate each adjusted value as \((m-i+1)p_{(i)}\) alone. Those raw products need not be nondecreasing. For the four-test example above, the rank-2 and rank-3 products are \(0.063\) and \(0.042\). Reporting \(0.042\) as the rank-3 adjusted p-value would incorrectly make it smaller than the rank-2 value and could suggest rejection at \(0.05\). The correction is to use the cumulative maximum: rank 3 receives \(\max(0.036,0.063,0.042)=0.063\).
Holm differs operationally from the single-step Bonferroni rule. Bonferroni compares every raw p-value with the same threshold \(\alpha/m\); Holm compares ordered p-values with a sequence of thresholds and stops at the first failure. Both require a correctly defined family and valid p-values. Holm’s adjusted p-values summarize its own sequential decisions; they are not the unadjusted p-values and should not be interpreted as posterior probabilities for hypotheses.
The calculation cannot repair invalid p-values, a family chosen after inspecting results, or a mismatch between the stated family and the scientific question. It also does not determine whether an association is biologically meaningful: a rejection is evidence against a null hypothesis under the test’s assumptions, not proof of a causal or clinically important effect. Ties caused by rounding deserve particular care; when possible, sort and calculate using unrounded p-values, because rounding can alter the apparent order or equality near a threshold.
Check Your Understanding
Use the Holm rule and adjusted p-value formula. Assume each set is a prespecified family of valid p-values, and use \(\alpha=0.05\) when a decision is requested.
- For three ordered p-values \(0.004,\ 0.030,\ 0.080\), calculate the thresholds, decisions, and adjusted p-values.
- For five ordered p-values \(0.010,\ 0.010,\ 0.025,\ 0.060,\ 0.300\), identify the rejection set and calculate all adjusted p-values.
- Four tests have labels and p-values \(G4:0.040,\ G1:0.003,\ G3:0.040,\ G2:0.015\). Calculate the Holm decisions and adjusted values, then report them by label.
- A student calculates the rank-3 adjusted p-value for ordered values \(0.005,\ 0.020,\ 0.020,\ 0.900\) as \(2(0.020)=0.040\). What is wrong, and what is the correct rank-3 adjusted value?
Solutions. (1) The thresholds are \(0.05/3=0.01667,\ 0.05/2=0.0250,\ 0.05\). Rank 1 passes and rank 2 fails, since \(0.030>0.0250\), so only rank 1 is rejected. The products are \(3(0.004)=0.012,\ 2(0.030)=0.060,\ 1(0.080)=0.080\); their cumulative maxima, the adjusted values, are \(0.012,\ 0.060,\ 0.080\).
(2) The thresholds are \(0.0100,\ 0.0125,\ 0.01667,\ 0.0250,\ 0.0500\). The first two values pass, including equality at rank 1; rank 3 fails because \(0.025>0.01667\). The first two tests are rejected. The products are \(0.050,\ 0.050,\ 0.075,\ 0.120,\ 0.300\), already nondecreasing, so the adjusted values are \(0.050,\ 0.050,\ 0.075,\ 0.120,\ 0.300\).
(3) In sorted order the labels and p-values are \(G1:0.003,\ G2:0.015,\ G3:0.040,\ G4:0.040\). The thresholds are \(0.0125,\ 0.01667,\ 0.0250,\ 0.0500\); ranks 1 and 2 pass, and rank 3 fails. The products are \(0.012,\ 0.045,\ 0.080,\ 0.040\), giving cumulative maxima \(0.012,\ 0.045,\ 0.080,\ 0.080\). By label: \(G1\) is rejected with adjusted p-value \(0.012\); \(G2\) is rejected with \(0.045\); \(G3\) is not rejected with \(0.080\); and \(G4\) is not rejected with \(0.080\).
(4) The products at ranks 1 through 3 are \(4(0.005)=0.020,\ 3(0.020)=0.060,\ 2(0.020)=0.040\). Adjusted values use a cumulative maximum, not the rank-3 product alone. The correct rank-3 value is \(\max(0.020,0.060,0.040)=0.060\).