Calculating Holm-Adjusted P-Values

From Holm Decisions to Adjusted P-Values

In a multi-arm trial, the prespecified family might contain one treatment-versus-control hypothesis for each experimental arm. Holm’s procedure controls the familywise error rate by testing the ordered raw p-values against increasingly permissive thresholds. Adjusted p-values provide a complementary summary: for each hypothesis, they express how small the familywise significance level must be before Holm’s procedure rejects that hypothesis.

The calculation is not simply “multiply each raw p-value by the number of hypotheses.” Holm’s rank-specific multipliers must be followed by a cumulative maximum. That second operation ensures the adjusted values respect the ordering of the raw p-values. The calculation uses the Holm step-down procedure and its strong familywise error control, established in the earlier material on Holm’s procedure.

Definition: Let a prespecified family contain $m$ hypotheses, with raw p-values $p_1,\ldots,p_m$. Sort them in nondecreasing order, writing $p_{(1)}\le \cdots \le p_{(m)}$. The Holm-adjusted p-value at rank $i$ is $$ \widetilde p_{(i)}=\min\left\{1,\ \max_{1\le j\le i}\left[(m-j+1)p_{(j)}\right]\right\}. $$ After calculating these values in sorted order, assign each one to the hypothesis that had the corresponding raw p-value.

At rank $j$, the Holm test compares $p_{(j)}$ with $\alpha/(m-j+1)$. The first rank uses the most stringent threshold, $\alpha/m$; each later rank uses a larger threshold. In the adjusted-value calculation, the corresponding multiplier is $m-j+1$. The maximum over all ranks up to $i$ matters because a hypothesis at rank $i$ is rejected only if every earlier step through $i$ passes.

Assume that the family is specified before the results are examined and that each raw p-value is valid for its own null hypothesis under the trial’s analysis assumptions. Holm’s calculation does not require independent p-values. Its multiplicity interpretation is for the stated family; changing which hypotheses belong to that family changes $m$ and can change every adjusted value.

Why the Cumulative Maximum Is Needed

For a fixed rank $i$, passing all Holm comparisons through that rank requires $p_{(j)}\le \alpha/(m-j+1)$ for every $j\le i$. Rearranging each inequality gives $\alpha\ge (m-j+1)p_{(j)}$. Thus the smallest level meeting all those requirements is the largest of the rank-specific products:

\alpha\ge (m-j+1)p_{(j)}\text{ for all }j\le i \quad\Longleftrightarrow\quad \alpha\ge \max_{1\le j\le i}\left[(m-j+1)p_{(j)}\right].

This explains the cumulative maximum. The raw products alone need not increase with rank, even though adjusted p-values must be nondecreasing in sorted order. Finally, values above 1 are reported as 1 because p-values are bounded by 1. For conventional significance levels $0<\alpha<1$, Holm rejects a hypothesis exactly when its adjusted p-value is at most $\alpha$. A reported adjusted value of 1 therefore does not lead to rejection at any such level.

Worked Example: Ordering and Restoring Three Treatment Comparisons

Suppose a hypothetical trial prespecified three comparisons with control. The raw p-values are 0.018 for arm A, 0.003 for arm B, and 0.041 for arm C. Calculate Holm-adjusted p-values and decide which hypotheses Holm rejects at familywise level $\alpha=0.05$.

There are $m=3$ hypotheses. In increasing order, the raw p-values are 0.003 for B, 0.018 for A, and 0.041 for C. The rank-specific products are $3(0.003)=0.009$, $2(0.018)=0.036$, and $1(0.041)=0.041$. Their cumulative maxima are 0.009, 0.036, and 0.041. None exceeds 1, so these are the adjusted values in ranked order.

Restoring the original labels gives 0.036 for A, 0.009 for B, and 0.041 for C. At $\alpha=0.05$, all three adjusted values are at most 0.05, so Holm rejects all three null hypotheses. This conclusion is also visible from the step-down comparisons: 0.003 is at most $0.05/3$, 0.018 is at most $0.05/2$, and 0.041 is at most 0.05.

The adjusted value 0.036 for arm A means that Holm would reject its null hypothesis at familywise levels of 0.036 or greater, provided the level is below 1 and the same family and p-values are used. It is not a probability that the null hypothesis for arm A is true.

Worked Calculations and Important Edge Cases

Worked Example: Why the Running Maximum Cannot Be Skipped

A hypothetical four-arm comparison has sorted raw p-values 0.012, 0.013, 0.014, and 0.70. The hypotheses, in that order, are D, A, C, and B. Calculate the adjusted values.

The rank-specific products are $4(0.012)=0.048$, $3(0.013)=0.039$, $2(0.014)=0.028$, and $1(0.70)=0.70$. The products at ranks 2 and 3 are smaller than the rank 1 product. Taking cumulative maxima gives 0.048, 0.048, 0.048, and 0.70. Therefore, the adjusted values are 0.048 for D, 0.048 for A, 0.048 for C, and 0.70 for B.

At $\alpha=0.05$, Holm rejects D, A, and C, but not B. Using the uncorrected products would give adjusted values that decrease across the first three ranks, which is not the Holm adjustment and would misrepresent the step-down decision rule.

Worked Example: Tied Raw Values and Truncation at One

Consider a hypothetical family of three hypotheses with sorted raw p-values 0.40, 0.40, and 0.90. The first two values belong to arms E and F, and the third to arm G. Find the adjusted values and interpret them at $\alpha=0.05$.

With $m=3$, the raw products are $3(0.40)=1.20$, $2(0.40)=0.80$, and $1(0.90)=0.90$. The cumulative maxima before truncation are 1.20, 1.20, and 1.20. Applying the cap at 1 gives adjusted values of 1 for E, F, and G.

No null hypothesis is rejected at level 0.05. The tied raw values can be assigned either order because they are equal; the rank-specific products differ, but the cumulative maximum gives both tied hypotheses the same adjusted value here. The cap does not mean the raw evidence has become stronger: it records that the calculated adjustment exceeds the possible p-value range.

Practical Calculation and Checks

A reliable hand calculation follows the ordered raw values through to the original hypothesis labels. Keep the rank attached to each hypothesis throughout; do not sort the adjusted values and then assign them to arms by memory. A short table can make the operations explicit:

Rank	Hypothesis	Raw p-value	Multiplier	Rank-specific product	Adjusted p-value
$i$	Label at rank $i$	$p_{(i)}$	$m-i+1$	$(m-i+1)p_{(i)}$	Cumulative maximum, capped at 1

Before interpreting the result, check that the family size is the number of hypotheses in the prespecified family, not merely the number of small p-values. Confirm that raw p-values are sorted from smallest to largest, that the multiplier decreases by one at each rank, and that the cumulative maximum is used. Adjusted values in ranked order should be nondecreasing and between 0 and 1. If either check fails, revisit the ordering, multiplication, maximum, or cap.

Ties require no special formula. For equal raw p-values, use the ranks they occupy in the sorted list and apply the same cumulative-maximum rule. The tied adjusted values will be equal when the tie occupies consecutive ranks, because the running maximum includes the earlier, larger multiplier product. Preserve the original hypothesis labels so that the results remain interpretable in the trial context.

How to Interpret and Compare Adjusted Values

An adjusted p-value is a summary of the Holm procedure applied to a particular family. Compare each adjusted value with the chosen familywise level: reject its null hypothesis if the adjusted value is at most $\alpha$, and otherwise do not reject it. This is a decision rule, not a statement about the probability that a null hypothesis is true. Nor does an adjusted p-value describe the probability of observing the data under a compound event involving the whole family.

The main simple alternative is the Bonferroni adjustment, which multiplies every raw p-value by $m$ and caps at 1. Holm uses rank-specific multipliers and the cumulative maximum instead. For each rank, the Holm-adjusted value is no larger than the corresponding Bonferroni-adjusted value: for every $j\le i$, $p_{(j)}\le p_{(i)}$ and $m-j+1\le m$, so $(m-j+1)p_{(j)}\le mp_{(i)}$. Taking the maximum over $j\le i$, and then capping at 1, preserves this inequality. Consequently, Holm can reject at least as many hypotheses as Bonferroni for the same family and level, while retaining familywise error control under arbitrary dependence when the raw p-values are valid.

Adjusted p-values do not repair invalid raw p-values, an incomplete or changed family, or analyses chosen after inspecting results. They also do not by themselves report treatment-effect size, clinical importance, or uncertainty in an effect estimate. Report them alongside the raw p-values and relevant effect estimates with their units and confidence intervals, and state the family and decision level clearly.

Common Mistakes

Multiplying every raw p-value by $m$. That is the Bonferroni adjustment, not the Holm calculation. Use $m-i+1$ at rank $i$.
Stopping after the rank-specific products. Take the cumulative maximum so the adjusted values are nondecreasing in sorted order.
Forgetting to restore hypothesis labels. The calculation is ordered, but the reported result must identify the original treatment comparison.
Changing the family after seeing the results. The family size and membership affect the adjustment and should be prespecified.
Calling an adjusted p-value the probability that a null is true. It is a procedure-specific quantity used for a familywise decision.

Key takeaway: Sort the $m$ raw p-values, multiply rank $i$ by $m-i+1$, take the cumulative maximum, cap at 1, and return the results to their original hypotheses. At a conventional familywise level $0<\alpha<1$, reject exactly those hypotheses whose Holm-adjusted p-values are at most $\alpha$.

Check Your Understanding

Answer each question, then compare your work with the solutions below.

A prespecified family has three raw p-values, in sorted order, 0.01, 0.04, and 0.20. Calculate the Holm-adjusted values and identify rejections at $\alpha=0.05$.
For sorted raw p-values 0.02, 0.025, and 0.60 in a family of three, why is it incorrect to report the rank-specific products as the adjusted values?
If two hypotheses have tied raw p-values, what determines the multiplier used for each one?
What does it mean in context if a hypothesis has Holm-adjusted p-value 0.03 and the familywise level is 0.05?

Solutions. (1) The products are $3(0.01)=0.03$, $2(0.04)=0.08$, and $1(0.20)=0.20$. The cumulative maxima are 0.03, 0.08, and 0.20, so only the first hypothesis is rejected at 0.05. (2) The products are 0.06, 0.05, and 0.60; they decrease from rank 1 to rank 2. The cumulative maxima, and hence the adjusted values, are 0.06, 0.06, and 0.60. (3) Each tied value occupies a rank in the ordered list, and its multiplier is $m-i+1$ for that rank. The cumulative-maximum calculation handles the tie without a special rule. (4) Holm rejects that null hypothesis at familywise level 0.05, because 0.03 is at most 0.05. This does not mean there is a 3% probability that the null hypothesis is true.