Holm Step-Down Testing After Multiarm Selection

Why Use a Step-Down Procedure After Selection?

A multiarm trial may retain only a subset of candidate treatments for confirmatory testing. If several retained arms are tested, declaring any one of them effective creates a familywise error risk: under a configuration where some tested treatments are truly ineffective, the probability of making at least one false efficacy claim should be controlled. Holm’s step-down procedure gives a simple way to test a family of hypotheses while controlling that risk, and it can be less conservative than applying the same Bonferroni cutoff to every hypothesis.

The qualification “after selection” matters. Holm’s guarantee depends on the validity of the p-values being tested. If the same noisy efficacy results are used both to choose the most promising arms and to calculate their ordinary p-values, selected p-values can be too small even when the null hypotheses are true. Holm’s ordering does not repair p-values made invalid by selection. The analysis must account for how the arms were selected.

Definition: For \(m\) selected treatment hypotheses \(H_1,\ldots,H_m\), let \(p_1,\ldots,p_m\) be their p-values, and write \(p_{(1)}\le\cdots\le p_{(m)}\) for the ordered values. Holm’s step-down procedure at level \(\alpha\) compares \(p_{(i)}\), in order, with \(\alpha/(m-i+1)\). Reject consecutively from the smallest p-value until the first comparison that fails; do not reject that hypothesis or any hypothesis after it.

Here, a p-value is calculated for a prespecified treatment hypothesis, such as no benefit relative to control, using the appropriate direction of benefit and a test statistic whose null distribution is valid under that hypothesis. For a one-sided efficacy test, smaller p-values provide stronger evidence against the null. The procedure does not require the p-values to be independent; it does require each true-null p-value to be valid under the conditions of the analysis.

Applying Holm to Selected Arms

Suppose the selected family contains \(m\) hypotheses and the target familywise error rate is \(\alpha\). First sort the p-values from smallest to largest. The first comparison uses \(\alpha/m\), the next uses \(\alpha/(m-1)\), and the final comparison uses \(\alpha\). A failure ends the procedure: a later p-value may be numerically small, but it cannot be rejected after an earlier ordered p-value has failed its stricter threshold.

Worked Example: Applying the Step-Down Thresholds

Suppose four selected experimental arms are tested against their corresponding null hypotheses at familywise level \(\alpha=0.05\). Assume the four p-values are valid for these selected hypotheses, for example because the selection rule was independent of the confirmatory data used to calculate them. The ordered p-values are \(0.006,\ 0.019,\ 0.028,\ 0.046\).

The hypotheses and p-values are matched by their order after sorting; the ordered arms are not necessarily the arms’ original labels. The thresholds are

\frac{0.05}{4}=0.0125,\qquad \frac{0.05}{3}=0.01667,\qquad \frac{0.05}{2}=0.025,\qquad \frac{0.05}{1}=0.05.

At the first step, \(0.006\le 0.0125\), so reject the hypothesis associated with the smallest p-value. At the second step, \(0.019>0.01667\), so the second hypothesis does not pass. Stop there: reject neither the second nor either of the remaining hypotheses. Thus only the arm associated with \(p_{(1)}=0.006\) is declared effective at the specified familywise level. The fact that \(0.028\) and \(0.046\) are each below \(0.05\) does not permit their rejection after the step-down procedure has stopped.

The result depends on the ordered sequence, not on separate unadjusted comparisons with \(\alpha\). Always report the number of hypotheses in the tested family, the p-value ordering, the threshold at the first failed step, and which hypotheses were rejected. This makes it possible to check that the stopping rule has been followed.

When Does Holm Control Familywise Error?

For a fixed family of \(m\) hypotheses, Holm controls the familywise error rate strongly at level \(\alpha\): the probability of rejecting at least one true null hypothesis is at most \(\alpha\), for any configuration of true and false null hypotheses. The result allows arbitrary dependence among valid p-values. To use it after selection, the validity condition must still hold for the selected family. A sufficient condition is that, conditional on the selection outcome, every selected true-null p-value is super-uniform: for every \(u\) between 0 and 1,

P(p_i\le u\mid S)\le u

for each selected true-null hypothesis \(H_i\), where \(S\) denotes the selection outcome. This condition can hold when selection uses independent interim data and the confirmatory p-values are computed from independent data, or when valid selection-adjusted p-values are used. It need not hold when the same efficacy observations drive both selection and testing.

Why the Step-Down Rule Controls Error

A short argument explains the strong familywise guarantee. Consider a fixed selected family with \(m_0\ge1\) true null hypotheses. Imagine the first true null hypothesis encountered in the ordered p-values. If any true null is rejected, this first true null must have passed its step-down threshold. At its position \(i\), at least the \(m_0\) true-null p-values remain among the \(m-i+1\) hypotheses still under consideration. Consequently, \(m-i+1\ge m_0\), so the threshold at that position is no larger than \(\alpha/m_0\).

\frac{\alpha}{m-i+1}\le \frac{\alpha}{m_0}.

Therefore, if Holm rejects any true null, at least one of the \(m_0\) true-null p-values must be at most \(\alpha/m_0\). By the union bound and validity of each true-null p-value,

P(\text{at least one true null is rejected}) \le \sum_{i:\,H_i\ \mathrm{true}}P\left(p_i\le\frac{\alpha}{m_0}\right) \le m_0\frac{\alpha}{m_0} =\alpha.

This proves strong control without assuming independent test statistics. If the selection outcome is random but the p-values are super-uniform conditional on each possible outcome, the same argument applies conditional on that outcome; averaging the conditional error probabilities also gives an unconditional error probability no greater than \(\alpha\).

Selection Can Break Ordinary P-Value Validity

The conditional validity requirement is not a technical detail that can be ignored once p-values are sorted. If arms are selected because their observed efficacy statistics look unusually favorable, the selected statistics are likely to overstate evidence, even when all treatments are ineffective. Testing the selected winner as if it had been specified in advance can inflate error. Holm accounts for multiplicity among the p-values it receives; it does not, by itself, account for the process that produced those p-values.

Worked Example: Selecting the Smallest of Four Null P-Values

Consider four candidate arms whose null hypotheses are all true. Suppose, for illustration, that their four p-values are independent and uniformly distributed between 0 and 1 under their null hypotheses. A selection rule chooses the arm with the smallest p-value, and the analyst then treats it as the only selected hypothesis. Holm with \(m=1\) rejects that hypothesis whenever its p-value is at most \(0.05\).

The selected p-value is the minimum of the four values. The probability it is at most \(0.05\) is the probability that at least one of the four p-values is at most \(0.05\):

P\left(\min(p_1,p_2,p_3,p_4)\le 0.05\right) =1-P(p_1>0.05,\ldots,p_4>0.05) =1-(0.95)^4 =0.18549375.

Thus this procedure falsely declares an ineffective arm effective with probability about \(18.55\%\), not \(5\%\). The ordinary p-value for the selected winner is not uniform conditional on being selected as the minimum. Using Holm with one selected hypothesis does not correct this selection effect. The example assumes independence to make the calculation explicit; dependence changes the numerical probability, but selection can still invalidate ordinary p-values.

A valid design needs a defensible basis for the p-values used in the selected family. One option is to select using interim data and test using independent confirmatory data, with the testing plan and hypotheses specified appropriately. Another is to use p-values whose distributions account for the selection rule. A conservative alternative, when valid p-values for all candidate arms are available, is to apply a multiplicity procedure to the full prespecified candidate family and make claims only for selected arms that are rejected. Selection cannot then add a rejection that the full-family test did not allow.

Worked Example: Testing the Full Candidate Family

Suppose four candidate arms have valid confirmatory p-values \(0.008,\ 0.019,\ 0.044,\ 0.20\). An independent interim rule selected the first and third arms for further consideration. To retain protection without relying on a selected-subset analysis, apply Holm to all four prespecified candidate hypotheses at \(\alpha=0.05\).

The ordered p-values are \(0.008,\ 0.019,\ 0.044,\ 0.20\), with thresholds \(0.0125,\ 0.01667,\ 0.025,\ 0.05\). The first comparison passes because \(0.008\le0.0125\). The second fails because \(0.019>0.01667\), so testing stops. Only the hypothesis associated with \(0.008\) is rejected. If that arm was among those selected, it can be declared effective; the selected arm with p-value \(0.044\) cannot be declared effective through this analysis. The full-family procedure may sacrifice power, but selection of a subset does not invalidate its familywise guarantee.

Holm Compared With Bonferroni

Bonferroni testing rejects a hypothesis only when its p-value is at most \(\alpha/m\). Holm begins with that same threshold for the smallest p-value, but relaxes the threshold as hypotheses are rejected. It therefore rejects at least the hypotheses Bonferroni would reject when both procedures use the same valid p-values and family. For example, with four tests at \(\alpha=0.05\), Bonferroni uses \(0.0125\) for each p-value, whereas Holm uses \(0.0125,\ 0.01667,\ 0.025,\ 0.05\) in rank order, subject to its stopping rule. Holm can gain power while retaining strong familywise error control under arbitrary dependence.

Neither procedure automatically corrects invalid p-values caused by selecting arms on the same data. Holm is also not a model for the correlation structure among treatment-versus-control contrasts; procedures tailored to shared-control comparisons are a distinct approach. The relevant choice depends on the available valid inferential quantities and the prespecified analysis plan, not simply on which method gives smaller thresholds.

Diagnostics and Common Mistakes

Applying Holm only to winners’ ordinary p-values. This is not justified merely because the step-down algorithm is multiplicity-adjusted. Establish that the p-values are valid after selection, or use a procedure that accounts for the selection or tests the full candidate family.
Continuing after a failed comparison. Holm is step-down: after the first failure, later hypotheses are not rejected, even if their p-values are below \(\alpha\).
Using the wrong family size. The denominator begins at the number of hypotheses in the family actually being tested. Define that family in advance and explain whether it is the selected family or the full candidate family.
Assuming independence is required. The standard Holm guarantee permits dependence, provided the true-null p-values are valid. Conversely, independence of test statistics alone does not fix p-values made selective by data-dependent screening.
Calling a non-rejected arm ineffective. Failure to reject means the evidence did not meet the familywise testing rule; it does not establish that the treatment has no benefit.

Key takeaway: Holm orders the p-values, tests against increasingly permissive thresholds, and stops at the first failure. Its strong familywise error guarantee applies only when the p-values are valid for the family being tested—including after any arm-selection step.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Three valid p-values are \(0.010,\ 0.022,\ 0.040\). Apply Holm at \(\alpha=0.05\). Which hypotheses are rejected?
Four selected hypotheses have ordered p-values \(0.004,\ 0.018,\ 0.031,\ 0.049\). Apply Holm at \(\alpha=0.05\). Where does testing stop?
Why does applying Holm to the single smallest p-value after choosing the winner from four arms not necessarily control error at \(0.05\)?
Name one design or analysis condition that can make selected-arm p-values suitable for Holm testing.

Solutions. (1) The thresholds are \(0.05/3=0.01667,\ 0.05/2=0.025,\ 0.05\). Each comparison passes: \(0.010\le0.01667\), \(0.022\le0.025\), and \(0.040\le0.05\). Reject all three hypotheses. (2) The thresholds are \(0.0125,\ 0.01667,\ 0.025,\ 0.05\). The first passes because \(0.004\le0.0125\); the second fails because \(0.018>0.01667\). Stop there and reject only the first hypothesis. (3) Choosing the smallest p-value makes it unusually small even under the global null; with four independent uniform null p-values, the chance the minimum is at most \(0.05\) is \(1-(0.95)^4=0.18549375\), or about \(18.55\%\). (4) For example, use interim data for selection and independent confirmatory data for testing, or use valid p-values adjusted for the selection rule. In either case, the true-null p-values must remain valid for the selected family.