Worked Holm Analysis After Selecting Two Arms

Why the Selection Step Matters

A multiarm trial may compare several experimental treatments with a control, then retain only two treatments for confirmatory evaluation. Applying Holm’s procedure to those two hypotheses is straightforward once their p-values are available. The difficult question is whether those p-values are valid after the selection process. Holm adjusts for testing multiple hypotheses; it does not automatically repair p-values made too small by choosing arms using the same evidence that generated them.

This tutorial uses a design in which a prespecified interim screening stage selects two treatments, and a separate confirmatory sample supplies the tests. Because the confirmatory data are independent of the screening data, selection does not favor treatments based on their confirmatory statistics. The selected family contains two hypotheses, so Holm’s procedure is applied with \(m=2\). If the same observations had been used both to select and test the arms, ordinary Holm adjustment on the selected p-values would not generally provide the stated error control.

Definition: For a family of \(m\) null hypotheses, Holm’s step-down procedure orders their valid p-values as \(p_{(1)}\le \cdots \le p_{(m)}\). At step \(i\), it compares \(p_{(i)}\) with \(\alpha/(m-i+1)\). It rejects in order until the first comparison that fails; that hypothesis and all hypotheses later in the order are not rejected.

Conditions for Applying Holm After Selection

A p-value for a null hypothesis is valid if, whenever that null is true, it is no more likely than a Uniform(0,1) random variable to be small: \(P(p\le t)\le t\) for every \(0\le t\le 1\). For a selected treatment, the relevant requirement is conditional validity: given the selection outcome, each true selected null must still have a valid p-value. Independent confirmatory data provide a standard way to meet this condition when the selection rule uses only the screening data and is fixed in advance.

The p-values may be dependent on one another. For example, treatment-control comparisons can share a control group and therefore have correlated statistics. Holm’s guarantee does not require independence among the p-values; it requires valid p-values for the hypotheses being tested. The statistical model or test used to produce each p-value must still be appropriate—for example, its variance estimate and reference distribution must support valid null inference.

The family must also be defined to match the claims. If the confirmatory claims concern only the two selected treatments, the Holm family has two hypotheses. If the intended claims concern all candidate treatments, adjusting only the two selected p-values is not automatically the right family-wide analysis. The family, selection rule, direction of testing, and significance level should be prespecified.

Worked Example: Two Treatments Selected Using Independent Screening Data

Suppose a hypothetical trial screens four candidate treatments, A, B, C, and D, using interim data. A prespecified rule retains the two with the largest interim estimates, which are A and B. An independent confirmatory sample is then used for the treatment-control tests. The endpoint is change in a clinical score, in which a larger treatment effect is beneficial. For illustration, assume the prespecified confirmatory analyses use two-sided Wald tests with valid standard errors and a standard normal reference distribution.

In the confirmatory sample, the estimated treatment effect for A is 5.0 score points with standard error 2.0 points. Its test statistic and two-sided p-value are

z_A=\frac{5.0}{2.0}=2.50,\qquad p_A=2\{1-\Phi(2.50)\}=0.0124.

For B, the estimated effect is 4.1 points with standard error 2.0 points:

z_B=\frac{4.1}{2.0}=2.05,\qquad p_B=2\{1-\Phi(2.05)\}=0.0404.

Both estimates are in score points, and each test evaluates its treatment against control. These tests rely on the stated Wald-test assumptions; the separate confirmatory sample makes their p-values valid conditional on which two treatments were selected. The comparisons may nevertheless be dependent if they share confirmatory control participants.

Order the p-values: \(p_{(1)}=0.0124\) for A and \(p_{(2)}=0.0404\) for B. At \(\alpha=0.05\), the first Holm threshold is \(0.05/2=0.025\). Since \(0.0124\le 0.025\), reject A’s null hypothesis. The second threshold is \(0.05/1=0.05\); since \(0.0404\le 0.05\), reject B’s null hypothesis as well.

The adjusted p-value for A is \(2(0.0124)=0.0248\). For B it is the larger of the first adjusted value and its own raw p-value, so it is \(0.0404\). Both adjusted p-values are at most 0.05. Thus, in this hypothetical example, both selected treatment hypotheses are rejected by Holm’s procedure at the 5% familywise level.

Adjusted P-Values and the Step-Down Calculation

For ordered p-values, Holm-adjusted p-values are the running maximum of the step-down multipliers, capped at 1. The running maximum is needed so that adjusted p-values remain ordered consistently with the original p-values.

p^{\mathrm{Holm}}_{(i)} = \min\left\{1,\ \max_{1\le j\le i}\left[(m-j+1)p_{(j)}\right]\right\}.

The adjusted value is then assigned back to the hypothesis associated with that ordered p-value. A hypothesis is rejected at level \(\alpha\) exactly when its Holm-adjusted p-value is at most \(\alpha\). For two hypotheses, the calculation simplifies to

p^{\mathrm{Holm}}_{(1)}=\min(1,2p_{(1)}),\qquad p^{\mathrm{Holm}}_{(2)}=\min\{1,\max(2p_{(1)},p_{(2)})\}.

For example, the second adjusted value is not necessarily twice the larger raw p-value. Its step-down multiplier is 1, but the running maximum ensures it cannot be smaller than the adjusted value of the first hypothesis.

Why the Procedure Controls Familywise Error

The familywise error rate (FWER) is the probability of rejecting at least one true null hypothesis in the family. Holm controls this probability at no more than \(\alpha\), under any configuration of true and false nulls, provided the individual p-values are valid. The argument also works conditional on the selection outcome when the selected hypotheses have conditionally valid p-values.

Theorem: If each true null hypothesis has a valid p-value, Holm’s step-down procedure has strong FWER control at level \(\alpha\): the probability of rejecting one or more true nulls is at most \(\alpha\), for any configuration of true and false null hypotheses. No independence assumption among the p-values is needed.

To see the justification, suppose \(m_0\) of the \(m\) null hypotheses are true. If Holm makes any false rejection, consider the first true null hypothesis in the ordered list that it rejects, at rank \(k\). The \(k-1\) earlier hypotheses must all be false nulls, so \(k-1\le m-m_0\). Therefore \(m-k+1\ge m_0\). Rejection at that rank requires

p_{(k)}\le \frac{\alpha}{m-k+1}\le \frac{\alpha}{m_0}.

Consequently, at least one of the \(m_0\) true-null p-values must be at most \(\alpha/m_0\). By the union bound and validity of each true-null p-value, the probability of that event is at most

\sum_{r=1}^{m_0}P\left(p_r\le \frac{\alpha}{m_0}\right) \le m_0\frac{\alpha}{m_0}=\alpha.

This proves strong control. The proof does not require the p-values to be independent. When the family consists of two selected hypotheses, the same argument applies conditional on selection if the confirmatory p-values remain valid under that conditioning.

Worked Example: The First Holm Step Fails

Suppose two independently selected treatments have valid confirmatory p-values 0.028 and 0.040. At \(\alpha=0.05\), the smaller p-value is tested first against \(0.05/2=0.025\). Because \(0.028>0.025\), the first comparison fails and the step-down procedure stops. Neither null hypothesis is rejected.

The ordered adjusted values are

p^{\mathrm{Holm}}_{(1)}=\min(1,2(0.028))=0.056, \qquad p^{\mathrm{Holm}}_{(2)}=\min(1,\max(0.056,0.040))=0.056.

Both adjusted p-values exceed 0.05, matching the step-down decisions. It would be incorrect to reject the hypothesis with raw p-value 0.040 by comparing it with 0.05 on its own: Holm’s stopping rule does not proceed after the first ordered test fails.

A Second Rejection Pattern

Holm can also reject the smaller p-value while leaving the larger one unrejected. This happens when the first p-value clears the stricter threshold but the second does not clear its threshold of \(\alpha\). The adjusted p-values communicate the same decisions and preserve the ordering.

Worked Example: Rejecting Only the Stronger Result

Suppose two selected treatment hypotheses have independent-confirmation p-values 0.006 and 0.34. The first p-value is compared with 0.025 and passes, since \(0.006\le0.025\). The second is compared with 0.05 and fails, since \(0.34>0.05\). Holm therefore rejects the first hypothesis but not the second.

The adjusted p-values are

p^{\mathrm{Holm}}_{(1)}=2(0.006)=0.012,\qquad p^{\mathrm{Holm}}_{(2)}=\max(0.012,0.34)=0.34.

At the 5% familywise level, the treatment associated with 0.006 has statistically significant evidence against its null hypothesis; the treatment associated with 0.34 does not. Neither conclusion establishes the probability that a null is true, and a failure to reject is not proof of no treatment effect.

Selection, Diagnostics, and Alternatives

Before reporting the results, check that the tested family and the selection mechanism match the analysis. Record which arms were eligible, how the two were selected, whether the confirmatory data were independent of screening, and whether all p-values come from the prespecified tests. Verify that p-values are valid for the intended null hypotheses and that the same \(\alpha\) and family are used when calculating adjusted p-values and decisions.

A key diagnostic is whether selection and testing reuse information. If the largest interim effects among four arms are selected and those same interim statistics are then treated as ordinary confirmatory p-values, the selected p-values tend to be unusually small even when all null hypotheses are true. Applying Holm with \(m=2\) to them does not, by itself, remove that selection bias. The analysis needs a design or inferential method that accounts for selection, such as independent validation data or an appropriately calibrated procedure.

Holm is a useful choice when valid p-values may be dependent and a simple, general strong-control procedure is desired. It is typically no more conservative than applying the ordinary Bonferroni threshold \(\alpha/m\) to every hypothesis, because Holm relaxes the threshold as it proceeds. Procedures that use the known correlation structure of treatment-control statistics can sometimes gain power, as covered by Dunnett and MaxT approaches. Such methods require their own assumptions and calibration; their potential efficiency does not make an unadjusted analysis valid.

Holm’s limitations are important in adaptive trials. It does not determine which treatments should be selected, correct biased effect estimates, or guarantee conditional validity when selection uses the testing data. Nor does a small adjusted p-value measure clinical importance. Report treatment-effect estimates with units and uncertainty alongside the raw and adjusted p-values, and distinguish statistical evidence from clinical relevance.

Common mistake: “Only two arms survived selection, so I can apply Holm to their p-values regardless of how they were chosen.” The correction is to check whether those p-values remain valid conditional on selection. Independent confirmatory data can support that condition; reusing selection data generally requires a selection-aware method rather than ordinary Holm adjustment alone.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

Two selected hypotheses have ordered p-values 0.018 and 0.047. At \(\alpha=0.05\), which hypotheses does Holm reject, and what are the adjusted p-values?
Why does Holm’s strong FWER guarantee not require independent p-values?
Four treatments are screened using interim data, and the two with the largest interim effects are tested using those same statistics. Why is applying Holm with \(m=2\) not automatically sufficient?
In the first worked example, why is the Holm family size two even though four treatments were screened?

Solutions: (1) The first comparison is \(0.018\le0.025\), so it passes; the second is \(0.047\le0.05\), so it also passes. The adjusted p-values are \(2(0.018)=0.036\) and \(\max(0.036,0.047)=0.047\), so both hypotheses are rejected. (2) The FWER proof uses validity of each true-null p-value and the union bound, not independence. (3) Choosing the largest interim effects tends to select unusually small p-values under the null, so the selected p-values may not be valid conditional on selection. (4) The confirmatory claims in that example concern only the two selected treatments, and independent confirmatory data make their p-values valid conditional on selection; claims about all four candidates would require an analysis calibrated for that broader family.