Pairwise Contrasts When Several Arms Survive Selection

Why Compare Surviving Treatments Directly?

When several experimental arms survive an interim selection rule, a trial may need to answer a different question from whether each arm beats control: which surviving treatments differ from one another? These are treatment-versus-treatment contrasts. For example, a trial might compare three surviving doses on change in a symptom score. The comparisons concern differences between active treatments, not differences between each treatment and a shared control.

A pairwise analysis has two linked decisions. First, define the set of comparisons that belongs to the family—for example, every pair of surviving arms, or only clinically specified pairs. Second, use a testing procedure that controls the chance of one or more false rejections across that family. Because arms survive a data-dependent selection process, it is also necessary to establish that the p-values supplied to the multiplicity procedure are valid after selection.

The sign has a direct interpretation only after the outcome scale is specified. If larger symptom improvement is better, a positive \(\hat{\theta}_{ij}\) favors treatment \(i\); if the outcome is symptom burden, where lower values are better, a negative contrast may favor \(i\). The estimand, scale, and direction should therefore be stated before interpreting a result.

Contrasts, Standard Errors, and the Comparison Family

For independent treatment groups, the estimated difference has variance equal to the sum of the variances of the two estimated means. More generally, any covariance induced by the design or analysis must be included:

A test statistic is the estimated contrast divided by its standard error. With a known standard error and a normal sampling distribution, \(Z_{ij}=\hat{\theta}_{ij}/SE(\hat{\theta}_{ij})\) has a standard normal distribution under \(H_{0,ij}\). With estimated variances, a suitable \(t\)- or large-sample Wald procedure may be used under its corresponding assumptions. For example, the conventional pooled two-sample \(t\)-test assumes independent observations, normally distributed outcomes within each group, and a common variance in the two groups. A model-based or robust standard error requires its own assumptions and should not be treated as interchangeable with the pooled standard error.

With \(k\) surviving arms, testing every unordered pair creates \(m=k(k-1)/2\) hypotheses. A protocol may instead specify a smaller family, such as comparisons among selected doses only. The family must be defined explicitly: changing it after seeing which contrasts look promising changes the multiplicity problem. Pairwise statistics are generally dependent because several contrasts use the same arm mean. For independent arm means, for instance, \(\hat{\theta}_{ij}\) and \(\hat{\theta}_{i\ell}\) both contain \(\bar{x}_i\), so their covariance is not zero. This dependence does not prevent Holm’s procedure from controlling familywise error when its input p-values are valid.

Holm Adjustment for the Specified Pairwise Comparisons

Holm’s step-down procedure applies to a specified family of \(m\) hypotheses and is valid under arbitrary dependence among their p-values, provided each p-value for a true null hypothesis is valid under the inference conditions in use. Order the p-values from smallest to largest, \(p_{(1)}\le \cdots \le p_{(m)}\). Starting with the smallest, compare \(p_{(r)}\) with \(\alpha/(m-r+1)\). Reject while the comparisons pass; stop at the first failure and do not reject that hypothesis or any later one. Equivalently, the adjusted p-value attached to rank \(r\) is the running maximum of \((m-s+1)p_{(s)}\) over \(s\le r\), capped at 1.

The procedure does not require independent pairwise tests. It does require valid p-values. In a selection design, that qualification matters: p-values computed from data that also determined which arms survived are not automatically valid merely because Holm is applied afterward.

Selection Must Be Addressed Before Multiplicity

A selection rule can affect the distribution of the statistics used for later testing. If the same interim outcomes both determine which arms survive and enter the pairwise tests, observed contrasts may be unusually favorable simply because the selection rule favored arms with large observed effects. Applying an ordinary Holm adjustment to naïve p-values does not repair that problem: Holm addresses multiplicity across valid p-values, not invalid p-values caused by data reuse.

One clean design is to select arms using interim data and make the confirmatory comparisons using an independent final-data increment. Conditional on the selected set, the independent increment supports ordinary tests for the prespecified contrasts, subject to the outcome model and variance assumptions. If the final comparisons reuse selection data, inference instead needs a method that accounts for the selection event—for example, selection-adjusted p-values or a joint procedure whose reference distribution incorporates both the selection rule and the dependence among contrasts. When valid p-values are available, Holm is a simple dependence-robust option; a calibrated maxT procedure can sometimes use the known correlation structure to gain power. Neither approach should be applied as a mechanical substitute for modeling the selection process.

Diagnostics, Comparisons, and Limitations

Before interpreting results, check that the analysis matches the design. Confirm the number of surviving arms and the exact comparison family; verify that every contrast uses the same outcome definition, population, and timepoint; inspect how each standard error was obtained; and establish whether the test data were used in selection. If a model is used, check that its assumptions and variance estimates are appropriate for the outcome and randomization. A variance estimate near zero, sparse outcomes, severe departures from the assumed distribution, or an unmodeled covariance structure can make nominal p-values unreliable.

The main alternatives answer different operational needs. Bonferroni uses the threshold \(\alpha/m\) for each test and controls FWER under arbitrary dependence when p-values are valid; Holm is at least as powerful and is never less conservative than Bonferroni in its step-down decisions. A maxT procedure uses the joint distribution of the test statistics and their correlations, which can be advantageous when the dependence is reliably modeled. Dunnett-type procedures are designed for comparisons of several treatments with a common control; they do not by themselves supply the desired family of all treatment-versus-treatment comparisons. None of these multiplicity procedures, on its own, resolves bias from selecting arms using the same data later tested.

Holm-adjusted p-values summarize evidence while controlling FWER for the specified family. If simultaneous intervals are needed, construct intervals using a method that guarantees the desired joint coverage; a simple Bonferroni construction uses a two-sided critical probability of \(1-\alpha/(2m)\) for each contrast. Such intervals are generally wider than unadjusted intervals. Regardless of procedure, pairwise tests do not provide a complete ranking with certainty, and a nonsignificant result is not evidence that two treatments are equivalent. Equivalence requires a prespecified clinically acceptable margin and an appropriate equivalence analysis.

Common Pitfalls

• Testing every pair without defining the family. State whether the family includes all \(k(k-1)/2\) pairs or a smaller prespecified set. The multiplicity adjustment depends on that choice.
• Assuming Holm fixes selection bias. Holm controls multiplicity only when the input p-values are valid under the analysis conditions. It does not make reused selection data independent or remove selection effects.
• Treating pairwise tests as independent. Contrasts that share an arm are correlated. Use a procedure valid under the dependence present, or supply a justified joint covariance model to a correlation-aware method.
• Confusing failure to reject with equality. A large p-value does not show that two treatments have identical effects or clinically equivalent effects.
• Reporting a contrast without its direction or units. State which treatment is subtracted from which, what a positive value means, and the outcome units.