Multiplicity Across Candidate Treatments

Why Candidate Treatments Create a Multiplicity Problem

A seamless phase 2/3 trial may begin with several candidate treatments, use phase 2 evidence to choose which ones continue, and then make confirmatory claims from the combined design. If the trial can claim success for more than one treatment, the relevant question is not only whether each treatment-level test has a small Type I error rate. It is also how likely the design is to make at least one false claim across the candidate set.

The candidates may share a control group, and the set taken forward may depend on interim data. Those features can make the tests dependent and can affect whether an individual p-value is valid after selection. They do not remove the need to define the family of claims. A simple, robust approach is to protect the familywise error rate (FWER) by allocating the total error budget across the candidate treatments, using p-values that remain valid for the adaptive design.

For example, if the trial is designed to make claims about any of four candidate treatments, the family ordinarily contains all four treatment-versus-control hypotheses, even if only two treatments are selected at the stage transition. This is the conservative default when the same trial data influence both selection and confirmatory testing. Counting only the selected candidates requires a separate justification, such as valid conditional calibration under a design that supports it.

Valid Treatment-Level Tests Come First

Multiplicity adjustment cannot repair an invalid p-value. For each candidate \(j\), the p-value used in the familywise procedure must account for how the trial generated that candidate’s confirmatory evidence. In a seamless design, that may mean using valid stagewise p-values and a prespecified combination test, or another procedure that controls Type I error after the stage transition. A naive pooled analysis or unadjusted use of a phase 2 result for a treatment chosen because it looked promising may not have this property.

For the procedures below, assume that each candidate’s treatment-level p-value is valid under its own null: when \(H_j\) is true, \(P(p_j\leq u)\leq u\) for every \(u\) between 0 and 1, accounting for the full adaptive design. If a p-value is available only when candidate \(j\) is selected, the corresponding requirement is that \(P(j\text{ is selected and }p_j\leq u)\leq u\) under \(H_j\). These are unconditional validity statements over the trial’s randomization and data; they do not automatically follow from a p-value being valid for a fixed treatment in a nonadaptive analysis.

The p-values may be dependent. In particular, treatment comparisons using a shared control group are generally correlated. Bonferroni control does not require independence, but it does require that the individual p-values satisfy the stated validity condition.

Bonferroni Control for Selected Claims

The equal-allocation Bonferroni procedure gives each of the \(K\) candidate hypotheses a local significance level of \(\alpha/K\). Reject \(H_j\) only when its valid p-value is at most that threshold. The decision is applied to whichever candidate hypotheses are eligible for confirmatory claims under the prespecified design.

The denominator is the number of hypotheses in the family, not automatically the number selected at the transition. This distinction protects against the opportunity to make claims across the original set, including the opportunity to select candidates using interim data.

Unequal Error Allocation

Equal allocation is transparent, but it may be unnecessarily conservative for candidates with different scientific priorities. A weighted Bonferroni procedure assigns prespecified nonnegative weights \(w_j\) with \(\sum_{j=1}^K w_j\leq1\). Candidate \(j\) receives local level \(\alpha w_j\), and its null is rejected when \(p_j\leq\alpha w_j\). Larger weights give a candidate a larger share of the familywise error budget.

The weights should be prespecified or chosen by a method whose validity is established for the design. Choosing larger weights after seeing which treatment has the smallest interim p-value and then applying the ordinary weighted rule is not covered by the argument below. For a fixed, valid allocation, the union bound gives $$ P\left(\bigcup_{j\in I_0}\{p_j\leq\alpha w_j\}\right) \leq\sum_{j\in I_0}\alpha w_j \leq\alpha\sum_{j=1}^K w_j \leq\alpha. $$

When Selection Does and Does Not Reduce the Family

A single selected treatment does not automatically permit testing at the full level \(\alpha\). If selection and confirmatory evidence reuse the same data, the selected candidate is likely to be one whose interim result was unusually favorable. An ordinary fixed-treatment p-value may then be too small too often under the null. Bonferroni applied to valid adaptive p-values is one way to protect against the multiplicity across the full candidate set.

There are designs where selection need not consume a share of the error budget for every original candidate. For example, suppose exactly one candidate is selected using phase 2 data, and the confirmatory test uses independent new data. If, under each candidate’s null, that new-data p-value is valid conditional on the selection, then testing the one selected candidate at level \(\alpha\) can control the FWER. Independence and conditional validity are substantive design conditions; they cannot be assumed simply because the trial has a stage boundary.

Diagnostics, Alternatives, and Limitations

Before making a multiplicity-adjusted claim, check the whole chain of reasoning rather than only the final threshold. The following questions expose common failures:

• Is the family complete? List every treatment hypothesis that could support a confirmatory claim under the design. Omitting a candidate that could be declared successful can make the adjustment too lenient.
• Are the p-values valid after adaptation? Check that the treatment-selection rule, stagewise evidence, and any reuse of phase 2 observations are covered by the inferential procedure. A nominal p-value from a fixed-treatment analysis is not automatically valid after selection.
• Were weights fixed appropriately? Confirm that the allocation was prespecified or has a separately justified adaptive rule. Data-dependent weights chosen to favor the interim winner need their own calibration.
• Are the hypotheses and estimands aligned? Treatment comparisons should refer to the intended endpoint, population, and treatment effect. A multiplicity correction cannot resolve a change in the scientific question between stages.
• Is dependence being mistaken for independence? Shared control creates dependence. Bonferroni remains valid, but an alternative relying on an independence assumption needs that assumption checked and justified.

Bonferroni’s advantage is its simple strong-control guarantee under arbitrary dependence. Its cost is conservatism: it protects against all candidates simultaneously, including configurations where several null hypotheses are false. Prespecified weighting can direct more error budget toward priority candidates, but it does not eliminate the need for valid p-values or a defensible allocation. A design with one independent confirmatory test after selection may avoid a treatment-count adjustment under the stated conditional-validity conditions. More elaborate procedures can use relationships among hypotheses to gain power, but their guarantees must be established for the adaptive design and the claim family; they are not interchangeable with an unadjusted test of the selected winner.