From an Adaptive Rule to a Testable Design
A multiarm trial is not fully specified by saying that “the best treatments will continue.” A reproducible design must state what information ranks the arms, exactly which arms can continue, what happens if none qualify, and how the final treatment claims will be tested. These details matter because an adaptive selection rule changes which hypotheses reach confirmatory testing.
This workshop constructs a three-treatment-versus-control design. Its central feature is that interim selection uses only stage-one participants, while confirmatory testing uses new, independent stage-two participants. This separation makes it possible to evaluate each selected treatment with a valid stage-two test conditional on the selection decision. The number of tests actually performed then determines the multiplicity adjustment in this design; the fact that other arms might have been selected does not by itself require testing every possible claim.
Specify the Trial and Its Decision Rule
Suppose the primary outcome is a continuous measure at a fixed follow-up time, with larger values considered better. There is one control group and three experimental treatments, labelled A, B, and C. Let \(\theta_j\) denote the mean outcome on treatment \(j\) minus the mean outcome on control, for \(j\in\{A,B,C\}\). The confirmatory null hypothesis for treatment \(j\) is \(H_{0j}:\theta_j=0\), against the two-sided alternative \(H_{aj}:\theta_j\ne0\). A positive estimated difference favours the treatment, but the confirmatory test remains two-sided as prespecified.
Assume the endpoint is measured in both stages using the same definition, eligibility criteria, follow-up, and analysis plan. For illustration, assume independent participants, normally distributed outcomes within each treatment group, and a common known standard deviation of 10 outcome units. Stage-one and stage-two participants are disjoint. Stage one uses a shared control group to estimate each treatment-control difference; the resulting estimates can be correlated because the contrasts share controls. Correlation among interim estimates does not invalidate a fully specified ranking rule.
The interim rule is: calculate each stage-one treatment-control estimate; retain the two treatments with the largest estimates among those whose estimate is greater than zero; drop all other treatments. If only one estimate is greater than zero, retain that treatment alone. If none is greater than zero, stop for lack of promising treatments and make no confirmatory claim. Ties are resolved by a prespecified alphabetical rule. The rule is a selection screen, not a confirmatory test: it does not establish treatment efficacy.
Stage two enrolls new participants to the control group and to each retained arm. No dropped treatment is restarted, no stage-one observations enter the confirmatory estimates, and no unplanned treatment comparison is added. These restrictions are part of the design assumptions, not administrative preferences.
Define the Confirmatory Family and Error Rule
Let \(S\) denote the set of treatments retained at interim, so \(S\) is empty, contains one treatment, or contains two treatments. For each \(j\in S\), the confirmatory test uses only the stage-two treatment and control observations. In this illustration, if both groups have \(n=60\) participants and the known standard deviation is \(\sigma=10\), the stage-two standard error of a treatment-control difference is \(\sqrt{\sigma^2/n+\sigma^2/n}\). The standardized statistic is the estimated stage-two difference divided by this standard error. Under \(H_{0j}\), it has a standard normal distribution under the stated model.
Here \(\widehat{\theta}_{j,2}\) is the stage-two difference in sample means. A two-sided p-value is \(P(|Z|\ge |z_j|)\) under a standard normal null distribution. If exactly one treatment is retained, test its null at level 0.05. If two are retained, test each at level \(0.025\), using Bonferroni allocation. Thus, for a retained set of size \(m\), each of its tests uses level \(0.05/m\). The familywise error rate (FWER) is the probability of rejecting at least one true treatment null in the family tested.
This family consists of the hypotheses for treatments actually retained and tested. Since selection is based on data independent of stage two, a single selected treatment can be tested at 0.05; it does not need an automatic adjustment for all three treatments that could possibly have been selected. When two are tested, the two 0.025 tests provide a simple FWER guarantee. This conclusion would not automatically carry over to a design that reused stage-one data in confirmatory tests.
Worked Example: Select Two Arms and Test Them
Suppose the stage-one estimated treatment-control differences are 4.8 units for A, 3.5 for B, and 1.1 for C. All three estimates are positive, so the rule retains A and B, the two largest, and drops C. Stage two then supplies independent data, with 60 participants in each of control, A, and B. Suppose the stage-two estimated differences are 5.0 units for A and 3.2 units for B.
The rule calls for two tests, each at \(0.05/2=0.025\). For A, \(Z_A=5.0/1.826=2.738\). Its two-sided p-value is approximately 0.0062, which is below 0.025; reject \(H_{0A}\). For B, \(Z_B=3.2/1.826=1.753\). Its two-sided p-value is approximately 0.0797, which exceeds 0.025; do not reject \(H_{0B}\).
The confirmatory conclusion is evidence of a difference between A and control, but not between B and control, at the design’s familywise 0.05 level. The conclusion concerns the stage-two population contrast under the trial’s analysis assumptions; the larger stage-one estimate for A is not added to the confirmatory estimate.
Why the Error Bound Holds
Strong FWER control means controlling the probability of at least one false rejection under every configuration of true and false null hypotheses, including cases where some treatments truly work and others do not. The relevant justification conditions on the selected set. Because stage-two participants are independent of stage one, conditioning on \(S\) does not alter the null distribution of a valid stage-two test for a true null. Each such test therefore has conditional rejection probability at most its assigned level.
Here \(I_0\) is the set of treatments whose null hypotheses are true, and the inequality is the union bound: the probability of a union of events is no greater than the sum of their probabilities. The last inequality follows from \(|S\cap I_0|\le |S|\). If \(S\) is empty, there are no confirmatory rejections and the conditional FWER is zero. Averaging the conditional bound over all possible selected sets gives the unconditional bound: \(P(\text{at least one false rejection})\leq0.05\).
This argument does not require the stage-two test statistics to be independent of one another. Their dependence, including correlation from a shared stage-two control group, does not defeat the union bound. It does require that each individual stage-two test has the stated conditional type I error bound and that the tested family and levels are implemented as specified.
Worked Example: One Selected Arm and a Partial-Null Configuration
Suppose the interim rule retains only A, so A is tested at 0.05. If A is in fact a true null and the stage-two test is valid, the conditional probability of falsely rejecting its null is at most 0.05. The other two possible treatment claims are not tested. Dividing 0.05 by three would also be conservative, but is not required by this independent-stage design.
Now suppose two arms are retained, A and B, with A a true null and B a false null. Each test has level 0.025. Conditional on selecting this pair, the probability of a false rejection is at most 0.025, because only A’s null is true. If both selected arms are true nulls, the union bound gives at most \(0.025+0.025=0.05\). These cases illustrate strong control, not merely control when every treatment is ineffective.
Implementation Checks and Failure Modes
A planned error guarantee can fail if the operational trial differs from the design used in its justification. The following checks should be resolved before enrollment and monitored during execution:
- Confirm stage separation. Verify that no participant or outcome used for selection contributes to the confirmatory treatment contrasts. If data are reused, the conditional-null argument above no longer applies as written.
- Lock the selection rule. Record the endpoint, analysis time, ranking direction, positivity criterion, maximum number retained, and tie rule. A discretionary choice after viewing results is not the prespecified rule.
- Preserve the testing levels. One retained arm receives level 0.05; two receive 0.025 each. Testing two at 0.05 each can give an FWER as high as 0.10 under two true nulls, so it does not meet the 0.05 target.
- Check test calibration. The normal test here assumes the stated normal model and known variance. With estimated variance, missing outcomes, non-normal endpoints, or a different estimand, use an appropriate valid stage-two test and ensure its null p-value is super-uniform (its probability of being at most \(u\) is no greater than \(u\)) under the null.
- Track the actual family. A dropped arm must not quietly re-enter confirmatory testing, and extra outcomes or treatment contrasts must not be treated as confirmatory without an error-control plan.
A fixed all-arm design is a useful alternative when every treatment comparison must remain available for testing regardless of interim results. It may use a procedure such as Bonferroni across all three comparisons or an appropriate method that accounts for shared-control correlation. That is a different family and can be more conservative than this selection design. Conversely, methods for selection-adjusted inference are needed when the selection data also contribute to the confirmatory test. The independent-stage design is attractive precisely because it avoids that data reuse, at the cost of reserving new participants and control data for stage two.
Worked Example: Audit a Proposed Change
A team proposes to retain two arms under the stated rule but test both stage-two contrasts at 0.05 because only the selected arms will be reported. If both null hypotheses are true and each test is correctly calibrated, the union bound gives an FWER upper bound of \(0.05+0.05=0.10\), not 0.05. Dependence might make the actual value smaller, but it does not establish the required bound. The planned correction is to test both at 0.025, or to prespecify a different valid joint testing procedure that controls the two-test family at 0.05.
If instead the team changes the design to test only one selected arm at 0.05 using independent stage-two data, that single test can control FWER at 0.05. The design must state in advance how that one arm is chosen and ensure no additional confirmatory test is performed.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Under the stated rule, what are the confirmatory test levels if zero, one, or two treatments are retained?
- If two arms are retained and both null hypotheses are true, what FWER bound follows from the specified procedure?
- Why does a single selected arm tested on independent stage-two data not automatically require division of 0.05 by three?
- Would Bonferroni’s two-test guarantee require the two stage-two test statistics to be independent? Explain.
- What key assumption is broken if stage-one participants are included in the confirmatory treatment estimates?
Solutions. (1) With zero arms there are no tests; with one arm the level is 0.05; with two arms each level is 0.025. (2) The union bound is \(0.025+0.025=0.05\). (3) The selected arm’s stage-two test is valid conditional on selection because the data used for testing are independent of the data that selected it; its conditional rejection probability under its true null is at most 0.05. The unselected claims are not tested. (4) No. Bonferroni uses the union bound, which holds regardless of dependence, provided each component test is valid at its assigned level. (5) Selection and testing would no longer use independent data, so conditioning on the selected set may change the null distribution of the reused-data test; the error argument here would need to be replaced by an appropriate selection-adjusted procedure.