Why Futility at the Phase Transition Needs Careful Definition
At the end of phase 2 in a seamless phase 2/3 trial, investigators may select a treatment and decide whether to continue into the confirmatory stage. A futility rule identifies phase 2 results that are considered too weak to justify that continuation. The choice between a binding and a nonbinding rule is not just about how disappointing results look: it specifies whether the confirmatory testing plan relies on the stopping rule being followed.
This distinction matters because the phase 2 result can be used both to decide whether the trial continues and, through a combination test, to contribute evidence to the final test. The earlier material on independent increments and combination tests supplies the testing framework. Here the focus is what a futility decision does to the set of possible rejections, and what that implies for power and type I error.
For a single selected treatment, let \(Z_1\) summarize phase 2 evidence and \(Z_2\) summarize independent phase 3 evidence. For illustration, suppose under the null hypothesis \(H_0\), the stage statistics are independent standard normal variables. An inverse-normal combination statistic can be written
If the prespecified one-sided type I error rate is \(\alpha\), the no-futility test rejects when \(C>c_\alpha\), where \(c_\alpha\) is the standard normal critical value for an upper-tail probability of \(\alpha\). For example, \(c_\alpha\) is approximately 1.960 when \(\alpha=0.025\). In a real seamless design, treatment selection and multiplicity must also be handled by the chosen valid testing procedure; this simple expression illustrates the futility issue, not a substitute for that adjustment.
What Stopping Does to Type I Error
Suppose a prespecified futility boundary is \(Z_1<f\). If the study stops at that boundary, no confirmatory rejection is made. Under the nonbinding approach, the combination test is calibrated as if phase 3 information were available, and the futility boundary is not used to make that test more permissive. If investigators nevertheless stop, the actual rejection event is a subset of the rejection event that would have been possible had the trial continued.
The argument is an event inclusion. Imagine, for the purpose of the argument, that the phase 3 data that would have been collected are available even when the actual trial stops. Let \(R\) be the event that the full-data combination test rejects, and let \(R_F\) be the event of rejection under the futility rule. A stopped trial cannot reject, and a trial that continues must still satisfy the full-data test. Therefore \(R_F\subseteq R\). Under \(H_0\),
This argument does not say the rule has no effect on type I error. A rule that prevents some otherwise possible rejections can make the actual error rate conservative. Nor does the argument justify changing the efficacy boundary after looking at the futility outcome. Any such change needs its own type I error justification.
A binding design makes a different commitment: stopping is part of the planned procedure and must be enforced. A test can be calibrated using the joint distribution of the interim and final statistics together with the binding stopping rule. For example, a design could choose its efficacy boundary to satisfy \(P_{H_0}(Z_1\geq f,\ C>c_b)=\alpha\). This calculation uses the specified joint distribution, and may give a different efficacy boundary \(c_b\) from the full-data critical value. The stopping rule alone does not guarantee control: the model, selection procedure, stopping rule, and test boundary must all be calibrated together.
Worked Example: Conditional Power at the Transition
Worked Example: Conditional Power at the Transition
Suppose the planned test uses \(w_1=0.6\), \(w_2=0.8\), and \(c_\alpha=1.960\). The weights satisfy \(0.6^2+0.8^2=1\). Assume the phase 3 statistic is independent of \(Z_1\) and, for planning, has distribution \(Z_2\sim N(\delta_2,1)\) with \(\delta_2=1.50\). Conditional power is the probability that the planned test rejects, given the observed \(Z_1\), under this specified assumption about phase 3 evidence.
At an observed \(Z_1=0\), rejection requires
Since \(Z_2\sim N(1.50,1)\), the conditional power is
If the protocol stops when conditional power is below 0.20, it stops at \(Z_1=0\). At \(Z_1=0.40\), rejection requires \(Z_2>(1.960-0.6(0.40))/0.8=2.150\), so conditional power is \(1-\Phi(2.150-1.50)=1-\Phi(0.650)\approx0.2578\); the rule continues. These are conditional probabilities under the stated phase 3 effect assumption, not probabilities that the treatment works.
Worked Example: The Type I Error Bound
Worked Example: The Type I Error Bound
Suppose the no-futility combination test uses \(w_1=0.6\), \(w_2=0.8\), and \(c_\alpha=1.960\). Under \(H_0\), assume \(Z_1\) and \(Z_2\) are independent standard normal variables. Then \(C=0.6Z_1+0.8Z_2\) is standard normal, so the full-data test has type I error \(P(C>1.960)\approx0.0250\). Now add a nonbinding rule to stop if \(Z_1<0.50\), with no rejection after stopping.
The rejection probability for the stopped trial is \(P(Z_1\geq0.50,\ C>1.960)\). Because this event requires \(C>1.960\), it is contained within the full-data rejection event. Thus
This establishes the error-rate bound without needing the exact joint probability. The actual type I error is generally less than the full-data rate, because some full-data rejections may be removed by stopping. A binding design could instead calibrate a boundary using the probability of the joint event, but must use the actual prespecified stopping and selection procedures in that calculation.
Power: What Is Lost and What Can Be Recovered?
Power is the probability of rejecting \(H_0\) when a specified alternative is true. If a futility rule is followed and the efficacy boundary remains unchanged, stopping can remove trials that would have crossed the full-data efficacy boundary had they continued. Thus the actual power cannot exceed the power of that same test with continuation guaranteed. The size of the reduction depends on how often the trial meets the futility criterion under the alternative and how likely those trials would otherwise have been to reject.
A nonbinding rule often has two power figures worth distinguishing: power for the planned test assuming continuation, and power for the operational trial that follows the rule. The first is not the actual power if the rule is routinely enforced. If the rule is genuinely nonbinding and the team may override it, actual power also depends on the continuation policy. That policy should be defined before results are seen, rather than improvised after an unfavourable interim outcome.
A binding rule can be included directly in power calculations. Its efficacy boundary may be calibrated jointly with the stopping rule to use the type I error budget efficiently. Consequently, it is not correct to claim that a binding design must always have lower power than every nonbinding design: the comparison depends on the boundaries, continuation policy, and alternative effect assumptions. With the same efficacy boundary and no rejection after stopping, however, enforcing futility cannot increase power relative to always continuing.
Worked Example: Translating a Conditional-Power Cutoff
Continue with \(w_1=0.6\), \(w_2=0.8\), \(c_\alpha=1.960\), and the planning assumption \(\delta_2=1.50\). Suppose the futility rule stops when conditional power is below 0.20. The standard normal quantile with lower-tail probability 0.80 is approximately 0.8416. Conditional power is below 0.20 when the standardized phase 3 threshold exceeds 0.8416:
Solving this inequality gives
Thus this conditional-power rule is equivalent, under the stated assumptions, to stopping when \(Z_1\) is below approximately 0.145. For an observed \(Z_1=0.10\), the trial stops; for \(Z_1=0.20\), it continues. The threshold depends on the planned effect \(\delta_2\): changing that assumption changes the stopping rule and its power.
Assumptions, Diagnostics, and Alternatives
A futility calculation is only as credible as its inputs. Before using it, verify that the phase 2 statistic and phase 3 statistic are defined for the same confirmatory claim, and that the combination test remains valid after treatment selection. If selection requires adjusted stagewise p-values or a multiplicity procedure, those adjustments must be retained; a simple unadjusted \(Z_1\) calculation does not replace them. Also verify the assumed stage independence or the covariance structure used in calibration, the planned weights and stage sizes, and the exact rule for missing or delayed phase 2 outcomes.
Conditional power is especially sensitive to the assumed future effect. Using the observed phase 2 effect as the phase 3 planning effect can be optimistic because the treatment was selected partly for looking promising. A prespecified clinically relevant effect, a conservative effect, or a sensitivity analysis over several plausible effects can make the decision more transparent. The calculation should be labelled with its effect assumption and not presented as an objective measure of the probability of eventual success.
| Approach | How futility enters | Main consequence |
|---|---|---|
| No futility rule | Continue regardless of phase 2 evidence, unless another prespecified rule applies. | Power and type I error are those of the full planned test; trial duration and cost may be greater. |
| Nonbinding futility | Stopping is allowed, but type I error control does not rely on stopping being enforced. | Stopping with no rejection can make type I error conservative; continuing after the boundary does not by itself invalidate the full-data test. |
| Binding futility | The design and test calibration assume the rule is followed. | Stopping is mandatory; error rate and power must be calculated for the joint procedure. |
Monitor implementation as well as statistical calculations. A protocol should specify the statistic or conditional-power formula, threshold, data cut, treatment-selection procedure, and what happens if information is incomplete. If the team continues after a binding boundary, the planned operating characteristics no longer describe the executed design. If it stops under a nonbinding rule, it should not report the power that assumed continued follow-up as the power actually achieved.
Common Mistake
A common error is to say, “The futility boundary is nonbinding, so stopping at it has no effect on power or type I error.” The correction is that nonbinding describes the basis of the type I error guarantee, not an absence of operational consequences. If the trial stops and cannot reject, its actual power may decrease and its actual type I error may be conservative. Conversely, continuing after a nonbinding boundary is compatible with the guarantee only when the full-data test and its selection adjustments remain the prespecified valid procedure.
Another error is to treat a binding rule as automatically safe because it sounds stricter. A binding stop can reduce the rejection set when used with an unchanged efficacy boundary, but any recalibration or other adaptation must still be justified for the complete procedure. The rule, test, and treatment-selection mechanism must be assessed together.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In one sentence, distinguish a binding futility rule from a nonbinding one.
- A valid full-data test has type I error 0.025. A futility stop results in no rejection, while all continuing trials use that unchanged test. Give an upper bound for the stopped trial’s type I error and justify it.
- For the combination test in the conditional-power example, calculate the conditional power at \(Z_1=0.40\), given \(\delta_2=1.50\).
- Why is it incorrect to report continuation-assumed power as the actual power when a futility rule is routinely enforced?
Solutions. (1) Binding means the type I error guarantee relies on the rule being followed; nonbinding means it does not. (2) The type I error is at most 0.025, because rejection after the rule is a subset of rejection under the valid full-data test. (3) The threshold is \(Z_2>(1.960-0.6(0.40))/0.8=2.150\). Under \(Z_2\sim N(1.50,1)\), conditional power is \(1-\Phi(0.650)\approx0.2578\). (4) Trials stopped for futility cannot reach the planned phase 3 rejection, so the actual rejection probability may be smaller than the probability calculated assuming every trial continues.