Reopening After A Futility Stop

Why Reopening Requires an Error Budget

A binding futility rule has a direct inferential consequence: once the interim result enters the prespecified stopping region, the original design makes no further rejection decision. Reopening that trial and allowing a later rejection changes the paths on which the null hypothesis can be rejected. The fact that the decision to reopen follows a futility stop does not, by itself, protect type I error.

The conditional error function makes this change precise. It asks how much probability of rejection the original design assigns to future data, conditional on the interim information already observed. Comparing a proposed reopening rule with that quantity gives a pathwise error constraint. A separate calculation is needed if the goal is only to remain below a nominal alpha level and the original design used less than that full allowance.

Definition: Let $D$ denote the interim data and let $R_{\mathrm{old}}$ be the event that the original design rejects $H_0$. Under a specified null model, the conditional error at $D=d$ is $$ A_{\mathrm{old}}(d)=P_0(R_{\mathrm{old}}\mid D=d). $$ A continuation rule preserves the original conditional error if its conditional probability of rejection, given $D=d$, is no greater than $A_{\mathrm{old}}(d)$ for every relevant interim result $d$.

Here $P_0$ denotes probability under the null hypothesis. The conditional-error preservation principle is a sufficient way to retain the original design's type I error control: if the new conditional rejection probability is bounded by the old one at every interim result, averaging over the interim data cannot increase the overall rejection probability. It is deliberately a pathwise requirement, stronger than merely requiring the new design's overall type I error to be at most a nominal $\alpha$.

What a Binding Futility Stop Implies

Suppose a trial has interim statistic $Z_1$, and its binding futility rule stops the trial when $Z_1\leq c_F$. The original design cannot reject after that stop. Consequently, for every interim result in this region,

A_{\mathrm{old}}(z)=0,\qquad z\leq c_F.

Let $r(z)$ be the probability that a proposed reopening procedure rejects $H_0$, conditional on the observed interim result $Z_1=z$. This probability includes any randomness in the later data and, if used, any randomization in the decision rule. Pointwise conditional-error preservation requires

r(z)\leq A_{\mathrm{old}}(z)=0,\qquad z\leq c_F.

Because probabilities cannot be negative, this condition forces $r(z)=0$ throughout the stopped region. Thus, under exact conditional-error preservation, a binding futility stop cannot be reopened for a confirmatory test that has a positive chance of rejection. Further data could still be collected for nonconfirmatory purposes, but a positive rejection probability in that region needs a different error-budget argument.

Worked Example: The Original Error in a Stopped Path

Suppose the binding futility cutoff is $c_F=0.5$. The observed interim value is $Z_1=0.2$, so the original design stops. If it is reopened and a new, independent standard normal statistic $Z_R$ is used to reject when $Z_R>1.645$, what is the conditional rejection probability?

Assume for this calculation that under $H_0$, $Z_R\sim N(0,1)$ and is independent of $Z_1$. The proposed conditional rejection probability is

r(0.2)=P_0(Z_R>1.645\mid Z_1=0.2) =1-\Phi(1.645)\approx 0.0500.

The original conditional error at $Z_1=0.2$ is zero, because the binding rule stops there. Hence $0.0500>0$: the proposed rule does not preserve the original conditional error. This comparison is about pathwise preservation; it does not yet determine whether the new design exceeds a nominal alpha level overall.

A Normal Combined-Test Illustration

Consider a one-sided two-stage normal test. Under the simple null, assume $Z_1$ and $Z_2$ are independent and each has a standard normal distribution. If the trial continues, the original design rejects when the combined statistic $Z_{\mathrm{comb}}=(Z_1+Z_2)/\sqrt{2}$ exceeds 1.96. The binding futility rule stops when $Z_1\leq c_F$. These assumptions specify the null distribution needed for the calculation; in applications, the statistic's distribution, its dependence across stages, and the decision rule must be justified by the trial design.

For an interim value $Z_1=z>c_F$, rejection occurs exactly when $(z+Z_2)/\sqrt{2}>1.96$. Solving this inequality for the future statistic gives

\frac{z+Z_2}{\sqrt{2}}>1.96 \quad\Longleftrightarrow\quad Z_2>1.96\sqrt{2}-z.

Therefore, on a path where the original trial continues, its conditional error is

A_{\mathrm{old}}(z) =P_0\!\left(Z_2>1.96\sqrt{2}-z\mid Z_1=z\right) =1-\Phi(1.96\sqrt{2}-z),\qquad z>c_F.

The independence assumption means that the conditional distribution of $Z_2$ given $Z_1=z$ remains standard normal under this null model. On a path where the binding rule stops, the original design's conditional error is instead zero, not the tail probability from hypothetically applying the combined-statistic test.

Worked Example: Checking the Combined-Test Threshold

Take $c_F=0.5$ and consider two interim results: $z=1$, where the original trial continues, and $z=0.2$, where it stops. Calculate the relevant conditional errors and explain what the calculations mean.

At $z=1$, the correct future-statistic threshold is

1.96\sqrt{2}-1 =1.96(1.4142)-1 \approx 1.7719.

Thus the original conditional error on this continuation path is

A_{\mathrm{old}}(1)=P_0(Z_2>1.7719)\approx 0.0382.

At $z=0.2$, applying the same combined-statistic rejection rule hypothetically would give the threshold $1.96\sqrt{2}-0.2\approx 2.5719$, with upper-tail probability about $0.0051$. But $0.2\leq 0.5$, so the binding rule stops. The original design's actual conditional error is therefore $A_{\mathrm{old}}(0.2)=0$, not $0.0051$. The latter number describes a different design that ignores the binding stop. Confusing these two quantities would assign positive original conditional error to a path on which the original rule never rejects.

Using Nominal Alpha Slack

The zero conditional error in stopped paths does not automatically mean that every reopening procedure exceeds nominal alpha. A binding futility rule can make the original design conservative: its actual null rejection probability $\alpha_{\mathrm{old}}$ may be less than the allowed level $\alpha$. That unused amount is overall alpha slack. A procedure that exceeds the original conditional error on stopped paths can still meet the nominal level if its added rejection probability, averaged over those paths, fits within the slack.

For a simple null model, suppose reopening leaves all original decisions unchanged on paths where the original trial continued, and uses conditional rejection probability $r(z)$ on stopped paths. Then the new type I error is exactly

P_0(R_{\mathrm{new}}) =\alpha_{\mathrm{old}}+ E_0\!\left[\mathbf{1}(Z_1\leq c_F)r(Z_1)\right].

The equality follows by partitioning the rejection event into the original continuation region and the stopped region. The second term is the additional rejection probability: conditional on $Z_1=z$, a stopped path contributes $r(z)$, and averaging over the null distribution of $Z_1$ gives its total probability. Accordingly, the exact overall-alpha condition for this particular unchanged-on-continuation setup is

E_0\!\left[\mathbf{1}(Z_1\leq c_F)r(Z_1)\right] \leq \alpha-\alpha_{\mathrm{old}}.

Unlike the pointwise conditional-error constraint, this permits $r(z)>0$ on some stopped paths if there is enough overall slack. It is not the same guarantee as preserving the original conditional error. With a composite null, the bound must hold for every null parameter value, or for the worst-case type I error over the null; a calculation at just one null distribution is not enough for strong control.

Worked Example: Reopening Within a Known Alpha Budget

Suppose a hypothetical two-stage design has $Z_1\sim N(0,1)$ under a simple null, stops for futility when $Z_1\leq 0.5$, and, if it continues, rejects when an independent $Z_2>1.645$. The nominal level is $\alpha=0.025$. A reopening rule rejects on stopped paths when a new independent standard normal statistic exceeds 2.326. Does this rule preserve the old conditional error, and does it meet the nominal level?

The original conditional rejection probability after continuation is $P_0(Z_2>1.645)\approx0.0500$. Since $P_0(Z_1>0.5)\approx0.3085$, the original overall type I error is

\alpha_{\mathrm{old}} =P_0(Z_1>0.5)P_0(Z_2>1.645) \approx 0.3085(0.0500)=0.01543.

The stopped-region probability is $P_0(Z_1\leq0.5)\approx0.6915$. The reopening test has conditional rejection probability $P_0(Z_R>2.326)\approx0.0100$, so its added type I error is

$$ 0.6915(0.0100)=0.006915. $$

The new total is approximately $0.01543+0.006915=0.02235$, which is below $0.025$. It fits the nominal error budget, since the available slack is approximately $0.025-0.01543=0.00957$. It does not preserve the original conditional error: the old error was zero on stopped paths, whereas the new conditional rejection probability is $0.0100$. The conclusion relies on the stated simple null and independence assumptions; broader null models require their own worst-case calibration.

Derivation, Diagnostics, and Alternatives

The key justification is conditional expectation. If the new rule has conditional rejection probability $A_{\mathrm{new}}(D)$, then its overall rejection probability under a specified null is

P_0(R_{\mathrm{new}}) =E_0\!\left[P_0(R_{\mathrm{new}}\mid D)\right] =E_0[A_{\mathrm{new}}(D)].

If $A_{\mathrm{new}}(D)\leq A_{\mathrm{old}}(D)$ for every relevant interim result, taking expectations preserves the inequality. Therefore $P_0(R_{\mathrm{new}})\leq E_0[A_{\mathrm{old}}(D)]=P_0(R_{\mathrm{old}})$. In the binding futility region, the old conditional error is zero, which proves why pointwise preservation rules out positive rejection probability there.

A practical review should check the following before approving a reopening proposal:

Reconstruct the original rule. Record exactly which interim results stopped, continued, or could lead to rejection. For a binding stop, do not substitute the hypothetical rejection rule for the rule actually specified.
Condition on the relevant information. Compute rejection probability given the observed interim data, not merely given the label “reopened.” If different interim results have different conditional distributions or rules, they cannot be treated as one interchangeable path.
Verify the null distribution and dependence. The normal tail calculation above requires independent standard normal stage statistics under the stated simple null. If that assumption fails, its displayed tail probability need not be correct.
Audit the total error separately. If using nominal alpha slack, calculate the original actual type I error and the added error under each relevant null configuration. Do not describe an overall-alpha calculation as pointwise conditional-error preservation.

The main alternative is to retain the binding stop and collect no further confirmatory data on stopped paths. Another is to reopen under a prospectively specified rule that satisfies a chosen error criterion: pointwise conditional-error preservation, or a separate overall-alpha calibration that explicitly accounts for available slack. These approaches answer different questions. Conditional-error preservation directly protects the old design's error path by path; overall calibration may permit error on previously stopped paths but requires a valid calculation for the complete new design.

A frequent mistake is to argue, “The original futility rule was conservative, so any reasonable reopening must still be below alpha.” The correction is to quantify both parts: the original rejection probability and the added probability of rejection on stopped paths. Conservatism creates only a finite budget, and the added error must not exceed it. Conversely, observing that a proposed rule has positive conditional error where the old error was zero establishes failure of exact conditional-error preservation, but does not alone establish that nominal alpha is exceeded.

Key takeaway: After a binding futility stop, the original conditional error is zero. Exact conditional-error preservation therefore permits no confirmatory rejection on that path. Positive rejection probability can be considered only under a separately justified overall error budget, with the full design calibrated over the relevant null hypotheses.

Check Your Understanding

Answer each question using the stated rules and assumptions.

A binding futility rule stops when $Z_1\leq0$. What is the original conditional error at $Z_1=-0.4$, and what does pointwise preservation require of a reopening test?
In the combined-test example, derive the threshold for $Z_2$ from $(z+Z_2)/\sqrt{2}>1.96$. Why is $1.96-z$ not the correct threshold?
A design has nominal $\alpha=0.05$, actual original type I error $0.04$, and reopening adds $0.012$ to the overall rejection probability. Does it preserve the original conditional error? Does it meet nominal alpha?
Why is a calculation under one simple null insufficient by itself to establish type I error control for a composite null?

Solutions. (1) The original conditional error is zero because the binding rule stops at $-0.4$; pointwise preservation requires conditional rejection probability zero there. (2) Multiplying by $\sqrt{2}$ gives $z+Z_2>1.96\sqrt{2}$, hence $Z_2>1.96\sqrt{2}-z$. The threshold $1.96-z$ omits the factor $\sqrt{2}$ on the critical value. (3) The added error is positive on paths where the old conditional error was zero, so pointwise preservation fails. The new total is $0.04+0.012=0.052$, which exceeds $0.05$; the available slack was only $0.01$. (4) Type I error control for a composite null requires the rejection probability to be bounded over all parameter values in the null, including any nuisance-parameter values that may yield the largest error.