Early Stopping and Conditional Error

Why Early Stopping Needs a Conditional Check

At an interim review, a trial may stop or continue according to the data observed so far. That decision changes what can happen afterward. A useful way to assess a proposed change is to ask: given these interim data, how much rejection probability did the original design still have available under the null hypothesis? This remaining conditional rejection probability is the conditional error. It is not the probability that the null hypothesis is true, nor is it a measure of the chance that the treatment is effective.

For a continuation, compare the chance of eventual rejection under the proposed remaining procedure with the conditional error available under the original design. For a futility stop with no rejection, that chance is zero. For an immediate efficacy rejection, it is one on the histories where rejection is certain. These cases can therefore have very different consequences for a conditional-error bound.

Definition: Let $Y$ denote all data available at an interim review, and let $R_0$ be the event that the original test rejects $H_0$. The conditional error at interim data $Y=y$ is $$ e(y)=P_0(R_0\mid Y=y). $$ Here $P_0$ denotes probability under the null model. If a modified procedure has conditional rejection probability $m(y)$, the conditional-error bound is respected at $y$ when $m(y)\le e(y)$.

The comparison is conditional on the actual interim information, not just on an average over possible interim results. If the pointwise inequality holds under the null, the modified procedure cannot have greater overall null rejection probability than the original test. The earlier material on conditioning and the conditional error bound develops this principle; here the emphasis is how early stopping enters the comparison.

A Two-Stage Normal Test and Its Conditional Error

Consider a one-sided test with two equally informative stages. Under the simple null hypothesis, assume the stage statistics $Z_1$ and $Z_2$ are independent and each has a standard normal distribution. The first-stage statistic $Z_1$ is observed at interim; $Z_2$ represents the independent information that would be collected in stage 2 if the trial continues. The original design rejects when the equally weighted combined statistic exceeds a fixed cutoff:

Z_{\mathrm{final}}=\frac{Z_1+Z_2}{\sqrt{2}}>c.

The test is one-sided and the cutoff is chosen for a null rejection probability of approximately 0.05. For numerical illustration, take $c=1.6449$, the usual rounded 5% standard-normal cutoff. These calculations assume the stated independent, standard-normal stage statistics and this fixed combination rule; changing the information weights, null distribution, or cutoff changes the conditional error.

Once $Z_1=z$ is observed, rejection by the original design requires $Z_2>\sqrt{2}c-z$. Under the null, $Z_2$ remains standard normal conditional on $Z_1=z$, by independence. Thus the exact conditional error relative to the specified cutoff $c=1.6449$ is:

e(z)=P_0\!\left(Z_2>\sqrt{2}(1.6449)-z\mid Z_1=z\right) =1-\Phi\!\left(\sqrt{2}(1.6449)-z\right).

Since $\sqrt{2}(1.6449)\approx 2.32624$, a convenient rounded calculation is $e(z)\approx 1-\Phi(2.326-z)$. The rounded expression is an approximation, not an exact equality. Also, $1.6449$ is itself a rounded cutoff, so an exactly calibrated test can instead define $c=\Phi^{-1}(0.95)$ and retain that value in the formula.

Worked Example: Calculating the Available Conditional Error

Suppose a hypothetical trial uses the two-stage design above and observes $Z_1=1.00$. Calculate the conditional error using the stated cutoff.

The condition for rejection at the end is $Z_2>\sqrt{2}(1.6449)-1.00$. Substituting the values gives:

\sqrt{2}(1.6449)-1.00 \approx 2.32624-1.00 =1.32624.

Therefore:

e(1.00)=1-\Phi(1.32624)\approx 1-0.9076=0.0924.

Under the null and conditional on this interim statistic, the original design has about a 9.24% chance of rejecting at the final analysis. This is a conditional probability for the remaining test, not the overall Type I error rate, which is approximately 5% before interim data are observed.

Representing Continuation and Checking the Bound

A continuation rule can be described by $q(y)$, the probability of continuing after interim data $y$. A deterministic rule has $q(y)=1$ when it continues and $q(y)=0$ when it stops. If stopping means no rejection, and the test used after continuation has conditional rejection probability $a(y)$ under the null, the modified conditional rejection probability is:

$$ m(y)=q(y)a(y). $$

The conditional-error check is therefore $q(y)a(y)\le e(y)$. This formulation makes the distinction between the chance of continuing and the chance of rejection after continuation explicit. A continuation decision alone does not determine the error: the remaining test matters too.

Worked Example: Checking a Continuation Chance

Return to the hypothetical interim result $Z_1=1.00$, for which the conditional error is approximately $0.0924$. Suppose a proposed continuation procedure continues with probability $q=0.40$. Conditional on continuing, its remaining test has null rejection probability $a=0.20$ at these interim data. Check the bound.

The modified conditional rejection probability is:

$$ m=q a=(0.40)(0.20)=0.0800. $$

Since $0.0800\le 0.0924$, the proposed continuation chance respects the conditional-error bound at this interim result. In contrast, if $q=0.50$ while $a=0.20$, then $m=(0.50)(0.20)=0.100$, which exceeds $0.0924$. That proposal does not satisfy the pointwise bound. This calculation checks only the stated interim result and the stated conditional rejection probability; a general rule must be checked over the interim histories to which it applies.

The same calculation can be read as a maximum permissible continuation probability when $a(y)>0$:

q(y)\le \frac{e(y)}{a(y)}.

The ratio is only informative when $a(y)>0$. If $a(y)=0$, continuing with that test creates no conditional rejection chance, whatever the continuation probability. If $e(y)=0$ and the continued test can reject with positive probability, the bound requires $q(y)=0$. In practice, $q(y)$ is also a probability and cannot exceed 1.

Futility Stops and Efficacy Stops Are Not Symmetric

A futility stop means the trial ends without rejecting $H_0$. Its modified conditional rejection probability is zero, which is no larger than the original conditional error $e(y)$, since probabilities are nonnegative. This statement concerns Type I error only: it does not say that the stopping rule is sensible for power, patient welfare, or other design goals.

An efficacy stop that rejects immediately has a different conditional consequence. On a history where the decision guarantees rejection, $m(y)=1$. If the original conditional error at that history is below 1, then immediate rejection fails the pointwise conditional-error comparison. The fact that the interim result appears favourable does not itself make this conditional probability acceptable.

Worked Example: Testing an Immediate Efficacy Stop

In the same hypothetical two-stage test, suppose the interim result is $Z_1=1.50$, and a proposed rule stops and rejects immediately. First calculate the original design's conditional error:

\sqrt{2}(1.6449)-1.50\approx 2.32624-1.50=0.82624,

e(1.50)=1-\Phi(0.82624)\approx 1-0.7955=0.2045.

The immediate-rejection rule has conditional rejection probability $m(1.50)=1$. Since $1>0.2045$, it does not preserve the original conditional-error bound at this history. The original design would reject only if the stage-2 statistic also crossed its required threshold; the proposed rule removes that requirement.

This does not imply that every design with an efficacy stop must have an excessive overall Type I error. A design planned and calibrated to include an early rejection boundary may control its overall error. The conclusion here is narrower: replacing this original final-only rule with immediate rejection at this history is not justified by preserving its original conditional error. It needs an appropriate error-control argument for the changed design.

Why the Conditional Bound Controls Overall Error

The link between the pointwise conditional comparison and overall error follows from conditioning on interim data. Let $R_1$ be rejection by a modified procedure. If its conditional rejection probability satisfies $P_0(R_1\mid Y=y)\le e(y)$ for every relevant $y$, then averaging over the null distribution of $Y$ gives:

P_0(R_1) =E_0\!\left[P_0(R_1\mid Y)\right] \le E_0[e(Y)] =E_0\!\left[P_0(R_0\mid Y)\right] =P_0(R_0).

The equalities use the law of total probability, and the inequality uses the conditional bound. Thus, if the original test has null rejection probability at most $\alpha$, the modified procedure also has null rejection probability at most $\alpha$, provided the conditional comparison is valid under the null model. This is a short justification of why a conditional check is stronger and more directly targeted than checking only an average continuation probability.

Diagnostics, Alternatives, and Limitations

A reliable audit starts by writing down the interim information being conditioned on, the original rejection event, and the null distribution of the remaining data. In the two-stage example, independence ensures that the conditional distribution of $Z_2$ remains standard normal after observing $Z_1$. If the increments are dependent, the threshold formula alone is not enough: the correct conditional distribution must be used. Similarly, changes to the statistic, information fraction, or rejection rule require recalculating the conditional error rather than reusing the displayed formula.

Common warning signs include comparing unconditional rejection probabilities when the proposed decision depends on interim data; confusing a continuation probability $q(y)$ with the conditional rejection probability $a(y)$; and rounding an intermediate cutoff and then presenting the resulting expression as exact. Keep sufficient precision in the calculation, report the approximation clearly, and check the inequality at the actual interim value. A rule that satisfies a bound at one observed value has not automatically been shown to satisfy it at every value where the rule may operate.

The main alternatives answer different questions. A fixed final analysis does not stop early. A futility stop can be assessed directly because it ends without rejection. A planned group-sequential design may permit early efficacy rejection, but its error control comes from the design's calibration rather than from assuming that an immediate rejection uses no more than the conditional error of a final-only test. Conditional-error comparison is useful when evaluating a change against an existing test; it is not a substitute for specifying the null model and the full rejection rule.

The derivation above is for a simple null with known standard-normal stage distributions. In applications with composite null hypotheses or nuisance parameters, a conditional bound may need to hold uniformly over the relevant null values; a calculation at only one parameter value is not automatically sufficient. Conditional error also concerns false rejection under the null, not power or the probability that the treatment effect is clinically important.

Key takeaway: At each interim history, compare the modified conditional rejection probability with the original conditional error. A no-rejection futility stop uses zero conditional rejection probability; immediate efficacy rejection uses one; continuation uses its continuation probability multiplied by the remaining test's conditional rejection probability.

Check Your Understanding

Use the two-stage model and stated cutoff $c=1.6449$ for Questions 1–3. Give each answer with the relevant conditional-error comparison.

At $Z_1=0$, calculate the original conditional error approximately.
At interim data with $e(y)=0.06$, a continuation rule has $q(y)=0.50$ and $a(y)=0.10$. Does it satisfy the conditional-error bound?
At a history where $e(y)=0.25$, the trial stops and rejects with certainty. Is the original conditional-error bound respected?
In one sentence, explain why checking only an average conditional rejection probability can miss a problem with a data-dependent rule.

Solutions. 1. At $Z_1=0$, $e(0)=1-\Phi(\sqrt{2}(1.6449))=1-\Phi(2.32624)\approx 1-0.9900=0.0100$, or about 1.00%. 2. The modified conditional rejection probability is $q a=(0.50)(0.10)=0.0500$; because $0.0500\le0.06$, it satisfies the bound. 3. Immediate rejection gives $m(y)=1$, and $1>0.25$, so it does not satisfy the original conditional-error bound. 4. An average may hide histories where the modified conditional rejection probability exceeds the available conditional error, whereas the pointwise check examines the bound at each relevant interim history.