Bias After Early Efficacy Stopping

Why Early Efficacy Stopping Selects Large Estimates

An interim efficacy rule stops a trial early only when the accumulating evidence is sufficiently favorable. If the treatment effect estimate is then taken from the data that triggered the stop, the estimate has been selected for being large. Even when the estimator would be unbiased in a fixed-sample design, the estimates from trials that cross the boundary tend, on average, to exceed the true treatment effect.

The key is to specify the event on which the estimate is being examined. Here the relevant event is crossing the efficacy boundary at the interim analysis, not merely that a trial ended early for any reason. The calculations below isolate this single-boundary case. They describe the conditional mean among trials that cross; they do not, by themselves, give the bias of every possible estimator used across both stopping and continuation paths.

Model and assumptions: Let \(\theta\) be the true treatment effect, oriented so that larger positive values favor the experimental treatment, and let \(\hat{\theta}_1\) be its interim estimate. Assume that at a fixed interim sample size, \(\hat{\theta}_1\sim N(\theta,SE_1^2)\), with known, nonrandom \(SE_1\). The estimate is unbiased before applying the stopping rule. For a one-sided efficacy test of \(H_0:\theta=0\), stop for efficacy when \(Z_1=\hat{\theta}_1/SE_1\ge c\), where \(c\) is the prespecified efficacy boundary. The reported estimate on this path is \(\hat{\theta}_1\).

The normal model is exact for some settings and an approximation in others. For example, it may be reasonable for a large-sample estimator with an approximately normal sampling distribution. The calculation treats the interim sample size and standard error as fixed and assumes a single efficacy look. If the variance is estimated, the statistic is not well approximated by a normal variable, or there are multiple looks, the simple formula may not apply unchanged.

The Conditional Mean After Crossing

The boundary crossing is the event \(\hat{\theta}_1\ge cSE_1\). Standardizing the interim estimate around its true mean gives \(Y=(\hat{\theta}_1-\theta)/SE_1\sim N(0,1)\). Thus crossing is equivalent to \(Y\ge a\), where \(a=(cSE_1-\theta)/SE_1\). Write \(\phi\) and \(\Phi\) for the standard normal density and cumulative distribution function. The probability of early efficacy stopping is \(1-\Phi(a)\).

E(\hat{\theta}_1\mid Z_1\ge c) =\theta+SE_1\frac{\phi(a)}{1-\Phi(a)}, \qquad a=\frac{cSE_1-\theta}{SE_1}.

The extra term is positive because both the standard error and the normal density are positive, and \(1-\Phi(a)>0\). Therefore the mean interim estimate among trials that cross the boundary exceeds \(\theta\). This is the upward selection bias in the early-stopped group under the stated model. The size of the excess depends on the boundary, the standard error, and the true effect through \(a\); it is not a fixed correction to subtract in every trial.

Why the Formula Holds

For \(Y\sim N(0,1)\), conditioning on \(Y\ge a\) gives \(E(Y\mid Y\ge a)=E[Y\,I(Y\ge a)]/P(Y\ge a)\), where \(I\) is the indicator of the event. Since \(\phi'(y)=-y\phi(y)\), integrating the numerator yields the density at the truncation point:

\int_a^\infty y\phi(y)\,dy =-\int_a^\infty \phi'(y)\,dy =\phi(a), \qquad P(Y\ge a)=1-\Phi(a).

It follows that \(E(Y\mid Y\ge a)=\phi(a)/[1-\Phi(a)]\). Substituting \(\hat{\theta}_1=\theta+SE_1Y\) gives the displayed conditional-mean formula. The derivation shows the mechanism directly: restricting attention to values above a cutoff leaves a truncated normal distribution with a positive mean relative to its untruncated mean of zero.

Worked Example: The Mean Estimate Among Efficacy Stops

Suppose a hypothetical trial estimates a treatment effect in units of symptom-score improvement. At its planned interim look, assume \(\theta=0.20\) units, \(SE_1=0.10\) units, and an efficacy boundary \(c=2.4\). Under the model, \(\hat{\theta}_1\sim N(0.20,0.10^2)\). The efficacy event is \(Z_1\ge2.4\), equivalently \(\hat{\theta}_1\ge 2.4(0.10)=0.24\) units.

First calculate the standardized cutoff relative to the true mean:

a=\frac{0.24-0.20}{0.10}=0.40.

The chance of stopping for efficacy is \(1-\Phi(0.40)\approx0.3446\), or about 34.46%. Using \(\phi(0.40)\approx0.3683\), the conditional mean estimate is

E(\hat{\theta}_1\mid Z_1\ge2.4) =0.20+0.10\frac{0.3683}{0.3446} \approx0.3069\text{ units}.

Thus, under these hypothetical assumptions, trials that cross the boundary have an average interim estimate of about 0.3069 units, rather than the true effect of 0.20 units. The excess is about 0.1069 units. This is a conditional average over the set of efficacy-stopped trials, not a prediction that every such trial will overestimate by that amount.

Worked Example: A Rare Stop Can Have a Large Conditional Excess

Now suppose a hypothetical treatment has \(\theta=0\) units and the interim standard error is \(0.10\) units, with the same one-sided boundary \(c=2.4\). The estimate is unbiased before applying the rule: \(\hat{\theta}_1\sim N(0,0.10^2)\). The stop event is \(\hat{\theta}_1\ge0.24\) units, so \(a=(0.24-0)/0.10=2.4\).

The probability of an efficacy stop is \(1-\Phi(2.4)\approx0.00820\), about 0.820%. With \(\phi(2.4)\approx0.02239\), the mean among those stops is

E(\hat{\theta}_1\mid Z_1\ge2.4) =0+0.10\frac{0.02239}{0.00820} \approx0.2731\text{ units}.

The boundary requires an estimate of at least 0.24 units, so a positive conditional mean is expected. Here the stop is uncommon under the assumed true effect, but the estimates in the selected group average about 0.2731 units. This does not mean that the treatment effect is likely to be positive after observing a stop; the calculation is an expectation under a specified model and conditions on the crossing event.

Worked Example: The Excess Depends on the Boundary Relative to the True Effect

Consider a hypothetical trial with the same true effect, \(\theta=0.20\) units, but a smaller interim standard error, \(SE_1=0.05\) units. Keep \(c=2.4\). The efficacy threshold for the estimate is now \(2.4(0.05)=0.12\) units, below the true effect. Assume \(\hat{\theta}_1\sim N(0.20,0.05^2)\), as in the model.

The standardized cutoff is \(a=(0.12-0.20)/0.05=-1.6\). Therefore the stopping probability is \(1-\Phi(-1.6)=\Phi(1.6)\approx0.9452\). Since \(\phi(-1.6)\approx0.1109\), the conditional mean is

E(\hat{\theta}_1\mid Z_1\ge2.4) =0.20+0.05\frac{0.1109}{0.9452} \approx0.2059\text{ units}.

The conditional excess is about 0.0059 units. Most estimates cross the boundary in this example, so the selection restriction removes relatively little of the distribution. Compared with the first example, the smaller conditional excess illustrates why the boundary must be considered relative to the true effect and the standard error; the boundary value \(c\) alone does not determine the bias.

Which Trials and Estimates Are Being Compared?

The conditional calculation concerns the distribution of \(\hat{\theta}_1\) given that the efficacy boundary was crossed. It does not say that the original fixed-look estimator has a positive unconditional bias: before conditioning, its mean is \(\theta\) by assumption. The difference arises because the stopping rule selects a subset of its possible values.

An adaptive design may use one estimate when it stops early and a different, later estimate when it continues. The expectation of that overall reporting rule depends on what happens after continuation and on exactly how estimates are constructed. It cannot be obtained just by applying the conditional formula for the stopped group. In particular, the conditional mean among early stops should not be confused with an unconditional mean across all randomized trials.

The relevant event must also match the actual decision rule. With a single one-sided look, it is \(Z_1\ge c\). If the protocol uses several efficacy looks, stopping can occur at the first look that crosses its boundary, and the selected event involves the sequence of interim statistics. If stopping can occur for reasons other than efficacy, those paths should not be grouped with efficacy stops when interpreting this selection mechanism.

Specify the estimate.
Name the interim estimator, its scale, and its standard error under the assumed sampling model.

Specify the selection event.
Write the efficacy rule as an inequality, such as \(Z_1\ge c\), and express it on the estimate scale.

Condition on that event.
Calculate or describe the estimate among trials meeting the efficacy criterion, not among an unspecified collection of completed trials.

State the scope of the conclusion.
Call the result a conditional upward shift for efficacy-stopped estimates; do not infer an overall estimator bias without analyzing every trial path.

Diagnostics, Comparisons, and Limitations

A useful diagnostic is to report how many trials crossed the efficacy boundary and to summarize their interim estimates separately from those of trials that did not cross. The protocol’s boundary and the timing of the analysis should be stated alongside those summaries. If the observed analysis has different stopping criteria from the one being evaluated, the single-boundary formula is not an adequate description of the selection event.

This mechanism differs from selecting the largest observed effect among several treatments: here, even one treatment estimate can be selected by crossing a prespecified efficacy boundary. It also differs from changing the sample size based on interim information. In both settings data-dependent decisions can affect estimation, but the relevant selection event and resulting estimator need to be specified separately. A fixed-sample analysis without efficacy-triggered selection does not condition its estimate on a boundary crossing, although its estimate can still vary by sampling error.

The formula is a benchmark, not a universal bias adjustment. It relies on a normal estimator with fixed standard error, a single specified one-sided efficacy boundary, and reporting the interim estimate on the stopping path. Estimated nuisance parameters, discrete outcomes, multiple interim looks, alternative stopping rules, or a different reported estimator may require a different calculation. The result also describes an average over repeated hypothetical trials under the model; it does not quantify the error in any one realized estimate.

Common Mistake

A common error is to reason, “The estimator is unbiased under the normal model, so the estimate from an early-stopped trial is unbiased too.” The first statement is about the distribution before selection: \(E(\hat{\theta}_1)=\theta\). The second concerns a conditional distribution. Once the analysis is restricted to \(Z_1\ge c\), values below the boundary are excluded, giving

E(\hat{\theta}_1\mid Z_1\ge c) =\theta+SE_1\frac{\phi(a)}{1-\Phi(a)}>\theta.

The correction is to name the conditioning event whenever describing the estimate. Also avoid saying that a boundary crossing proves the treatment effect is large: the conditional expectation is not a guarantee about an individual trial, and it does not give the probability that \(H_0\) is true.

Key takeaway: Under the single-look normal model, efficacy stopping selects interim estimates above a threshold. The resulting mean among stopped trials is the true effect plus a positive truncation term. Interpret that result conditionally on the actual efficacy-crossing event.

Check Your Understanding

Answer each question using the single-look normal model stated above. Solutions follow the questions.

A hypothetical trial has \(\theta=0.15\) units, \(SE_1=0.08\) units, and \(c=2\). What estimate threshold defines an efficacy stop, and what is the conditional mean estimate?
Under \(H_0:\theta=0\), let \(SE_1=0.12\) units and \(c=2\). Find the probability of an efficacy stop and the conditional mean estimate among stopped trials.
Why does the conditional mean after crossing not establish that the estimator across all trial paths is upward biased?
For a design with several interim looks, why is conditioning only on “the trial stopped early” insufficient to apply the single-look formula?

Solutions. (1) The estimate threshold is \(cSE_1=2(0.08)=0.16\) units. Here \(a=(0.16-0.15)/0.08=0.125\), so the stop probability is \(1-\Phi(0.125)\approx0.4503\). Using \(\phi(0.125)\approx0.3958\), the conditional mean is \(0.15+0.08(0.3958/0.4503)\approx0.2203\) units. (2) The threshold is \(2(0.12)=0.24\) units and \(a=2\). The stop probability is \(1-\Phi(2)\approx0.02275\). Thus the conditional mean is \(0+0.12[\phi(2)/(1-\Phi(2))]\approx0.12(0.05399/0.02275)\approx0.2848\) units. (3) This conditional mean uses only trials that cross the boundary. The estimator across all paths also depends on the estimates and reporting rule for trials that continue, so its expectation requires analyzing those paths too. (4) With multiple looks, the event depends on the sequence of statistics and on when the first crossing occurs. It is not generally the same as crossing one fixed boundary at one look.