Stopping-Induced Bias in Treatment Effect Estimates

Why the Reported Estimate Can Be Extreme

A treatment-effect estimator can be unbiased when the sample size is fixed in advance and still be biased when the study reports the estimate from a look chosen in response to interim results. The distinction is the selection: an efficacy rule tends to stop when the observed estimate is unusually favorable, so estimates from early stops are selected values, not a random sample of all possible estimates at that look.

This does not mean every estimate from a sequential trial is necessarily biased in the same direction, or that a particular early estimate is known to be too large. The direction and size of the bias depend on the stopping rule, the true effect, and which outcomes are included in the expectation. The central point is that the rule for choosing when to report can change the repeated-sampling distribution of the reported estimator.

Fixed-Look Unbiasedness and Selection

For a simple reference model, suppose that at each prespecified look \(k\), the estimator satisfies \(\hat{\theta}_k\sim N(\theta,1/I_k)\), where \(I_k\) is the information available at that look. Assume the treatment-effect model is correctly specified, the variance or information scale is known, and the estimator is unbiased at each fixed look. The normal model is a useful approximation in many large-sample settings; its assumptions should not be treated as automatic for every trial.

A fixed-look estimator has expectation \(\theta\) before the data are observed. But a stopping rule makes the event \(\{\tau=k\}\) depend on interim data. Consequently, even if \(E_\theta(\hat{\theta}_k-\theta)=0\), the selected error \((\hat{\theta}_k-\theta)\) need not have mean zero among trials that stop at \(k\). In general,

Here \(\mathbf{1}\{\tau=k\}\) equals 1 when stopping occurs at look \(k\), and 0 otherwise. Fixed-look unbiasedness says that the expectation of the error without this indicator is zero. It does not say that the error multiplied by an outcome-dependent selection indicator has expectation zero. The indicator can give more weight to favorable estimation errors.

Two-Look Derivation: How the Stopping Rule Changes the Mean

The following canonical normal example isolates the mechanism. Let the true standardized treatment effect be \(\theta=0\). At the first look, let the information be \(I_1=1\), so the score \(Z_1=\sqrt{I_1}\hat{\theta}_1\) has distribution \(N(0,1)\). The second look has total information \(I_2=2\). Write its cumulative score as \(Z_2=Z_1+\varepsilon\), where \(\varepsilon\sim N(0,1)\) is independent of \(Z_1\). Then \(\hat{\theta}_1=Z_1\) and \(\hat{\theta}_2=Z_2/2\); both estimates are unbiased at their respective fixed looks.

Suppose the rule stops early if \(Z_1\geq c\), with \(c=1.96\), and otherwise continues to the second look. This is a hypothetical one-sided efficacy rule. It assumes a prespecified boundary, the stated independent normal increments, and that no other stopping rule changes which estimate is reported.

In the last line, \(\phi\) is the standard normal density. The identity \(E[Z_1\mathbf{1}\{Z_1\geq c\}]=\phi(c)\) follows by integrating \(z\phi(z)\) from \(c\) to infinity and using \(\phi'(z)=-z\phi(z)\). Thus, although both fixed-look estimators have expectation zero, the reported estimator has positive expectation, about \(0.02922\) standardized treatment-effect units under this particular rule and model.

Reading a Stopped Estimate in Context

Consider a separate hypothetical trial with first-look information \(I_1=25\) and an efficacy boundary \(Z_1\geq2.40\). Suppose the observed first-look estimate is \(0.48\) treatment-effect units and its standard error is \(0.20\) units. The statistic is \[ Z_1=\frac{0.48}{0.20}=2.40. \] The boundary is reached, so the protocol reports the estimate \(0.48\) units and stops. The calculation verifies the rule’s application; it does not calculate how much this particular estimate is biased. That would require a repeated-sampling model and an expectation over the stopping procedure.

The appropriate interpretation is that \(0.48\) is the observed estimate at the selected stopping look. It should not be described as though it came from a sample size fixed independently of the observed treatment effect. Nor does the selection argument prove that the true treatment effect is smaller than \(0.48\). It warns that, across repetitions of the same design, early efficacy stops preferentially occur when interim estimates are favorable.

Assumptions, Diagnostics, and Alternatives

The normal derivation depends on a correctly stated information scale, unbiased fixed-look estimators, independent increments, and a stopping rule based on the interim statistic. Actual analyses may depart from these assumptions because of model misspecification, estimated nuisance parameters, delayed outcomes, changes to the look schedule, or additional futility and safety rules. The direction of bias also need not be upward in every design: an upper efficacy boundary favors positive estimates among early stops, while a lower boundary or a different reporting rule can select in another direction.

Useful checks focus on whether the analysis accounts for how the estimate was selected. Confirm that the actual stopping rule, look timing, and boundary used in the trial match the rule assumed for inference. In repeated simulations under plausible parameter values, compare the mean of the fixed-look estimator with the mean of the reported stopped estimator, and examine early-stop estimates separately from all reported estimates. A large difference is evidence that selection matters under the simulated model; it is not a universal bias correction for the real trial.

The main alternative is not to pretend that the sample size was fixed. One may report the ordinary estimate with clear disclosure that it was selected by a stopping rule, and use inferential procedures calibrated to the sequential design when estimation uncertainty is needed. A fixed-sample analysis can remain a useful descriptive reference, but its usual sampling properties do not automatically carry over after data-dependent selection. Procedures that adjust the estimate or construct intervals accounting for stopping require additional methods; the key prerequisite is to specify the stopping rule and the target of inference.

Stopping-induced bias matters because an estimate often communicates the magnitude of benefit, not merely whether a boundary was crossed. If the estimate is selected to be favorable, presenting it without the stopping context can overstate how representative it is of the design’s repeated-sampling behavior. The bias is a property of the complete procedure—data collection, interim decisions, and reporting—not simply a defect in the arithmetic of the estimator.