Why Early Stopping Selects Large Estimates

Why a Benefit Stop Can Produce an Overlarge Estimate

A trial that stops early for efficacy has not merely collected fewer observations than planned. Its stopping decision depends on unusually favorable evidence: the observed treatment effect, or a statistic that measures it, had to be large enough to cross a pre-specified boundary. This selection can make the effect estimate reported at the stop larger on average than the true treatment effect.

The mechanism is easiest to see in a normal model. The argument is about selection, not a claim that every early-stopped trial must overestimate its effect. A particular estimate may be above or below the truth, and the direction and size of bias across a sequential design depend on the stopping rule and the parameter value.

Model and assumptions: Let \(\theta\) be the treatment effect, in units where larger values indicate greater benefit, and let \(\hat{\theta}\) be its estimate at a specified analysis. Assume \(\hat{\theta}\sim N(\theta,SE^2)\), with a known, fixed \(SE\) for this analysis. An efficacy rule stops for benefit when \(Z=(\hat{\theta}-\theta_0)/SE\geq a\), where \(\theta_0\) is the reference effect and \(a\) is a fixed positive threshold. For the examples, \(\theta_0=0\). The model treats the analysis schedule, standard error, and boundary as specified in advance.

Rearranging the stopping rule shows the scale of effect estimate required for a crossing:

Z\geq a \quad\Longleftrightarrow\quad \hat{\theta}\geq \theta_0+a\,SE.

Thus, a crossing is not a random selection from all possible estimates. It selects only estimates at or above a threshold. When the true effect is below that estimate threshold, every estimate that crosses is above the truth. Even when the threshold is below the true effect, conditioning on crossing still favors positive estimation errors in the one-look normal model; the derivation below explains why.

A One-Look Derivation of the Selection Effect

Standardize the estimation error as \(U=(\hat{\theta}-\theta)/SE\). Under the stated normal model, \(U\sim N(0,1)\). The efficacy crossing event is equivalent to

U\geq c, \qquad c=\frac{\theta_0+a\,SE-\theta}{SE}.

For any finite \(c\), conditioning a standard normal variable on being greater than \(c\) gives it a positive mean. This sign can be justified without calculating the mean's numerical value. If \(c\geq0\), then \(U>c\) implies \(U\) is positive (apart from a probability-zero endpoint). If \(c<0\), the excluded values \(U\leq c\) have a negative conditional mean. Since the unconditional mean is zero, removing that negative tail leaves the conditional mean among the remaining values positive. Therefore, whenever the crossing has positive probability,

E(\hat{\theta}\mid Z\geq a) =\theta+SE\,E(U\mid U\geq c) >\theta.

This establishes the direction of the one-look conditional bias: among analyses that cross the efficacy threshold, the mean estimate exceeds the true effect. The conclusion concerns a conditional average over repeated trials under this model, not a guarantee about a single trial. The exact size of the conditional bias requires more than this sign argument and is not needed to understand the selection mechanism.

Worked Example: A Null Effect and a First-Look Crossing

Suppose a hypothetical trial estimates a treatment effect in percentage points. At its first analysis, assume \(\theta=0\), \(SE=0.40\) percentage points, and a one-sided efficacy threshold \(a=2.00\), with \(\theta_0=0\). These assumptions specify the normal model and the scale of the boundary.

The standardized statistic is \(Z=\hat{\theta}/0.40\). The crossing rule \(Z\geq2.00\) is therefore equivalent to

\hat{\theta}\geq 2.00(0.40)=0.80\text{ percentage points}.

Under the null true effect, \(Z\sim N(0,1)\), so the probability of crossing at this single analysis is \(P(Z\geq2.00)\approx0.0228\). Conditional on crossing, the estimate must be at least \(0.80\) percentage points, whereas the true effect is \(0\). With a continuous distribution, the conditional mean is strictly greater than \(0.80\), not merely equal to it. This example separates two quantities: \(0.0228\) is the crossing probability under the assumed true effect; it is not the size of the estimate or the probability that the treatment is ineffective.

What Changes With More Than One Look?

A group sequential trial examines accumulating information at scheduled analyses. In the usual normal-increment model, information at look \(k\) is \(I_k\), the estimate has standard error \(1/\sqrt{I_k}\), and \(Z_k=(\hat{\theta}_k-\theta_0)\sqrt{I_k}\). The accumulated information increases across looks. A rule with efficacy boundaries \(a_k\) stops at the first look \(k\) for which \(Z_k\geq a_k\); reaching a later look also means that no earlier efficacy boundary was crossed.

The first-crossing condition is path-dependent: at look \(k\), the observed estimate is selected both for crossing the current boundary and for having avoided earlier crossings. Consequently, the one-look calculation is a transparent demonstration of the selection mechanism, not a formula for the bias at every look of every sequential design. The unconditional mean across all possible outcomes, the mean among trials stopping at a particular look, and the mean among trials stopping early are different targets. Their values depend on the full stopping rule.

Worked Example: A Positive Effect Below the Boundary Threshold

Consider a hypothetical first-look analysis with true effect \(\theta=0.30\) percentage points, \(SE=0.20\) percentage points, and efficacy threshold \(a=2.10\), again with \(\theta_0=0\). Assume the estimator is normal with this standard error. The threshold estimate is

\theta_0+a\,SE=0+2.10(0.20)=0.42\text{ percentage points}.

The crossing probability can be calculated under the specified true effect. The standardized statistic has mean \(\theta/SE=0.30/0.20=1.50\) and variance \(1\). Thus,

P(Z\geq2.10)=P\{N(1.50,1)\geq2.10\} =P\{N(0,1)\geq0.60\}\approx0.2743.

Here the boundary threshold, \(0.42\) percentage points, exceeds the true effect of \(0.30\). Every crossing estimate therefore exceeds the truth by at least \(0.12\) percentage points. For instance, a hypothetical observed estimate of \(0.46\) gives \(Z=0.46/0.20=2.30\), so it crosses; its observed excess over the specified true effect is \(0.46-0.30=0.16\) percentage points. The crossing probability describes how often this rule would be met under the assumed model; it does not say that the effect estimate will be \(0.46\) or that every crossing will have the same excess.

Worked Example: Following a Hypothetical Two-Look Path

Suppose a hypothetical trial uses information \(I_1=25\) and \(I_2=100\), with efficacy boundary \(a_k=2.00\) at both looks and \(\theta_0=0\). Assume the normal-increment model, so \(SE_k=1/\sqrt{I_k}\). Let the true effect be \(0.10\) units. Consider the possible observed estimates \(\hat{\theta}_1=0.32\) and \(\hat{\theta}_2=0.25\), in those same effect units.

At look 1, \(SE_1=1/\sqrt{25}=0.20\), so

Z_1=\frac{0.32-0}{0.20}=1.60<2.00.

The trial does not stop for efficacy at this look. At look 2, \(SE_2=1/\sqrt{100}=0.10\), and

Z_2=\frac{0.25-0}{0.10}=2.50\geq2.00.

The trial stops for efficacy at look 2. The reported estimate is \(0.25\) units, which is \(0.25-0.10=0.15\) units above the hypothetical true effect. The look-2 threshold estimate is \(2.00(0.10)=0.20\) units. This path illustrates why an early efficacy report can be optimistic: the observed estimate had to be sufficiently favorable to trigger stopping. It does not establish the average bias among all trials with this two-look rule; that calculation must account for the joint distribution of both looks and the condition of no crossing at look 1.

Interpretation, Diagnostics, and Alternatives

The practical concern is estimation, not whether a correctly calibrated sequential test can control its type I error. A sequential p-value or boundary calibration addresses the behavior of the test under its specified null model. It does not, by itself, make the ordinary estimate at a stopping time unbiased. Likewise, a confidence interval that ignores the stopping rule may not have its advertised coverage conditional on the observed stopping event.

When reviewing a report, check whether the observed analysis was the first efficacy crossing, whether the boundary and look schedule were pre-specified, and whether the estimate is presented with its standard error and units. Compare the estimate with the effect threshold implied by the boundary, \(\theta_0+a_k SE_k\). An estimate just over this threshold can still be materially above the truth, particularly when the true effect is small. These checks explain the selection risk; they do not quantify its magnitude.

The main alternatives answer different questions. Continuing to the planned fixed sample size avoids stopping specifically because an interim estimate crossed an efficacy boundary, although other design or analysis choices can still affect estimation. If the trial stops, investigators can report the ordinary estimate transparently and also use an estimation or interval procedure that accounts for the stopping rule. Such procedures require the rule, boundaries, and analysis schedule to be known; merely relabeling an unadjusted estimate does not correct it. A calibrated sequential test remains appropriate for testing, but is not a substitute for a stopping-aware estimate or interval.

Key takeaway: Under a normal estimator model, crossing an efficacy threshold selects favorable estimation errors. In a one-look analysis, the estimate conditional on crossing has mean greater than the true effect. In a multi-look trial, the same mechanism motivates concern, but the exact bias depends on the complete first-crossing rule.

Common Pitfalls and Limits

Confusing crossing probability with bias. A probability such as \(P(Z\geq a)\) tells how often the threshold is crossed under a specified parameter. It is not the amount by which the estimate is biased.
Claiming every stopped estimate is too large. A particular estimate can fall below the true effect. The upward statement is about a conditional mean under specified assumptions, not an individual guarantee.
Ignoring earlier looks. Conditioning only on \(Z_k\geq a_k\) does not fully describe stopping at look \(k\), because stopping there also requires no earlier crossing.
Assuming the normal model automatically applies. The examples assume a normal estimator with a fixed standard error and a pre-specified threshold. Unstable variance estimates, changing analysis plans, or a different data-generating model require separate justification.
Treating a valid p-value as a bias correction. Testing calibration and estimation bias are distinct properties. Correcting one does not automatically correct the other.

The normal model gives a clean explanation, but its conclusions must be matched to the design. The bias can be small or substantial, and it is not determined by the fact of early stopping alone. A careful analysis states the estimand, the stopping event being conditioned on, and the assumptions used to describe repeated trials.

Check Your Understanding

Answer each question using the normal-model selection argument. Solutions follow the questions.

With \(\theta_0=0\), \(SE=0.15\) units, and \(a=2.40\), what estimate threshold corresponds to efficacy crossing?
Under a true effect of \(0.20\) units, the threshold estimate in Question 1 is crossed. Is every crossing estimate above the true effect? By how much at minimum?
In a one-look normal model, why is the conditional mean estimation error positive after crossing, even if the boundary threshold is below the true effect?
At look 1 a trial does not cross, and at look 2 it does. Why is the one-look conditional calculation not by itself the exact bias calculation for estimates stopping at look 2?
Does a sequentially calibrated p-value imply that the ordinary effect estimate at the stopping look is unbiased? Explain briefly.

Solutions. (1) The estimate threshold is \(\theta_0+aSE=0+2.40(0.15)=0.36\) units. (2) Yes. Since \(0.36>0.20\), each crossing estimate is at least \(0.16\) units above the true effect (strictly above the threshold with probability one under a continuous model). (3) Write the estimation error in standard-error units as \(U\sim N(0,1)\). Crossing conditions on \(U\geq c\). If \(c\geq0\), the retained values are positive; if \(c<0\), the excluded lower tail has a negative mean, so the mean of the remaining values must be positive because the original mean was zero. (4) Stopping at look 2 requires both crossing at look 2 and not crossing at look 1. The earlier noncrossing condition is part of the selection event, and the looks are dependent through accumulated information. (5) No. Test calibration concerns rejection probabilities under the null; it does not by itself remove the selection effect in the ordinary estimate or guarantee interval coverage conditional on stopping.