Choosing an Estimator and Interval for a Stopped Trial

Why the Choice Depends on the Stopping Rule

A stopped trial does not automatically call for one particular estimator or interval. The estimate at the observed stopping stage may be easy to report, but the repeated-sampling behavior of that estimate depends on which results would have led the trial to stop there. The relevant question is therefore not simply, “Which formula is familiar?” It is, “Which inferential target and error guarantee does this analysis need to meet under this design?”

The introductory material on reporting stopped estimates and their uncertainty distinguishes the observed estimate from the uncertainty statement attached to it. Here the choice is made deliberately: specify the target, identify the stopping pattern, and then compare candidate procedures using the bias, coverage, and precision criteria relevant to that target. A procedure suitable for unconditional coverage across all stopping stages need not provide the same guarantee conditional on the stage actually reached.

Definition: Let \(\theta\) be the treatment effect, \(K\) the random stopping stage, \(\hat{\theta}_K\) a point estimator reported at that stage, and \(C_K\) a confidence interval. The unconditional bias is \(E_\theta(\hat{\theta}_K)-\theta\), averaging over all possible trial paths. The conditional bias at stage \(k\) is \(E_\theta(\hat{\theta}_K-\theta\mid K=k)\), when \(P_\theta(K=k)>0\). Unconditional coverage of a nominal \(100(1-\alpha)\%\) interval is \(P_\theta(\theta\in C_K)\); conditional coverage at stage \(k\) is \(P_\theta(\theta\in C_K\mid K=k)\). These are distinct operating characteristics.

Precision also needs a defined criterion. Common choices include the estimator’s variance or mean squared error, \(E_\theta[(\hat{\theta}_K-\theta)^2]\), and an interval’s expected width. Mean squared error combines variance and squared bias:

E_\theta[(\hat{\theta}_K-\theta)^2] = \operatorname{Var}_\theta(\hat{\theta}_K) + \{E_\theta(\hat{\theta}_K)-\theta\}^2.

Consequently, a bias reduction is not automatically an overall improvement: it may increase variance or interval width. Nor does a nominal 95% label by itself specify whether the intended guarantee is unconditional or conditional on stopping stage. The trial’s design and the scientific reporting objective must make that choice explicit.

How Efficacy Stopping Selects Estimates

Use a canonical known-variance information model to see the selection mechanism. At information level \(I>0\), let the cumulative score be \(S\sim N(I\theta,I)\). The standardized statistic is \(T=S/\sqrt I\sim N(\sqrt I\theta,1)\). The information-scale estimate is \(\hat{\theta}=S/I=T/\sqrt I\), with standard error \(1/\sqrt I\). If efficacy stopping occurs when \(T>c\), the reported estimate among those efficacy stops is drawn from the upper tail of the distribution, not from its unselected distribution.

Justification: Write \(\mu=\sqrt I\theta\) and \(a=c-\mu\). For \(T\sim N(\mu,1)\), change variables to \(Y=T-\mu\sim N(0,1)\). Since \(\int_a^\infty y\phi(y)\,dy=\phi(a)\), it follows that \(E(T\mid T>c)=\mu+\phi(a)/\{1-\Phi(a)\}\). Therefore \(E(\hat{\theta}\mid T>c)=\theta+\phi(a)/[\sqrt I\{1-\Phi(a)\}]\), which exceeds \(\theta\). This is conditional selection bias for the efficacy-stopping subset, not a claim that the estimator is biased by the same amount across all trial paths.

Worked Example: Quantifying Selection at an Efficacy Boundary

Suppose a hypothetical trial has \(I=100\), a true effect \(\theta=0.20\) units, and an efficacy rule \(T>2.40\). Under the stated model, \(T\sim N(\sqrt{100}(0.20),1)=N(2.00,1)\). The estimate is \(T/\sqrt I=T/10\), in effect units.

Here \(a=2.40-2.00=0.40\). Using \(\phi(0.40)=0.3683\) and \(1-\Phi(0.40)=0.3446\),

E(T\mid T>2.40) = 2.00+\frac{0.3683}{0.3446} \approx 3.069.

Thus the mean reported estimate among efficacy stops is \(3.069/10=0.3069\) units. Its conditional excess over the true effect is \(0.3069-0.2000=0.1069\) units. This calculation concerns only trials crossing the specified boundary. It is not the unconditional bias across trials that may continue to a later look or stop for another reason.

Choose the Target Before the Procedure

A useful selection process begins by writing down what the analysis must guarantee. For example, an overall estimate intended to describe the treatment effect across the design’s possible stopping paths may prioritize unconditional bias and mean squared error. A report specifically concerned with efficacy-stopped trials may require performance conditional on that event. For intervals, an unconditional 95% coverage requirement is not interchangeable with 95% coverage at each stopping stage.

Candidate approaches include reporting the raw information-scale estimate, applying a bias adjustment calibrated for the stopping rule, or using a likelihood-based estimate that accounts for the observed stopping information. Interval candidates may include an ordinary interval, an interval calibrated across the full stopping design, or a stage-conditional interval. These labels do not guarantee performance: each method must be evaluated for the specified model, design, estimand, and target. The earlier material on likelihood-based, bootstrap, and stage-specific intervals provides constructions; the selection task is to compare their operating characteristics for the decision at hand.

State the estimand and conditioning.
Specify the effect \(\theta\) and whether the desired bias or coverage statement averages across all stopping paths or conditions on the observed stopping stage or event.

Represent the actual design.
Include the planned information levels, efficacy and futility rules, unequal information increments where relevant, and the rule that determines the final stage.

Compare estimators and intervals on the target scale.
Assess bias, mean squared error or another stated precision measure, coverage for the required conditioning target, and expected width.

Check operating characteristics and limitations.
Use analytic results when available and simulation that reproduces the stopping rule otherwise. Examine performance across plausible values of \(\theta\), not only at one favorable value.

Worked Example: Selecting a Point Estimator for a Stopped Trial

Consider a hypothetical two-look normal-endpoint trial with \(I_1=25\) and \(I_2=100\). At the first look, the trial stops for efficacy if \(T_1>2.50\); otherwise it continues to the final look, with the prespecified final stopping rule applied there. The target is the effect \(\theta\) in units of the endpoint. Investigators require small conditional bias among first-look efficacy stops, while also requiring unconditional root mean squared error no greater than \(0.11\) units over all stopping paths.

A hypothetical design-based simulation with 100,000 trial replicates at \(\theta=0.20\) compares the raw stopped estimate with a bias-adjusted estimator calibrated to this stopping rule. The figures below are stipulated simulation summaries for this example, not data from an actual study.

Estimator	Conditional bias at first-look efficacy stop	Unconditional bias	Unconditional root mean squared error
Raw stopped estimate	+0.071 units	+0.008 units	0.104 units
Stopping-rule-calibrated adjustment	+0.019 units	+0.002 units	0.108 units

The raw estimate meets the unconditional precision limit because \(0.104<0.11\), but its first-look conditional bias, \(0.071\) units, exceeds a prespecified conditional-bias limit of \(0.03\) units. The calibrated estimator has conditional bias \(0.019<0.03\) units and unconditional root mean squared error \(0.108<0.11\) units. It therefore meets both stated criteria. Its slightly larger root mean squared error is a visible trade-off, not a reason to ignore the target. The conclusion is limited to the simulated design and parameter values; further evaluation over the plausible effect range is needed before using the procedure.

Selecting an Interval: Coverage and Precision

An interval-selection decision should specify both the nominal level and the probability statement it is intended to satisfy. For unconditional coverage, the repeated-sampling probability averages over all possible stopping stages. For stage-conditional coverage, the repeated-sampling probability is evaluated among trials that stop at a particular stage. A method calibrated for one target can fall short of the other. Interval width is a separate consideration: a very wide interval may achieve coverage but convey limited precision, while a narrow interval can be misleading if its coverage is inadequate.

Worked Example: Matching an Interval to a Coverage Requirement

Suppose a hypothetical two-look trial uses the stopping plan described above. The reporting requirement is at least 95% coverage conditional on each stopping stage. A design simulation reproducing the complete stopping rule gives these illustrative summaries at \(\theta=0.20\). The intervals and values are hypothetical.

Interval procedure	Overall coverage	Lowest estimated stage-conditional coverage	Mean width (effect units)
Ordinary symmetric interval at the stopped estimate	91.8%	86.7%	0.392
Unconditional calibrated interval	95.1%	88.7%	0.438
Stage-conditional calibrated interval	95.2%	95.0%	0.472

The ordinary interval fails the requirement because its lowest stage-conditional coverage is \(86.7\%<95\%\). The unconditional calibrated interval also fails that requirement: its overall coverage is \(95.1\%\), but its lowest stage-conditional coverage is \(88.7\%\). The stage-conditional procedure meets the specified minimum, with estimated coverage of at least 95.0% at each stage. Its mean width is \(0.472-0.438=0.034\) effect units greater than the unconditional interval’s. That extra width is the observed price of meeting the stronger stated coverage target in this illustration. The simulation summary is not a universal ranking; other designs or effect values may change both coverage and width.

Diagnostics, Alternatives, and Limits

Simulation is often the practical way to compare procedures when the stopping pattern and sampling model make analytic calculations difficult. Each simulated replicate must generate the interim and final data under the assumed model, apply the actual stopping rules, and calculate the estimate or interval that would be reported. If the simulation continues to the planned maximum sample size regardless of a boundary crossing, it does not reproduce the design and may give irrelevant bias and coverage estimates.

Check results across a grid of effect values, including values near the null and near the design alternative, and inspect each stage separately when conditional performance matters. For simulated coverage \(\hat{q}\) based on \(R\) independent replicates, its approximate Monte Carlo standard error is \(\sqrt{\hat{q}(1-\hat{q})/R}\). A coverage estimate just above 95% should not be treated as decisive if its Monte Carlo uncertainty is large. Also check whether some stopping stages are rare: conditional summaries based on very few replicates will be unstable.

The principal alternatives serve different objectives. Raw estimates are simple and can be useful as transparent summaries, but efficacy selection may produce upward conditional bias. Bias adjustment can target a specified selection effect, but its success depends on correctly representing the design and model; adjustment may also increase variance. Likelihood-based methods use the likelihood information under the specified model, but the chosen likelihood construction and interval calibration still need to match the desired inferential guarantee. Unconditional intervals target coverage averaged over stages; stage-conditional intervals target coverage within stages and may be wider. No method is best independently of the target.

Important failure modes include a misspecified outcome distribution, incorrect information levels, unmodeled changes to the stopping rule, and adaptation not represented in the analysis. A method calibrated for known variance may not retain its advertised properties when variance is estimated. A method evaluated only at one value of \(\theta\) may perform poorly elsewhere. Report the target, the stopping design represented in the evaluation, and the performance criterion used; avoid describing unconditional coverage as a guarantee for the particular stage observed.

Common Mistakes

Confusing standardized statistics with cumulative scores. In this model \(S\sim N(I\theta,I)\), whereas \(T=S/\sqrt I\sim N(\sqrt I\theta,1)\). The corresponding estimate is \(S/I=T/\sqrt I\); using the wrong scale changes both the distribution and the standard error.
Calling overall coverage stage-conditional coverage. An interval can reach 95% coverage averaged across stages while missing that target at one particular stage. State the conditioning event.
Choosing only by interval width. A shorter interval is not preferable if it fails the required coverage criterion.
Assuming adjustment always improves every property. Evaluate bias, variance or mean squared error, coverage, and width separately; they can move in different directions.

Key takeaway: Select an estimator and interval by first defining the inferential target and conditioning, then evaluating candidate procedures under the actual stopping rule. Bias, coverage, and precision are design-dependent properties, not automatic consequences of a method’s name.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the known-variance model, state the distributions of the cumulative score \(S\) and standardized statistic \(T\), and give the estimate and standard error at information \(I\).
Why can efficacy stopping create positive conditional bias even when the unselected information-scale estimate is unbiased?
An interval has 95% unconditional coverage. Does that establish 95% coverage conditional on every stopping stage? Explain.
A procedure has lower mean squared error but fails a prespecified stage-conditional coverage requirement. Is it suitable for that requirement?
Name two checks needed when simulation is used to compare stopped-trial procedures.

Solutions. (1) \(S\sim N(I\theta,I)\), \(T=S/\sqrt I\sim N(\sqrt I\theta,1)\), \(\hat{\theta}=S/I=T/\sqrt I\), and \(SE=1/\sqrt I\). (2) The efficacy rule selects observations with unusually large standardized statistics, shifting their conditional mean upward. (3) No; unconditional coverage averages across stopping paths and does not imply the same coverage within each stage. (4) No, not if stage-conditional coverage is a required guarantee; its precision advantage does not repair that failure. (5) At minimum, reproduce the actual stopping rule and examine performance across relevant effect values; also check stage-specific replicate counts and Monte Carlo uncertainty when those summaries matter.