Coverage Failure of Fixed-Sample Intervals After Stopping

Why Stopping Can Change Interval Coverage

A conventional confidence interval is calibrated for a specified sampling plan. In a group sequential design, however, the estimate reported may come from a stage selected by the observed data: unusually large interim results can trigger an early stop, while other results lead to further observations. The estimate and the decision to report it are therefore statistically linked.

This link matters even when the interval formula is familiar and its standard error is calculated correctly. The interval may still have coverage different from its nominal level because the estimate’s distribution among trials that stop at a particular look is not its fixed-sample distribution. A useful way to isolate this issue is to start with a normal model and a known standard error, then examine exactly which estimates are selected.

The distinction is between the interval formula and the sampling procedure to which its coverage claim applies. The ordinary normal interval \(\hat{\theta}\pm1.96\,SE\) has 95% coverage when, for a fixed sample size, \((\hat{\theta}-\theta)/SE\sim N(0,1)\). After stopping, the relevant question is whether that standardized estimate still has the required distribution among the reported estimates.

A Normal Two-Look Setup

Consider independent observations \(Y_i\sim N(\theta,\sigma^2)\), with known \(\sigma\), and planned sample sizes \(n_1<n_2\). At look \(k\), let \(\hat{\theta}_k\) be the mean of the first \(n_k\) observations, with \(SE_k=\sigma/\sqrt{n_k}\). For illustration, suppose the trial stops for efficacy at the first look if the standardized statistic for testing \(H_0:\theta=0\) is at least a positive boundary \(c\); otherwise it continues to \(n_2\).

At whichever look supplies the reported estimate, suppose the ordinary two-sided 95% interval is used:

Under the fixed-sample normal model, \((\hat{\theta}_k-\theta)/SE_k\sim N(0,1)\), giving the interval its familiar nominal coverage. For the calculations below, take the true value to be \(\theta=0\). Under this value, \(W_1\) and \(W_2\) are standard normal, and their correlation is \(\rho=\sqrt{n_1/n_2}\). This follows because the first \(n_1\) observations are included in both stage estimates; the later observations are independent of them. Thus the two stage statistics are not independent.

Why the Conditional Coverage Changes

The selection event \(W_1\geq c\) removes all estimates below the boundary from the early-stopped group. Under the null, an estimate from that group is therefore biased upward as a selected value, even though the unselected stage-one estimator is unbiased. The standardized estimation error is no longer distributed as an unrestricted standard normal within the selected group.

The calculation above can be justified directly from the definition of conditional probability. Under \(\theta=0\), interval coverage is \(-1.96\leq W_1\leq1.96\). Intersecting this event with the stopping event \(W_1\geq1.5\) leaves \(1.5\leq W_1\leq1.96\). Dividing the probability of that intersection by the probability of stopping gives the conditional coverage. In general, for \(0\leq c<1.96\),

This expression also shows how the effect depends on the boundary. When \(c\) approaches 1.96 from below, the interval-compatible part of the stopping region shrinks toward zero. For \(c>1.96\), every early-stopped interval excludes zero. At \(c=1.96\), the interval includes zero only when \(W_1=1.96\), an event of probability zero under the continuous normal model; thus conditional coverage is zero for all \(c\geq1.96\). This is not a paradox: stopping for a sufficiently strong positive result selects estimates outside the interval’s null-containing range.

Coverage Across Both Looks

Coverage among early-stopped trials is not the same quantity as coverage across all trials. In the two-look design, some trials report \(C_1\) after crossing the boundary and the rest report \(C_2\). The overall coverage probability adds the coverage probability on each disjoint stopping path. This overall probability can also differ from 95%, although it need not differ by as much as the coverage conditional on early stopping.

For a concrete calculation, take \(n_1=100\), \(n_2=200\), \(c=1.5\), and \(\theta=0\). Then \(\rho=\sqrt{100/200}=0.7071\), and the standard deviation of either stage statistic is 1. The first-look coverage contribution is \(P(1.5\leq W_1\leq1.96)=\Phi(1.96)-\Phi(1.5)\). If the trial continues, it covers when \(-1.96\leq W_2\leq1.96\) and \(W_1<1.5\). Since the conditional distribution of \(W_2\) given \(W_1=x\) is normal with mean \(\rho x\) and variance \(1-\rho^2\), overall coverage is

Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. Numerical evaluation of the integral with \(\rho=0.7071\) gives overall coverage of approximately \(0.9394\). Thus the reported ordinary interval covers in about 93.9% of all trials under this design and null value, rather than 95%. This is distinct from the approximately 62.6% coverage among early-stopped trials: the overall result averages over both stopping paths according to how often each occurs.

Diagnostics, Alternatives, and Limitations

The central diagnostic is to write the stopping event and the interval coverage event separately, then check their overlap. If an analysis reports an ordinary interval after stopping, ask whether its nominal calibration was derived for the sampling plan actually used or only for a fixed sample size. In simulations, a direct diagnostic is to repeatedly generate complete trials under a specified \(\theta\), apply the stopping rule, construct the reported intervals, and compute the fraction containing \(\theta\). The simulation must reproduce the planned looks, boundaries, estimator, and variance treatment.

This normal example assumes independent observations, a known common variance, prespecified looks and boundary, and no other adaptations that alter the data or reporting rule. Real designs may use estimated standard errors, nuisance parameters, discrete outcomes, or additional stopping rules. Those features change the calculation, but they do not make the fixed-sample coverage claim automatically valid. In particular, an estimated standard error addresses uncertainty in scale estimation; it does not by itself account for selection caused by stopping.

A stopping-aware interval is an alternative: its construction incorporates the stopping rule into the sampling distribution used for calibration. Another route is to invert tests whose rejection probabilities are calibrated under that rule. These approaches require choices about the coverage target and can involve more computation. The ordinary fixed-sample interval remains useful as a transparent descriptive summary, but after data-dependent stopping its usual nominal coverage should not be asserted without justification. The precise construction and differing coverage targets require separate treatment.

• Mistake: “The estimate is unbiased at each fixed look, so its ordinary interval must still have 95% coverage after stopping.” Correction: unbiasedness at a fixed look does not imply that the estimate conditional on a data-dependent stopping event has the same distribution.
• Mistake: “The interval contains the observed estimate, so it covers the true effect.” Correction: an interval is centered on the estimate by construction; coverage is whether it contains the unknown parameter over repeated trials.
• Mistake: “The early-stop result represents all trials.” Correction: coverage among early-stopped trials and coverage across all stopping paths are different probabilities and should be labelled separately.