Unconditional Confidence Intervals Across Stopping Stages

Why Calibrate Across Stopping Stages?

A confidence interval calculated as though the observed sample size had been fixed in advance can have the wrong coverage after sequential monitoring. The reason is selection: the estimate and the stage at which sampling stops are linked by the stopping rule. A procedure calibrated only among trials that stop at one particular stage answers a different coverage question from one calibrated over repeated trials that may stop for efficacy, stop for futility, or reach maximum information.

Here the target is unconditional coverage: under a specified parameter value, what proportion of repeated trials would produce an interval containing that value, allowing the stopping stage to vary according to the design? The construction below uses the distribution of the estimator at its actual stopping stage. This extends the earlier material on conditional versus unconditional confidence coverage and on inverting stopping-aware tests.

Definition: For a parameter \(\theta\) and a confidence interval \(C\) computed after applying a specified stopping rule, its unconditional coverage at \(\theta\) is \(P_\theta\{\theta\in C\}\), where the probability includes the data path and the resulting stopping stage. A nominal \(1-\alpha\) interval has exact unconditional coverage if this probability equals \(1-\alpha\) for every parameter value in the model; it has at least nominal coverage if the probability is no less than \(1-\alpha\).

Model the Stopped Experiment

For a clear derivation, suppose the endpoint follows a canonical normal group-sequential model with known variance scale. At look \(j\), information is \(I_j\), and \(X_j\) is the accumulated score. Assume \(0<I_1<\cdots<I_K\), independent increments, and \(X_j-X_{j-1}\sim N(\theta(I_j-I_{j-1}), I_j-I_{j-1})\), with \(X_0=I_0=0\). Thus \(X_j\sim N(\theta I_j,I_j)\), and the stage-\(j\) estimate is \(\hat\theta_j=X_j/I_j\). The parameter \(\theta\) and estimate are measured in the endpoint’s effect units per unit of information.

The planned boundaries and stopping rule are fixed before analysis. Let \(A_j\) denote the set of data paths that stop at look \(j\), whether for efficacy, futility, or completion at the maximum. These regions are disjoint and cover all possible paths. The observed stopping stage is \(T\), and the reported estimator is \(\hat\theta_T=X_T/I_T\). The normal independent-increment model, known information values, complete specification of the stopping rule, and continuous sampling distribution are assumptions of the construction. If these assumptions are not reasonable, the resulting calibration need not apply.

The distribution function of the stopped estimator is \(F_\theta(y)=P_\theta(\hat\theta_T\le y)\). It combines the probability of stopping at each stage with the estimator’s distribution within that stopping region:

F_\theta(y)=\sum_{j=1}^{K}P_\theta\left(A_j,\frac{X_j}{I_j}\le y\right).

The joint event in each term matters. Multiplying the chance of stopping at stage \(j\) by an ordinary fixed-sample distribution for \(\hat\theta_j\) generally does not give that term: the stopping event itself depends on the data that determine the estimate.

Equal-Tailed Unconditional Inversion

For a candidate value \(\theta\), evaluate \(F_\theta(y_{\mathrm{obs}})\), where \(y_{\mathrm{obs}}\) is the estimate at the observed stopping stage. If the distribution is continuous, this probability-transform value is uniform on \((0,1)\) when the candidate is the true parameter. This gives an equal-tailed test: values for which \(F_\theta(y_{\mathrm{obs}})<\alpha/2\) or \(F_\theta(y_{\mathrm{obs}})>1-\alpha/2\) are excluded from a \(1-\alpha\) confidence set.

C(y_{\mathrm{obs}})= \left\{\theta:\frac{\alpha}{2}\le F_\theta(y_{\mathrm{obs}}) \le 1-\frac{\alpha}{2}\right\}.

If \(F_\theta(y_{\mathrm{obs}})\) is suitably monotone in \(\theta\), the set has endpoints \(L\) and \(U\) that solve \(F_L(y_{\mathrm{obs}})=1-\alpha/2\) and \(F_U(y_{\mathrm{obs}})=\alpha/2\), respectively. The reversal of the tail probabilities occurs because increasing \(\theta\) shifts the estimator distribution upward: a large candidate value makes an observed estimate look unusually low. Monotonicity and the existence of these roots should be checked for the design and model in use; if they fail, the inverted set need not be a single interval.

Specify the model and design.
Record the information levels, the increment distribution, and the complete efficacy, futility, and maximum-information stopping regions.

Evaluate the stopped distribution.
For each candidate \(\theta\), calculate \(F_\theta(y_{\mathrm{obs}})\), integrating over every stopping region rather than conditioning on the observed one.

Invert the equal-tailed test.
Find the candidate values at which the distribution function reaches \(1-\alpha/2\) and \(\alpha/2\). Check roots, monotonicity, and numerical accuracy.

Derivation and Numerical Calibration

The coverage justification follows directly from inversion. Under the true value \(\theta\), let \(Y=\hat\theta_T\). If \(F_\theta\) is continuous, then \(F_\theta(Y)\sim\mathrm{Uniform}(0,1)\). Therefore:

P_\theta\{\theta\in C(Y)\} =P_\theta\left\{\frac{\alpha}{2}\le F_\theta(Y)\le1-\frac{\alpha}{2}\right\} =1-\alpha.

The equality uses the fact that a uniform random variable falls between \(\alpha/2\) and \(1-\alpha/2\) with probability \(1-\alpha\). This is a repeated-sampling result under the stated model, not a probability statement about the parameter after observing the interval. For a discrete estimator, the probability transform need not be uniform; tail probabilities or other adjustments must then be chosen to achieve the intended coverage, possibly conservatively.

For a two-look design, with \(X_1=x\) and increment \(X_2-X_1=u\), the distribution can be evaluated by integrating over the actual stopping regions. For example, if \(A_1\) is the region stopping at look 1 and \(A_1^c\) continues to look 2:

F_\theta(y)= \int_{A_1\cap\{x/I_1\le y\}} f_{\theta,I_1}(x)\,dx +\iint_{A_1^c\cap\{(x+u)/I_2\le y\}} f_{\theta,I_1}(x)f_{\theta,I_2-I_1}(u)\,du\,dx,

where \(f_{\theta,v}\) is the normal density with mean \(\theta v\) and variance \(v\). If the design distinguishes efficacy and futility stops at the first look, both are included in \(A_1\); the continuation integral covers only paths that proceed. This representation makes explicit why ordinary fixed-sample quantiles do not generally calibrate the stopped estimator.

Worked Example: Unconditional Coverage Is a Weighted Average

Suppose a hypothetical two-look trial has three possible outcomes: efficacy stopping, futility stopping, or completion at the maximum information. Under a particular parameter value, integration over the specified normal model and boundaries gives stage probabilities \(0.20\), \(0.15\), and \(0.65\). A proposed interval includes the true value on \(88\%\), \(96\%\), and \(98\%\) of paths within those respective stages.

The overall coverage is the sum of the joint probabilities of stopping at a stage and covering the parameter. Equivalently, it is the stage probability times conditional coverage, summed over stages:

P_\theta(\theta\in C) =(0.20)(0.88)+(0.15)(0.96)+(0.65)(0.98) =0.176+0.144+0.637=0.957.

Thus the interval has \(95.7\%\) unconditional coverage at this parameter value, even though its conditional coverage differs by stage. The arithmetic is a coverage decomposition, not a calibration recipe by itself: the stage-specific probabilities must come from the model and the actual stopping rule.

Worked Example: Inverting a Stopped-Estimator Distribution

Consider a hypothetical analysis with estimate \(y_{\mathrm{obs}}=0.50\) log-odds units and target coverage \(95\%\). Numerical integration over all stopping regions gives the following values near the equal-tailed roots. These are illustrative model-based calculations, not results from a real trial.

For the lower endpoint, the target is \(1-\alpha/2=0.975\). Suppose \(F_{0.08}(0.50)=0.981\) and \(F_{0.10}(0.50)=0.970\). Linear interpolation gives:

L\approx 0.08+ \frac{0.981-0.975}{0.981-0.970}(0.10-0.08) =0.0909\ \text{log-odds units}.

For the upper endpoint, the target is \(\alpha/2=0.025\). Suppose \(F_{0.88}(0.50)=0.029\) and \(F_{0.90}(0.50)=0.021\). Then:

U\approx 0.88+ \frac{0.029-0.025}{0.029-0.021}(0.90-0.88) =0.890\ \text{log-odds units}.

The approximate unconditional equal-tailed interval is therefore \([0.0909,\,0.890]\) log-odds units. In an actual analysis, a root-finding method would evaluate the distribution repeatedly rather than rely on two-point linear interpolation. Rounding and integration error should also be checked against the desired precision.

Diagnostics, Comparisons, and Limitations

Calibration is design-specific. It must use the boundaries actually applied, including whether futility stopping is binding, and the information levels actually planned. A useful diagnostic is to evaluate \(P_\theta\{\theta\in C\}\) over a grid of parameter values using numerical integration or simulation that reproduces the full stopping rule. Under the ideal continuous model and exact inversion, the target coverage follows from the probability transform; noticeable departures can signal inaccurate integration, incorrectly coded stopping regions, or a mismatch between the assumed and implemented design.

Compare the unconditional construction with a fixed-sample interval and a stage-conditional interval. A fixed-sample interval uses a reference distribution that ignores selection by stopping. A stage-conditional interval calibrates coverage among trials stopping at the observed stage. The unconditional interval instead targets coverage over the full repeated-trial population, including all stages. These procedures answer different questions; unconditional calibration does not promise nominal coverage within every individual stage.

The method also has limits. It relies on a specified sampling model and stopping rule, and exact calculations can be demanding when there are many looks, complicated boundaries, or nuisance parameters. Plugging in estimated information or variance without accounting for its uncertainty can change coverage. Moreover, if inversion is not monotone, the confidence set may be disconnected; reporting its interval hull would include extra parameter values and is not the same procedure. These cases call for explicit checks, not silent substitution of fixed-sample limits.

Worked Example: Check a Proposed Stage-Wise Calibration

For a second hypothetical design, suppose integration at a chosen parameter value gives stage probabilities \(0.30\), \(0.10\), and \(0.60\) for efficacy stop, futility stop, and maximum completion. A proposed interval covers conditionally with probabilities \(0.90\), \(0.94\), and \(0.98\).

Its unconditional coverage at that parameter is:

$$ (0.30)(0.90)+(0.10)(0.94)+(0.60)(0.98) =0.270+0.094+0.588=0.952. $$

The result is \(95.2\%\), close to but not exactly \(95\%\). Because the component probabilities are rounded, this arithmetic alone cannot establish exact calibration. The calculation must be repeated at relevant parameter values with adequate numerical precision. A nominal label is justified by the procedure’s calibrated operating characteristics, not by a single rounded example.

Common Pitfall

Common mistake: “The trial stopped at the efficacy boundary, so I should calculate coverage only among efficacy stops; that is the unconditional coverage of the reported interval.”

Correction: That calculation is conditional on efficacy stopping. Unconditional coverage sums coverage across every possible stopping stage, weighted by each stage’s probability under the parameter value. The observed stage selects which estimate is reported, but it does not redefine the repeated-trial target.

Check Your Understanding

Answer each question using the model and definitions above.

A design has three stopping outcomes with probabilities \(0.25\), \(0.25\), and \(0.50\). Conditional coverages are \(0.92\), \(0.96\), and \(0.98\). What is unconditional coverage?
Why can one not generally calculate the stopped-estimator distribution by multiplying each stage’s stopping probability by a fixed-sample distribution for that stage?
For a \(90\%\) equal-tailed interval, what distribution-function values define the lower and upper endpoints?
What does the probability-transform argument establish, and what continuity condition does it require?
If the inverted set is not a single interval, why is reporting its interval hull a change to the procedure?

Solutions. (1) \(0.25(0.92)+0.25(0.96)+0.50(0.98)=0.230+0.240+0.490=0.960\), or \(96.0\%\). (2) The stopping event depends on the same data that determine the estimate, so the estimate’s distribution within a stage is selected and need not be its fixed-sample distribution. (3) The lower endpoint solves \(F_\theta(y_{\mathrm{obs}})=0.95\); the upper solves \(F_\theta(y_{\mathrm{obs}})=0.05\). (4) Under the true parameter, it makes \(F_\theta(\hat\theta_T)\) uniform, giving exact equal-tailed coverage; the stopped estimator must have a continuous distribution for this direct argument. (5) The hull adds parameter values lying between disconnected components, changing the confidence set and its coverage behavior.