Why the Stopping Stage Changes the Coverage Question
A confidence interval reported after a group sequential trial is a random result of the whole design: the data evolve across interim looks, the stopping rule determines when the trial ends, and the interval is calculated from the observations available at that time. Coverage can therefore be assessed in two different ways. One asks how often the interval contains the true parameter among trials stopping at a particular look. The other asks how often it contains the parameter across all trials, allowing each stopping stage to occur with its design probability.
These are not competing definitions of the same conditional probability. They describe different repeated-sampling targets. An interval can have good average coverage over all stopping stages while performing poorly among trials that stop early. Conversely, if coverage is at its target separately at every possible stopping stage, then its overall coverage is at that target too.
Here, “conditional” means conditional on the event that the procedure stops at stage \(k\), not conditional on a particular value of the test statistic or on all the data observed at that stage. The interval and the stopping event are both generated by the trial’s sampling process. The parameter \(\theta\) is held fixed when these probabilities are calculated; the probabilities describe repeated trials under that parameter value.
A nominal \(100(1-\alpha)\%\) coverage target is a probability target, such as 0.95 when \(\alpha=0.05\). To assess it, the stopping rule, analysis, and interval procedure must be specified. Their joint repeated-sampling distribution determines the coverage. In particular, the stage probabilities and the conditional coverage rates can both depend on \(\theta\).
The Relationship Between the Two Targets
The stopping stages partition all possible trial outcomes: each completed trial stops at exactly one stage. The law of total probability therefore gives unconditional coverage as a weighted average of the stage-specific conditional coverages.
This identity does not require independent looks, normally distributed statistics, or a particular form of interval. It requires a well-defined sampling distribution for the complete design and a partition of completed trials by stopping stage. Any dependence between the data and the stopping decision is already reflected in \(P_\theta(\theta\in C\mid K=k)\). If \(P_\theta(K=k)=0\), conditional coverage at stage \(k\) is undefined at that parameter value, although that stage contributes zero to unconditional coverage.
The distinction matters especially after stopping because the data leading to a particular stage are selected by the stopping rule. For example, trials that stop early for efficacy tend to have interim results that crossed an efficacy boundary. The conditional distribution of the data in that subset need not match the distribution assumed by a fixed-sample calculation. This selection can affect the chance that an interval contains \(\theta\). The earlier material on coverage failure of fixed-sample intervals after stopping addresses why ignoring the stopping rule can be problematic; here the focus is on specifying which coverage probability is being targeted.
Worked Example: Averaging Coverage Across Three Stages
Suppose a hypothetical three-stage design has stopping probabilities 0.25, 0.35, and 0.40 at a fixed value of \(\theta\). Under the complete procedure, suppose the interval’s conditional coverage rates at those stages are 0.80, 0.92, and 0.97. Calculate the unconditional coverage.
First check that the stage probabilities form a partition: \(0.25+0.35+0.40=1.00\). Apply the weighted-average identity:
Thus, the unconditional coverage is 0.910, or 91.0%, at this parameter value. The stage-specific coverages range from 80.0% to 97.0%, so none of those values should be replaced by the overall result when the question concerns a particular stopping stage. The calculation is coherent because it uses probabilities from the same hypothetical design and the same fixed \(\theta\).
Why One Coverage Target Does Not Guarantee the Other
Unconditional coverage is an average, not a promise about every subgroup of trials. A stage that occurs rarely receives little weight in the average; a stage that occurs often receives more. This permits a low conditional coverage at one stage to be offset by higher coverage at another. The reverse is also possible: high coverage at one stage can coexist with low overall coverage if coverage is poor at stages that occur frequently.
Worked Example: Nominal Overall Coverage With Weak Early-Stage Coverage
Consider a hypothetical two-stage procedure at a specified \(\theta\). Suppose 30% of trials stop at stage 1 and 70% stop at stage 2. The interval covers \(\theta\) in 75% of stage-1 trials and 95% of stage-2 trials. Find the overall coverage and compare it with a 95% target.
The joint probability of stopping at stage 1 and covering is \(0.30\times0.75=0.225\). At stage 2 it is \(0.70\times0.95=0.665\). Since these events are disjoint, add them:
The unconditional coverage is 89.0%, below 95.0%; the stage-2 coverage alone does not establish nominal overall coverage. The stage-1 conditional coverage is only 75.0%, which is a separate and more pronounced shortfall for trials stopping early. This example illustrates two distinct comparisons: overall coverage against an unconditional target, and stage-specific coverage against a conditional target.
The weighted-average identity also shows exactly what conditional performance would be needed to reach an overall target, if the other stage rates and stage probabilities were known. This arithmetic does not provide an interval-construction method; it clarifies the implications of the chosen coverage target.
Worked Example: Required Coverage at One Stage for a Given Overall Target
A hypothetical three-stage design has stopping probabilities 0.20, 0.30, and 0.50. Suppose the conditional coverages at stages 2 and 3 are 0.95 and 0.98. What stage-1 conditional coverage would make the unconditional coverage exactly 0.95?
Let \(c_1=P_\theta(\theta\in C\mid K=1)\). Substitute the known weights and coverage rates into the identity, then solve:
The required stage-1 conditional coverage is 87.5%. It is lower than the 95% overall target because the other two stages have coverage above 95%. Thus, achieving 95% unconditional coverage does not imply 95% conditional coverage at each stage. If this calculation produced a value below 0 or above 1, the stipulated stage probabilities and other coverage rates could not yield the target through any possible stage-1 coverage rate.
Assumptions, Diagnostics, and Interpretation
Coverage calculations are only meaningful when they correspond to the actual procedure. The stopping rule, the maximum number of looks, the treatment of efficacy and futility boundaries, and the interval reported at each stopping stage all need to be represented in the sampling distribution. If the design or analysis changes, its stopping probabilities and conditional coverage rates may change too. For a procedure with estimated standard errors or other random components, those components must also be included in the repeated-sampling calculation.
A practical diagnostic is to examine the stage-specific rates alongside the overall rate over parameter values relevant to the analysis. Exact calculation, enumeration, or simulation may be used when appropriate to the model and design. In simulation, estimate a conditional rate at stage \(k\) by counting coverage among simulated trials with \(K=k\), and estimate unconditional coverage by counting coverage among all trials. Also report how many trials reached each stage: a conditional estimate based on very few simulated trials can be unstable. These are diagnostics of a specified procedure, not guarantees that its true coverage equals the simulation estimate.
Coverage is a property of the repeated-sampling procedure at a fixed \(\theta\), not a probability statement about the parameter after one interval has been observed. For instance, 95% unconditional coverage means that across repeated trials under that parameter and design, 95% of the reported intervals would contain it. It does not mean there is a 95% probability that the parameter lies in the particular interval from the completed trial.
| Question being asked | Coverage quantity | What it averages over |
|---|---|---|
| Among trials stopping at stage \(k\), how often does the interval cover? | \(P_\theta(\theta\in C\mid K=k)\) | Repeated trials conditional on stopping at stage \(k\) |
| Across all trials, how often does the interval cover? | \(P_\theta(\theta\in C)\) | All stages, weighted by their stopping probabilities |
These targets serve different purposes. Unconditional coverage assesses the complete procedure in the population of repeated trials governed by the design. Conditional coverage addresses reliability within a specified stopping-stage subset, which may be important when interpreting an interval after observing that stage. Neither should be described as universally preferable without stating the inferential objective. A conditional target at every stage is stronger than an unconditional target, but it can require different operating characteristics; an unconditional target alone permits uneven stage-specific performance.
Common Pitfalls
- Treating an overall rate as a stage-specific guarantee. An overall 95% coverage result does not establish 95% coverage among early stops. Report or assess the conditional rates when that is the question.
- Taking an unweighted average of stage rates. The overall rate weights each stage by \(P_\theta(K=k)\). An equal-weight average is correct only when the stage probabilities are equal.
- Conditioning on the wrong event. Conditioning on \(K=k\) means restricting to trials that stop at stage \(k\). Conditioning on a particular observed statistic is a different probability calculation.
- Ignoring parameter dependence. Stage probabilities and coverage rates can vary with \(\theta\). A coverage calculation at one value does not establish performance at every value.
- Reporting a conditional rate for an impossible stage. If \(P_\theta(K=k)=0\), conditional coverage at that stage is undefined at that \(\theta\); it is not a zero-coverage result.
Check Your Understanding
Answer each question using the coverage definitions and weighted-average identity.
- A two-stage design has stopping probabilities 0.40 and 0.60, with conditional coverages 0.90 and 0.96. What is unconditional coverage?
- If every stage with positive stopping probability has conditional coverage 0.93, what is unconditional coverage? State the reason.
- Can unconditional coverage of 0.95 coexist with conditional coverage of 0.80 at one stage? Explain what other stage coverage could do.
- What does \(P_\theta(\theta\in C\mid K=2)\) condition on, and what does it not mean?
Solutions. (1) \(0.40(0.90)+0.60(0.96)=0.36+0.576=0.936\), so unconditional coverage is 93.6%. (2) It is 0.93, because the weighted average of identical stage-specific rates is that same rate and the stage probabilities sum to 1. (3) Yes. A sufficiently high coverage rate at another stage can offset the 0.80 rate, with both rates weighted by their stopping probabilities; the overall rate alone does not reveal either conditional rate. (4) It conditions on the trial stopping at stage 2. It does not condition on a specific test statistic value or mean that the parameter has a probability of being in the observed interval.