Confidence Intervals for Two-Look Designs

Why Two-Look Intervals Need to Account for Stopping

In a two-look design, the reported estimate may come from an interim analysis or from a later, final analysis. Those estimates have different sampling distributions: the interim estimate is based on fewer observations and is generally less precise. Also, the analysis time is selected by a rule that depends on the interim data. An ordinary confidence interval calculated only at the observed look does not, by itself, account for that selection.

This tutorial develops a direct, conservative solution for a normal endpoint with known variance. Construct a confidence interval at each possible look, allocating error across the two intervals so that they cover simultaneously with at least the desired probability. Then report the interval corresponding to the look actually reached. The guarantee holds whatever prespecified stopping rule selects the look, provided the model and sampling assumptions are satisfied.

Write \(\bar{X}_1\) for the interim sample mean and \(\bar{X}_2\) for the cumulative mean at the final look. Their marginal sampling distributions are

Thus the standard errors are \(SE_1=\sigma/\sqrt{n_1}\) and \(SE_2=\sigma/\sqrt{n_2}\). These estimates are not independent: the first \(n_1\) observations contribute to both. In fact, \(\operatorname{Cov}(\bar{X}_1,\bar{X}_2)=\sigma^2/n_2\), so their correlation is \(\sqrt{n_1/n_2}\). The method below does not require independence between the looks.

Constructing a Simultaneous Pair of Intervals

Let the desired overall confidence level be \(1-\alpha\). Allocate a total error probability of at most \(\alpha/2\) to each look. Since each interval is two-sided, its probability of missing \(\theta\) in either tail is at most \(\alpha/2\), with at most \(\alpha/4\) assigned to each tail. Define \(z_q\) as the \(q\)-quantile of the standard normal distribution and set \(c=z_{1-\alpha/4}\). The two look-specific intervals are

For the prescribed normal model, each interval has marginal coverage \(1-\alpha/2\). If the stopping rule stops at the interim look, report \(C_1\); if it continues, report \(C_2\). The central justification is that simultaneous coverage of both intervals guarantees coverage of whichever one the rule selects.

To see why, let \(A_k\) be the event that \(C_k\) contains \(\theta\). The normal marginal distribution gives \(P_\theta(A_k^c)=\alpha/2\) for each look. The union bound, which does not require the two events to be independent, gives

Whenever both intervals cover, the interval reported at the stopping look covers as well. If \(T\) denotes the look reached, then \(\{\theta\in C_1\cap C_2\}\) is a subset of \(\{\theta\in C_T\}\). Consequently,

This is a simultaneous-coverage argument, not an assertion that the reported interval has exactly \(1-\alpha\) coverage. The union bound can be conservative, and the amount of conservatism depends in part on the dependence between the two look-specific estimates and on the stopping rule.

Worked Calculations at Each Possible Look

What the Guarantee Does—and Does Not—Mean

The procedure gives a repeated-sampling coverage guarantee averaged over all data paths and the stopping decisions they produce. It does not generally guarantee at least 95% coverage among trials that stop at look 1, nor among those that reach look 2. Conditioning on a particular stage changes the distribution of the estimate because the stage is selected using the interim data. Conditional coverage is a different target and requires a method designed for that target.

The procedure also does not claim that the reported interval is the shortest possible interval with the stated coverage. It protects two intervals simultaneously, so it pays a price for guarding against a miss at either look. For comparison, the introductory material on exact intervals for a single-look normal design gives the usual fixed-look construction; simply applying that construction at the observed look does not provide the simultaneous protection established here. Stopping-aware test inversion, covered in the earlier material on inverting stopping-aware tests, can use the stopping rule more directly and may yield a less conservative interval. This tutorial’s construction is transparent and applies to any prespecified two-look rule without requiring independence between looks.

Diagnostics, Limitations, and Common Mistakes

Before using this interval, verify that the analysis matches the assumptions: the endpoint model is normal with known variance, the planned sample sizes are fixed, the final estimate is cumulative, and the interim stopping decision follows the prespecified rule. A final mean calculated only from the observations added after the interim look is not \(\bar{X}_2\), so substituting it into the displayed final interval would use the wrong sampling distribution. Also check the standard errors separately; the interim interval should be wider when \(n_1<n_2\).

The normal critical value is not automatically justified if \(\sigma\) is estimated, the endpoint distribution is materially nonnormal, looks occur at unplanned times, or sample size changes in response to interim results. In those settings, the marginal distributions and simultaneous error allocation must be recalibrated to the actual design. Likewise, if additional looks are possible, allocating error across only two intervals does not protect the additional analyses.

• Mistake: using the ordinary 95% critical value at both looks. Two marginal 95% intervals do not in general give at least 95% simultaneous coverage. Correction: allocate error across looks, as in \(c=z_{1-\alpha/4}\) for two two-sided intervals.
• Mistake: treating the two estimates as independent. They share observations. Correction: the union-bound proof needs only the marginal error probabilities, not independence.
• Mistake: rounding the critical value down. At \(\alpha=0.05\), \(2.241\) is slightly below \(z_{0.9875}=2.2414027\ldots\); each interval then misses with probability slightly greater than 0.025. Correction: calculate with the unrounded quantile or round upward, for example to 2.242.
• Mistake: claiming conditional 95% coverage at each stopping stage. The theorem is unconditional over the stopping process. Correction: state the coverage target accurately and use a method specifically calibrated for conditional coverage when that is the scientific requirement.