Why Repeated Looks Change Interval Coverage
In a group sequential trial, treatment effects may be estimated at several planned interim analyses as information accumulates. An interval with 95% coverage at each individual look is not automatically a set of intervals that covers the true effect with probability 95% at every look. The distinction matters when investigators inspect estimates repeatedly, present results from several looks, or stop the trial and report the estimate from the stopping look.
This tutorial treats a prespecified sequence of information levels and the standard canonical normal model for a scalar treatment effect. The target is simultaneous coverage across all planned looks. That is a stronger guarantee than coverage at one fixed look. A stopping-look-only interval has a different target and must account for how the stopping decision was made; the earlier material on confidence intervals after efficacy stopping addresses that setting.
Model and Dependence Across Looks
Assume the planned information levels \(I_1<\cdots<I_K\) are fixed in advance, and that, at look \(k\), the estimator satisfies \(\hat{\theta}_k\sim N(\theta,1/I_k)\). Information is expressed in inverse squared units of the treatment effect, so the standard error at look \(k\) is \(1/\sqrt{I_k}\). Under the canonical model, information increments are independent and normally distributed. Equivalently, the standardized estimation errors \(Z_k=\sqrt{I_k}(\hat{\theta}_k-\theta)\) have a joint multivariate normal distribution with mean zero and $$ \operatorname{Corr}(Z_j,Z_k)=\sqrt{\frac{I_j}{I_k}},\qquad j<k. $$ The looks are therefore dependent: later estimates reuse the data accumulated at earlier looks.
These assumptions are appropriate, for example, for a prespecified sequence of analyses of a normally distributed treatment contrast with known variance, or for a score process adequately represented by the canonical model. The variance scale and the information schedule must be correctly specified. If information levels or analysis times are changed in response to accumulating data, the stated joint distribution may no longer apply without a design-specific adjustment.
Worked Example: Correlation Between Interim Estimates
Suppose a hypothetical trial has three planned information levels \(I_1=25\), \(I_2=37.5\), and \(I_3=50\), measured in inverse mmHg squared for a blood-pressure treatment effect. Under the canonical model, $$ \operatorname{Corr}(Z_1,Z_2)=\sqrt{\frac{25}{37.5}}=0.8165,\qquad \operatorname{Corr}(Z_1,Z_3)=\sqrt{\frac{25}{50}}=0.7071, $$ and $$ \operatorname{Corr}(Z_2,Z_3)=\sqrt{\frac{37.5}{50}}=0.8660. $$
These positive correlations arise because each later estimate includes much of the earlier trial data. Treating the three looks as independent would therefore give the wrong joint coverage calculation. The correlation structure is part of the repeated-interval design, not a minor computational detail.
Constructing Simultaneous Intervals
A direct construction uses one critical value \(c\) chosen for the joint distribution of all standardized errors. At look \(k\), report \(C_k=[\hat{\theta}_k-c/\sqrt{I_k},\hat{\theta}_k+c/\sqrt{I_k}]\). Choose \(c\) so that \(P(\max_{1\leq k\leq K}|Z_k|\leq c)\geq 1-\alpha\). If the joint normal distribution is used, the smallest such \(c\) can be found from its multivariate normal distribution and the specified correlations.
The coverage claim follows by standardization: \(\theta\in C_k\) exactly when \(|\sqrt{I_k}(\hat{\theta}_k-\theta)|\leq c\), or \(|Z_k|\leq c\). Thus the event that all intervals contain \(\theta\) is the event that the maximum absolute standardized error is at most \(c\). A joint critical value targets this maximum directly.
A simple alternative is Bonferroni allocation. Choose nonnegative \(\alpha_1,\ldots,\alpha_K\) with \(\sum_k\alpha_k\leq\alpha\), and use the pointwise normal critical value \(z_{1-\alpha_k/2}\) at look \(k\). At each look, the probability of missing \(\theta\) is \(\alpha_k\). By the union bound, the probability of at least one miss is no more than \(\sum_k\alpha_k\); no independence assumption is needed. Bonferroni intervals are often wider than intervals based on the exact joint distribution, but they are transparent and robust to the correlation among looks.
Worked Example: A Bonferroni Band Across Three Looks
Suppose a hypothetical trial estimates a mean blood-pressure difference in mmHg at information levels \(25\), \(37.5\), and \(50\) inverse mmHg squared. The estimates are \(2.40\), \(2.10\), and \(1.80\) mmHg. For an overall error probability \(\alpha=0.05\), allocate \(\alpha_k=0.05/3=0.01667\) to each of the three looks. The two-sided critical value is approximately \(z_{1-0.05/6}=2.394\).
The standard errors are \(1/\sqrt{25}=0.2000\), \(1/\sqrt{37.5}=0.1633\), and \(1/\sqrt{50}=0.1414\) mmHg. The interval margins are therefore \(2.394(0.2000)=0.4788\), \(2.394(0.1633)=0.3909\), and \(2.394(0.1414)=0.3386\) mmHg.
The three repeated intervals are $$ [2.40-0.4788,\,2.40+0.4788]=[1.921,\,2.879]\text{ mmHg}, $$ $$ [2.10-0.3909,\,2.10+0.3909]=[1.709,\,2.491]\text{ mmHg}, $$ and $$ [1.80-0.3386,\,1.80+0.3386]=[1.461,\,2.139]\text{ mmHg}. $$
Each interval has marginal coverage of approximately \(98.33\%\). The probability that any of the three misses is at most \(3(0.01667)=0.05\), so the probability all three cover is at least 95%. This guarantee does not require the looks to be independent.
Why Pointwise Intervals Are Not a Repeated Band
A conventional 95% interval at a prespecified fixed look has 95% marginal coverage under its model. Repeating that construction at \(K\) looks does not make the probability of covering at every look equal to 95%. If the look-specific errors were independent, the probability that all \(K\) nominal 95% intervals covered would be \(0.95^K\); for three looks this is \(0.95^3=0.8574\). In a group sequential trial the errors are correlated, so this independence calculation is not the actual joint coverage. It still illustrates why marginal coverage alone cannot establish simultaneous coverage.
Worked Example: Comparing Pointwise and Simultaneous Margins
Use the information levels and estimates from the previous example, but now construct ordinary two-sided 95% intervals at each look. The pointwise critical value is \(1.96\). The margins are \(1.96(0.2000)=0.3920\), \(1.96(0.1633)=0.3201\), and \(1.96(0.1414)=0.2772\) mmHg.
The pointwise intervals are \([2.008,2.792]\), \([1.780,2.420]\), and \([1.523,2.077]\) mmHg. Each is designed for 95% coverage at its own fixed look. However, these calculations do not show that the three intervals jointly cover with probability 95%, nor do they guarantee 95% coverage for whichever one is selected after reviewing the data. The Bonferroni intervals in the preceding example are wider because they control the chance of a miss across the whole set of looks.
Simultaneous Coverage Versus Stopping-Look Coverage
A simultaneous band is designed to cover at every planned look, including looks that will not ultimately be reported. Its guarantee automatically applies to the interval at a random stopping look \(T\): if all planned intervals cover, then the selected one covers. This is useful when interim estimates are examined, discussed, or reported as the trial proceeds, or when one wants protection regardless of which planned look triggers stopping.
A stopping-look-only interval targets coverage for the estimate reported when the trial stops. It need not cover at unused looks, and it can use the particular stopping rule to obtain a less conservative interval. But a usual fixed-look interval cannot simply be selected after a data-dependent stopping decision and assumed to retain its nominal coverage. One general valid construction is to invert tests that already account for the full stopping rule: include a candidate value \(\theta_0\) in the confidence set if the observed sequence would not reject \(H_0:\theta=\theta_0\) under that sequential test. If the test rejects a true \(\theta\) with probability at most \(\alpha\), the inverted set covers the true value with probability at least \(1-\alpha\). Such a set is adjusted to the stopping design and is not necessarily identical to a simultaneous band.
Worked Example: Allocating Error Unequally Across Two Looks
Consider a hypothetical two-look trial estimating a treatment effect in mmHg, with \(I_1=20\), \(I_2=40\), and estimates \(1.20\) and \(1.00\) mmHg. Allocate \(\alpha_1=0.01\) and \(\alpha_2=0.04\), so the total is \(0.05\). The two-sided normal critical values are \(z_{0.995}=2.576\) and \(z_{0.98}=2.054\).
At the first look, the standard error is \(1/\sqrt{20}=0.2236\) mmHg, giving margin \(2.576(0.2236)=0.5760\) mmHg and interval \([1.20-0.5760,1.20+0.5760]=[0.624,1.776]\) mmHg.
At the second look, the standard error is \(1/\sqrt{40}=0.1581\) mmHg, giving margin \(2.054(0.1581)=0.3247\) mmHg and interval \([1.00-0.3247,1.00+0.3247]=[0.675,1.325]\) mmHg. The union bound limits the probability that either interval misses to \(0.01+0.04=0.05\). This allocation gives more protection to the first look; other allocations may be chosen to reflect which looks matter most, while preserving the total error bound.
Diagnostics, Choices, and Limitations
Before applying a repeated-interval calculation, verify that the number and timing of looks, information levels, outcome scale, and variance assumptions match the analysis plan. In practice, diagnostics include checking that the planned information fractions were achieved, that the interim estimates follow the assumed canonical dependence reasonably well, and that the reported intervals use the intended joint critical value or error allocation. A mismatch between planned and actual information, an unplanned additional look, or an incorrectly treated variance can invalidate the nominal coverage guarantee.
The main options have different costs and targets:
| Approach | Coverage target | Main trade-off |
|---|---|---|
| Joint-normal critical value | All prespecified looks simultaneously | Can be more efficient, but depends on a correctly specified joint model and numerical calibration |
| Bonferroni intervals | All prespecified looks simultaneously | Simple and dependence-robust, often conservative |
| Stopping-rule test inversion | Coverage for the result under the specified stopping design | Can use the stopping rule, but the resulting set may be design-specific and need not cover unused looks |
| Ordinary pointwise interval | One prespecified look | Does not by itself protect against repeated inspection or data-dependent selection of a look |
Neither a simultaneous band nor a stopping-rule-adjusted interval protects against every possible change in the analysis. The guarantee applies to the parameter, model, planned looks, and stopping or reporting procedure used in its construction. Repeated intervals across looks also do not, by themselves, address multiplicity across several treatment arms or subgroups; those are separate sources of selection.
Common Mistake
Wrong reasoning: “Every look has a 95% confidence interval, so the interval reported when the trial stops has 95% coverage, and all the intervals together have 95% coverage.” This confuses marginal coverage at a fixed look with coverage after repeated inspection or selection. Correction: specify the coverage target first. Use a simultaneous band if all planned looks must be protected, or invert a test calibrated for the stopping rule if inference is required only for the stopping result. Do not infer either guarantee from pointwise intervals alone.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In one sentence, distinguish 95% pointwise coverage at each look from 95% simultaneous coverage across all looks.
- A trial has four planned looks and uses Bonferroni allocation with total \(\alpha=0.04\). What is the per-look two-sided error allocation if it is equal across looks?
- Under the canonical model, \(I_1=30\) and \(I_2=60\). What is the correlation between the two standardized estimation errors?
- Why does simultaneous coverage across all planned looks protect the interval at a data-dependent stopping look?
- Does a standard 95% interval at a fixed look automatically retain 95% coverage when that look is selected because the trial stopped there? Explain.
Solutions. (1) Pointwise coverage refers to each interval separately; simultaneous coverage is the probability that every interval in the planned set covers, and is at least 95%. (2) The allocation is \(0.04/4=0.01\) per look, or 1% two-sided error per look. (3) The correlation is \(\sqrt{30/60}=\sqrt{0.5}=0.7071\). (4) Whenever every planned interval covers, the interval at whichever planned look was selected also covers; the simultaneous event is a subset of the event that the selected interval covers. (5) No. Selection through a data-dependent stopping rule can change the distribution of the reported estimate; use a simultaneous construction or an interval calibrated for the specified stopping rule.