Reading a Boundary as Part of a Design
A classical efficacy boundary is not just a list of cutoffs to apply whenever results happen to be available. It belongs to a design: the test direction, overall significance level, planned information fractions, and analysis schedule together determine the rejection rule. Reading a cutoff without those features can lead to incorrect claims about type I error or about which of two designs is more conservative.
This tutorial focuses on three recurring errors: treating each look’s cutoff as an independent alpha level, comparing designs at different information fractions as if their looks matched, and ignoring the schedule used to calibrate the boundary. The key is to evaluate the entire path of test statistics under the null, rather than interpreting each look in isolation.
For the standard canonical null model, each \(Z_k\) has a standard normal marginal distribution, and for \(i<j\) the correlation is \(\sqrt{t_i/t_j}\). Thus the statistics are generally dependent. This is why the overall error is not found by simply adding the marginal probabilities, nor is it generally equal to any one look’s nominal level \(P_{H_0}(Z_k\ge c_k)\). The classical cutoffs are interpreted together, under the joint null distribution and the stated schedule.
Why Look-Specific Cutoffs Are Not Separate Alpha Levels
At look \(k\), the nominal one-sided p-value is \(1-\Phi(Z_k)\), where \(\Phi\) is the standard normal cumulative distribution function. Comparing that p-value with a nominal level describes the marginal extremeness of \(Z_k\). It does not, by itself, describe the probability that the full sequential design will ever reject \(H_0\). The latter is a probability over all planned looks and accounts for their dependence.
A useful way to see the distinction is to write the probability of crossing either of two upper boundaries:
The overlap must be subtracted because a path that crosses both boundaries is still only one path that leads to rejection. For more looks, the same principle applies to the union of all crossing events. A boundary’s nominal p-value cutoff is therefore not a separate, independent allocation of the design’s error.
Worked Example: Two Nominal Cutoffs Do Not Define Two Independent Tests
Suppose a hypothetical one-sided design has two planned looks at \(t_1=0.5\) and \(t_2=1\), and uses \(c_1=c_2=1.96\). Under the canonical null model, each look has marginal crossing probability approximately \(0.0250\). Are these cutoffs evidence that the overall type I error is \(0.0250\), or that it is \(0.0500\)?
Neither conclusion follows from the marginal probabilities alone. The null correlation is $$ \operatorname{Corr}(Z_1,Z_2)=\sqrt{\frac{0.5}{1}}=0.7071. $$ Therefore the overall crossing probability is $$ 0.0250+0.0250-P_{H_0}(Z_1\ge1.96,\ Z_2\ge1.96). $$
The joint crossing probability is positive because the statistics are positively correlated. Evaluating the bivariate normal probability gives an overall crossing probability of about \(0.039\), not \(0.025\) and not \(0.050\). The two nominal probabilities cannot simply be treated as independent alpha levels; the full joint crossing probability is the quantity that answers the design-level question.
This calculation assumes the stated two-look schedule and the canonical null distribution. It would not establish the error rate for a design with different information fractions or extra analyses.
Match Information Fractions Before Comparing Designs
The look number is not enough to identify a point in a sequential design. “The first interim look” could occur at \(t=0.40\) in one design and \(t=0.75\) in another. These correspond to different amounts of information and different dependence with later statistics. For two looks, the null correlation is \(\sqrt{t_1/t_2}\); changing the information fractions changes that correlation even when the displayed critical values are unchanged.
Consequently, comparing the first cutoff from one schedule with the first cutoff from another is not a like-for-like comparison unless the information fractions and other design features match. A lower numerical cutoff at one interim look does not alone establish that a design is more liberal overall. The later boundaries, the dependence among looks, and the schedule all contribute to the chance of a crossing.
Worked Example: The Same Boundary Values Under Different Schedules
Consider two hypothetical two-look designs with an upper cutoff of \(2.00\) at each look. Design A plans looks at \(t_1=0.50\) and \(t_2=1\); Design B plans looks at \(t_1=0.75\) and \(t_2=1\). Compare their null dependence.
For Design A, $$ \operatorname{Corr}(Z_1,Z_2)=\sqrt{0.50/1}=0.7071. $$ For Design B, $$ \operatorname{Corr}(Z_1,Z_2)=\sqrt{0.75/1}=0.8660. $$
Both designs have the same marginal null crossing probability at each look, \(P(Z_k\ge2.00)\approx0.0228\). But their joint crossing probabilities differ because their correlations differ. For either design, the overall probability is $$ 0.0228+0.0228-P(Z_1\ge2.00,\ Z_2\ge2.00), $$ with the joint probability evaluated using that design’s correlation. Thus the identical cutoff labels do not make these equivalent designs, and a comparison that omits the information fractions is incomplete.
This example compares probability structure, not two calibrated designs. To claim that either design controls a specified overall alpha, its boundaries must be calibrated for its own schedule. The correlation calculation alone does not establish that either set of cutoffs meets a target error rate.
The Planned Analysis Schedule Is Part of the Rule
A classical boundary is calibrated for its planned analyses. If an investigator adds a look, the rejection event gains another opportunity to cross; if a planned look is omitted, the actual stopping rule no longer matches the original rule. Moving a look changes an information fraction and therefore changes the joint null distribution used to assess crossing probabilities. Continuing to quote the original design’s overall alpha without checking these changes is not justified.
For example, suppose a design plans upper-boundary evaluations at \(t=0.50\) and \(t=1\), both with cutoff 2.00. If a third analysis is added at \(t=0.75\) and also uses cutoff 2.00, the actual rejection event becomes $$ \{Z_{0.50}\ge2.00\}\cup\{Z_{0.75}\ge2.00\}\cup\{Z_1\ge2.00\}. $$ This event contains the original two-look rejection event and may include paths that cross only at the added look. Its null probability is therefore at least as large as that of the original rule. It need not equal the originally calibrated error rate.
The implication is not that any operational change inevitably produces a large error inflation. Rather, the original error guarantee applies to the rule that was specified and calibrated. A changed schedule requires an appropriate evaluation or a design method built to handle that change; one should not assume that old cutoffs automatically retain their interpretation.
List every planned analysis and its information fraction, including any change from the intended plan.
Record the test direction and the cutoff at each look; a single cutoff read out of context is not the rule.
Use the joint null distribution for the stated information fractions to evaluate the probability of at least one crossing.
Do not transfer an error-rate claim to a different schedule or to unplanned analyses without recalibration or a method valid for that change.
A Null-Path Check and Common Diagnostics
A compact check is to describe the rejection event under \(H_0\) before interpreting any observed result. For a planned two-look upper-boundary design, this is the union of the two crossing events, not “a test at alpha at each look.” Under the canonical model, the joint probability can be calculated from the bivariate normal distribution with correlation \(\sqrt{t_1/t_2}\). With more looks, the relevant event is the union over all planned looks, using their joint null distribution.
Several warning signs indicate that a boundary is being read incorrectly:
- Per-look alpha language. A statement such as “the study uses alpha 0.025 at each interim look, so overall alpha is 0.025” confuses a marginal tail probability with the probability of any crossing.
- Adding marginal probabilities without overlap. A sum of look-specific crossing probabilities counts paths crossing more than once multiple times. The union probability must account for overlap and dependence.
- Comparing only look labels. “Look 1” does not mean the same information fraction in different schedules. Check the fractions and compare corresponding design features.
- Ignoring operational changes. An extra or moved analysis changes the event or its joint distribution. The original calibration cannot simply be assumed to apply.
The classical fixed-schedule approach is useful when the analyses are planned in advance and the boundary can be calibrated for that schedule. It is different from applying an unadjusted fixed-sample test repeatedly, which does not account for repeated opportunities to cross. It also differs from procedures designed to update boundaries as information accrues under flexible timing. Those approaches address different operational requirements; they are not reasons to interpret a fixed classical boundary as schedule-free.
These checks have limits. A correct interpretation of the boundary does not establish that the statistical model is appropriate, that the test statistic is well behaved, or that the efficacy and futility rules were implemented as specified. Nor does a nominal p-value at an interim look alone summarize the design’s operating characteristics. The error-rate claim must be tied to the complete design and its assumptions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A two-look design has marginal null crossing probabilities of 0.01 at each look. Why is the overall type I error not determined by either 0.01 alone or by automatically adding the two values?
- Under the canonical null model, two looks occur at \(t_1=0.36\) and \(t_2=1\). Calculate the correlation between their test statistics.
- A calibrated design planned looks at \(t=0.50\) and \(t=1\). An analyst adds a look at \(t=0.75\) but keeps the old cutoffs. What happens to the rejection event, and what conclusion about the original error guarantee is appropriate?
- Two designs have the same cutoff at their first interim analysis, but one looks at \(t=0.40\) and the other at \(t=0.70\). Is that enough to conclude that the designs are equally conservative? Explain.
Solutions. (1) The overall error is the probability of the union of the crossing events. It depends on their overlap and dependence; adding marginal probabilities without subtracting overlap generally double-counts paths. (2) The correlation is \(\sqrt{0.36/1}=0.6000\). (3) The rejection event includes an additional opportunity to cross at \(t=0.75\), so its null probability is at least that of the original event. The original calibration cannot automatically be claimed for the changed schedule. (4) No. The looks occur at different information fractions, so their dependence with later statistics differs. Compare complete schedules and evaluate each design’s joint null crossing probability before drawing an overall conservativeness conclusion.