Why the Number of Looks Matters
At each interim analysis, a trial has another opportunity to cross its efficacy boundary. Even when the null hypothesis is true, a statistic can fluctuate far enough from zero to cross by chance. Therefore, controlling the type I error rate requires controlling the chance of crossing at any planned analysis, not merely the chance of crossing at one analysis.
The key object is a joint probability: the probability that at least one test statistic crosses its boundary under the global null. Adding looks changes that probability, and therefore generally requires recalibrating the boundary. The earlier material on first-crossing probabilities and equally spaced looks provides the setting for examining this effect for Pocock and O’Brien–Fleming designs.
For the standard canonical group-sequential model under \(H_0\), the vector of statistics is jointly normal, each \(Z_i\) has mean zero and variance one, and the correlation for analyses \(i<j\) is $$ \operatorname{Corr}(Z_i,Z_j)=\sqrt{\frac{t_i}{t_j}}, $$ where \(t_i\) is the information fraction at analysis \(i\), with \(0<t_1<\cdots<t_K=1\). This model follows from independent increments of the accumulated score or information process. The calculations below assume that the planned information times and this joint null distribution are appropriate for the design. A different test statistic or a different dependence structure requires corresponding calibration rather than automatic reuse of these cutoffs.
Why an Additional Look Raises the Chance of Crossing
For fixed cutoffs and a fixed set of earlier analyses, adding a later analysis adds another event to the union defining the overall crossing probability. If \(A_i=\{|Z_i|\ge b_i\}\), then $$ P_{H_0}(A_1\cup\cdots\cup A_K\cup A_{K+1}) = P_{H_0}(A_1\cup\cdots\cup A_K) + P_{H_0}\left(A_{K+1}\setminus\bigcup_{i=1}^{K}A_i\right). $$ The second term is nonnegative, and is positive when the new analysis has a positive chance of crossing without an earlier crossing. Thus, with the earlier looks and their cutoffs held fixed, another genuine opportunity to reject cannot reduce the null crossing probability.
Equally spaced schedules require a small qualification. When the number of looks changes but the final information is kept fixed, the intermediate information fractions change too: for \(K\) equally spaced looks, \(t_i=i/K\). The new schedule is not necessarily the old schedule with just one extra analysis appended. Its joint distribution and the locations of its analyses both change. The direction and size of the change in overall crossing probability should therefore be evaluated under the new schedule, not inferred from event inclusion alone.
The calibration goal remains the same: find cutoffs for which the joint null crossing probability is \(\alpha\). What changes with \(K\) is the set of joint events being controlled. Consequently, a cutoff calibrated for one number of looks is not generally valid for another.
Worked Example: Why Reusing a One-Look Cutoff Is Not Calibration
Suppose a two-sided test uses an overall significance level of \(\alpha=0.05\). A single analysis at the final information has the usual standard-normal cutoff of about 1.96, since \(P(|Z_1|\ge1.96)\) is about 0.05. Now consider a two-look design with equally spaced information, \(t_1=0.5\) and \(t_2=1\), but keep the cutoff at 1.96 at both looks.
Each statistic separately has a standard normal null distribution, so each marginal crossing probability is approximately 0.05. Let \(A_1=\{|Z_1|\ge1.96\}\) and \(A_2=\{|Z_2|\ge1.96\}\). The overall crossing probability is $$ P_{H_0}(A_1\cup A_2) = P_{H_0}(A_1)+P_{H_0}(A_2\setminus A_1) \approx 0.05+P_{H_0}(A_2\setminus A_1). $$ The correlation is \(\sqrt{0.5/1}=\sqrt{0.5}\), not 1, so there is positive probability of crossing at the second look without crossing at the first. Hence \(P_{H_0}(A_2\setminus A_1)>0\), and the overall crossing probability is greater than 0.05.
This establishes the issue without needing to approximate the joint probability numerically: the one-look cutoff cannot simply be copied to both analyses while claiming the same overall error rate. The exact inflation depends on the joint distribution, so it must be calculated using that distribution.
How Pocock Cutoffs Respond to More Looks
A Pocock design uses a common positive cutoff \(c_K\) at every analysis. For a two-sided design, its calibration condition is $$ P_{H_0}\left(\max_{1\le i\le K}|Z_i|\ge c_K\right)=\alpha. $$ Because the statistics are correlated, this is not the same as treating the analyses as independent. The cutoff is obtained from the joint null distribution at the planned information fractions.
For illustration, consider two-sided \(\alpha=0.05\) designs with equally spaced looks under the canonical model. Numerical calibration gives the following approximate common cutoffs:
| Number of looks \(K\) | Equally spaced information fractions | Approximate Pocock cutoff \(c_K\) |
|---|---|---|
| 1 | 1 | 1.960 |
| 2 | 0.5, 1 | 2.178 |
| 3 | 0.333, 0.667, 1 | 2.289 |
| 4 | 0.25, 0.5, 0.75, 1 | 2.361 |
The table shows the general effect clearly: more planned opportunities require a more stringent common cutoff to keep the overall null crossing probability at 0.05. The values increase because a statistic may cross at any one of the planned looks. They do not imply that the marginal distribution of \(Z_i\) at any particular look has changed; each marginal remains standard normal under the model.
Worked Example: Comparing Two- and Three-Look Pocock Designs
Suppose two trial designs both use a two-sided overall \(\alpha=0.05\) and the canonical model. Design A has two equally spaced looks, and Design B has three equally spaced looks. The calibrated cutoffs are approximately 2.178 and 2.289, respectively.
For Design A, the rejection rule is to reject \(H_0\) if \(|Z_1|\ge2.178\) or \(|Z_2|\ge2.178\). For Design B, reject if any of the three statistics has absolute value at least 2.289. The increase in cutoff is $$ 2.289-2.178=0.111. $$ Thus, the three-look design uses a common cutoff about 0.111 standard-normal units higher than the two-look design.
This difference is not obtained by dividing 0.05 among the analyses or by assuming independent tests. Each cutoff comes from solving the joint crossing probability under its own information schedule. Under the stated model and schedule, both rules have an overall null crossing probability of approximately 0.05, allowing for rounding of the displayed cutoffs.
O’Brien–Fleming Shape and Recalibration
An O’Brien–Fleming design uses a boundary that is stringent early and approaches a less stringent final cutoff. One classical shape is $$ b_i=\frac{c_K}{\sqrt{t_i}},\qquad i=1,\ldots,K, $$ where \(c_K\) is chosen to satisfy $$ P_{H_0}\left(\bigcup_{i=1}^{K}\left\{|Z_i|\ge\frac{c_K}{\sqrt{t_i}}\right\}\right)=\alpha. $$ The shape makes early cutoffs larger because \(t_i<1\) at interim analyses. The parameter \(c_K\) is still calibrated jointly; a boundary shape alone does not specify a valid overall type I error rate. The exact calibrated cutoffs depend on \(\alpha\), the number of analyses, and the information fractions.
Worked Example: Calculating a Two-Look O’Brien–Fleming-Shaped Boundary
Consider two equally spaced looks at \(t_1=0.5\) and \(t_2=1\), with two-sided overall \(\alpha=0.05\). Joint null calibration of the O’Brien–Fleming shape gives \(c_2\) approximately 1.98 (the displayed value is rounded). The corresponding boundaries are $$ b_1=\frac{1.98}{\sqrt{0.5}}\approx\frac{1.98}{0.7071}=2.800,\qquad b_2=\frac{1.98}{\sqrt{1}}=1.980. $$
Suppose the observed statistics are \(Z_1=2.60\) and \(Z_2=2.05\). At the first look, \(|2.60|<2.800\), so the boundary is not crossed. At the second look, \(|2.05|\ge1.980\), so the final boundary is crossed and the design rejects \(H_0\). The decision uses the planned sequential rule, not a separate 0.05 test at each look.
The calibration depends on the full joint null distribution, including \(\operatorname{Corr}(Z_1,Z_2)=\sqrt{0.5}=0.7071\). The rounded boundaries illustrate their shape and decision rule; software or accurate multivariate normal integration should be used when exact design cutoffs are needed.
What Changes and What Does Not
Increasing the number of planned looks changes the number of opportunities to cross and, for an equally spaced schedule, the intermediate information fractions. It does not change the target overall significance level, the definition of a boundary crossing, or the need to account for dependence among the statistics. A calibrated design keeps its overall null crossing probability at the prespecified \(\alpha\) by adjusting its critical values.
Pocock and O’Brien–Fleming designs respond differently because their boundary shapes differ. Pocock applies a common cutoff, so its calibration raises that shared cutoff as more looks are planned. The O’Brien–Fleming shape makes early boundaries relatively stringent and the final boundary closer to a conventional single-look cutoff; its scale parameter must nonetheless be recalibrated for the planned schedule. Neither design can use a cutoff from a different number of looks without checking the resulting joint crossing probability.
These classical designs are not the only way to control the error rate. One alternative is a prespecified sequence of per-look cutoffs chosen directly to make the joint crossing probability equal to \(\alpha\), without requiring a Pocock or O’Brien–Fleming shape. A naive alternative is to apply the ordinary one-look cutoff at every analysis; as shown above, this generally inflates the overall error rate. The shaped designs trade the timing of stringency differently, but the overall calibration target is common.
Calibration also has limits. It relies on the planned analysis times, the assumed null distribution, and the specified sidedness. Changing the schedule or adding an unplanned analysis can invalidate the original calibration. Likewise, using a canonical-model cutoff when the test statistic has materially different null behavior may fail to control the intended error rate. These are design issues: the remedy is to calibrate under the actual planned model and schedule, rather than to interpret a familiar cutoff as universally valid.
Diagnostics and Common Pitfalls
- Confusing marginal and overall error. A 0.05 crossing probability at each individual look does not imply a 0.05 probability of crossing at any look. The latter is a joint probability.
- Assuming independent looks. Under the canonical model, statistics are correlated according to their information fractions. Multiplying marginal non-crossing probabilities as if the looks were independent gives the wrong calibration.
- Reusing a cutoff after changing \(K\). A cutoff calibrated for two looks is not automatically valid for three. Recompute the joint crossing probability for the new number and schedule of analyses.
- Interpreting shape as calibration. Knowing that a boundary is flat or O’Brien–Fleming-shaped does not determine its valid numerical cutoffs. The scale must be calibrated to the target overall \(\alpha\).
- Ignoring the schedule. The number of looks alone is insufficient to determine a cutoff. Information fractions, sidedness, and the null distribution also matter.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In a two-look design, why is the overall null crossing probability not generally equal to the single-look probability at either analysis?
- A two-sided Pocock design has two equally spaced looks and \(\alpha=0.05\). What is its approximate common cutoff, and how does it compare with the one-look cutoff?
- For a two-look O’Brien–Fleming-shaped design with \(t_1=0.5\) and calibrated \(c_2=1.98\), calculate the first-look cutoff to three significant figures.
- Why should a cutoff calibrated for three equally spaced looks not automatically be used in a four-look design?
Solutions. 1. The overall event is the union of crossing at either analysis. It includes crossings at the second look that did not occur at the first, so it depends on both marginal probabilities and their joint dependence. 2. The approximate cutoff is 2.178. It is higher than the one-look cutoff of about 1.960 because the two-look design has an additional opportunity to cross. 3. The first-look cutoff is \(1.98/\sqrt{0.5}=1.98/0.7071\approx2.80\). 4. The number and information fractions of the planned analyses determine the joint null crossing probability. A four-look design has a different joint event and schedule, so its cutoff must be calibrated for that design to control the same overall \(\alpha\).