Familywise Type I Error Across Planned Looks

Why the Error Rate Belongs to the Whole Sequence

In a group sequential design, efficacy may be assessed at more than one planned look. Under the global null hypothesis \(H_0\), a Type I error occurs if the trial rejects at any of those looks. The relevant error rate is therefore the probability of that whole event, not the nominal significance level attached to one isolated analysis. The earlier material on sequential efficacy boundaries in Z-statistic designs establishes the role of look-specific critical values; here the focus is how their rejection probabilities combine.

The key accounting device is to partition rejection by the first look at which the boundary is crossed. These first-crossing events cannot occur together: a trial that first crosses at look 2 did not cross at look 1. Their probabilities therefore add exactly to the probability of rejecting anywhere. This remains true whether the Z statistics at different looks are independent or correlated.

Definition: Let \(Z_1,\ldots,Z_K\) be the test statistics at \(K\) planned looks, and let \(c_1,\ldots,c_K\) be upper efficacy critical values for a one-sided test. The global-null rejection event is \(R=\bigcup_{i=1}^K\{Z_i\ge c_i\}\). The first-crossing event at look \(i\) is \(A_i=\{Z_j<c_j\text{ for every }j<i,\ Z_i\ge c_i\}\). These events are mutually exclusive, and \(R=\bigcup_{i=1}^K A_i\). Thus the design’s familywise Type I error is \(P(R\mid H_0)=\sum_{i=1}^K P(A_i\mid H_0)\).

The Joint Null Model Across Looks

To calculate the probability of crossing at any look, one needs the joint null distribution of the sequential statistics. In the standard canonical Z-statistic design, information increases across looks and the score increments are independent under \(H_0\). If \(I_i\) is the information at look \(i\), then \(Z_i\) has a standard normal distribution under the null, and for \(i<j\),

\operatorname{Corr}(Z_i,Z_j)=\sqrt{\frac{I_i}{I_j}}.

This correlation reflects shared accumulated data: the later statistic includes information already used in the earlier statistic. A calculation that treats repeated Z statistics as independent will generally give the wrong probability of crossing the boundary at least once. The canonical model assumes the planned information levels and analysis times are correctly specified, the null distribution is adequately represented by the standard normal model, and the test direction and stopping rule match the design being evaluated.

For an upper-boundary design, the first-crossing probability at look \(i\) is a joint probability involving all earlier statistics, not merely the marginal probability \(P(Z_i\ge c_i\mid H_0)\). For example, at the second look it is the probability of crossing now while having stayed below the first boundary. In general, numerical integration of the joint null distribution is used to evaluate these probabilities and to choose critical values whose sum equals the target \(\alpha\).

Worked Example: Two Looks With the Same Critical Value

Consider a hypothetical one-sided design with two planned looks at information levels \(I_1=50\) and \(I_2=100\). Suppose both looks use the upper critical value \(c_1=c_2=2.00\). Under the canonical null model, \(Z_1\) and \(Z_2\) are standard normal with correlation \(\sqrt{50/100}=0.7071\). Calculate the overall Type I error.

Let \(B_i=\{Z_i\ge2.00\}\) denote crossing at look \(i\), without regard to whether an earlier crossing occurred. The global rejection event is \(B_1\cup B_2\), so the two-event addition rule gives

P(B_1\cup B_2\mid H_0) =P(B_1\mid H_0)+P(B_2\mid H_0)-P(B_1\cap B_2\mid H_0).

Each marginal upper-tail probability is \(P(Z_i\ge2.00)=0.02275\). For this correlated bivariate normal distribution, numerical integration gives \(P(Z_1\ge2.00,Z_2\ge2.00\mid H_0)\approx0.0075\). Therefore,

P(B_1\cup B_2\mid H_0) \approx 0.02275+0.02275-0.0075=0.0380.

The same result follows from first-crossing events. The first look contributes \(P(A_1\mid H_0)=0.02275\). The second look contributes \(P(A_2\mid H_0)=0.02275-0.0075=0.01525\), because crossings at both looks are not a first crossing at look 2. The total is \(0.02275+0.01525=0.0380\). Thus a nominal one-look tail probability of 0.02275 at each look does not mean the whole design has Type I error 0.02275; the overall probability is about 0.0380.

Why the First-Crossing Probabilities Add

The first-crossing partition gives a direct justification for the global error calculation. If rejection occurs, there is a unique earliest look at which a boundary is crossed. The outcome must therefore belong to one of the events \(A_1,\ldots,A_K\). Conversely, membership in any \(A_i\) implies that rejection occurs. These two facts establish that the global event is exactly their union.

A_i\cap A_j=\varnothing\quad(i\ne j), \qquad R=\bigcup_{i=1}^K A_i, \qquad P(R\mid H_0)=\sum_{i=1}^K P(A_i\mid H_0).

This argument does not require independent looks. It requires the event definitions to match the actual decision rule and the probabilities to be computed under the appropriate joint null distribution. Marginal probabilities alone are not enough to determine the sum: the chance of crossing now depends on the chance that earlier boundaries were not crossed.

Worked Example: Checking a Three-Look Calibration

Suppose a hypothetical three-look, one-sided design has a target \(\alpha=0.050\). A numerical calculation under its planned joint null model gives first-crossing probabilities of 0.014 at look 1, 0.021 at look 2, and 0.015 at look 3. Check whether the design meets its target.

The events refer to the first boundary crossing, so they are disjoint. Add their probabilities:

P(R\mid H_0) =P(A_1\mid H_0)+P(A_2\mid H_0)+P(A_3\mid H_0) =0.014+0.021+0.015=0.050.

The calculated global Type I error equals the target 0.050. The look-specific contributions are not each 0.050; they are portions of the one overall error budget. In contrast, if an alternative candidate boundary set produced first-crossing probabilities 0.018, 0.025, and 0.019, its global error would be \(0.018+0.025+0.019=0.062\), exceeding the target. It would not be acceptable for a design intended to control the global-null Type I error at 0.050.

Two-Sided Tests and Tail Accounting

For a two-sided design, rejection at look \(i\) occurs when the statistic crosses either the upper or lower boundary. If the boundaries are symmetric, the look-\(i\) crossing condition is \(|Z_i|\ge c_i\). A first-crossing event still excludes all earlier crossings, so the global event is still partitioned by its earliest crossing. One can also separate each first crossing by direction: first crossing above the upper boundary or first crossing below the lower boundary. Those directional events are mutually exclusive across looks and directions.

Worked Example: Summing Both Tails Across Two Looks

Consider a hypothetical symmetric two-sided design with two looks. Suppose a joint null calculation gives first-crossing probabilities of 0.006 in the upper tail and 0.006 in the lower tail at look 1, followed by 0.019 in the upper tail and 0.019 in the lower tail at look 2. Find its global Type I error.

The four events are disjoint: each specifies a unique first crossing and its direction. Therefore,

P(R\mid H_0)=0.006+0.006+0.019+0.019=0.050.

This design has global two-sided Type I error 0.050 under the stated null calculation. The two tails together, across both looks, consume the target error rate. Assigning 0.050 to each tail at each look would not be a 0.050 global-error design.

Calibration, Alternatives, and Practical Checks

A boundary set is calibrated by evaluating the probability of rejection anywhere under \(H_0\) and adjusting the critical values until that probability equals the prespecified target \(\alpha\). This is different from assigning the same nominal one-look significance level to every analysis. The latter ignores the union of rejection opportunities. The dependence between looks matters, too: for positively correlated statistics, crossing events overlap, and their overlap must be accounted for rather than simply adding their marginal tail probabilities.

A Bonferroni approach is a useful alternative way to control familywise error. It assigns per-look error allowances whose sum is no greater than \(\alpha\); by the union bound, the probability of any rejection is then at most \(\alpha\), without requiring independence. It may be conservative because it does not use the detailed joint distribution to account for overlapping crossings. Joint calibration instead uses the planned dependence structure to target the global error rate. A single final analysis has no repeated-look union to calibrate, but it also gives up opportunities for earlier efficacy stopping.

A global-null calculation is valid only for the plan it represents. Important checks include:

Look timing: use the planned information levels and verify that the covariance model corresponds to them.
Decision rule: include the correct number of looks, test direction, and upper or two-sided crossing condition.
First-crossing accounting: ensure each contribution excludes all earlier crossings; do not sum marginal crossing probabilities as though they were disjoint.
Implementation: verify that the actual analysis uses the calibrated critical values and stopping rule. Changes to looks or boundaries can change the global error.

These checks matter because an apparently reasonable collection of look-specific tests need not control the intended trial-wide error rate. A mis-specified correlation model, an unplanned extra look, or a mismatch between a one-sided calculation and a two-sided decision rule can all invalidate the calibration. The calculation also addresses Type I error under the specified global null; it does not by itself describe power or error probabilities under alternatives.

Common Mistake

Wrong reasoning: “Each look uses a test at level \(\alpha\), so the design’s Type I error is \(\alpha\).” This treats every look as if it were the only analysis, or assumes that later crossings cannot add any rejection probability.

Correction: The design rejects if any planned look crosses its boundary. Define the first-crossing events under the global null and calculate their probabilities using the joint distribution of the sequential statistics. The global Type I error is their sum; it equals the target only when the boundary set has been calibrated to make that sum equal \(\alpha\).

Key takeaway: The trial-wide Type I error is the probability of at least one boundary crossing under the global null. Partitioning that event by the first crossing makes the contributions disjoint and their probabilities add exactly to the target error calculation.

Check Your Understanding

Try each question, then compare your reasoning with the solutions below.

Why can first-crossing probabilities be added even when sequential test statistics are correlated?
In a two-look one-sided design, the marginal crossing probabilities are 0.020 at each look and the probability of crossing both boundaries is 0.006. What is the probability of crossing at least once?
A three-look design has first-crossing probabilities 0.010, 0.012, and 0.009. Does it meet a global target \(\alpha=0.025\)?
What is the main error in claiming that a level 0.05 test at each of four looks automatically gives global Type I error 0.05?
Name one design change that can make a planned global error calculation no longer apply.

Solutions. (1) The first-crossing events are mutually exclusive by definition; correlation affects their probabilities, not their disjointness. (2) Use the union rule: \(0.020+0.020-0.006=0.034\). (3) Sum the disjoint contributions: \(0.010+0.012+0.009=0.031\), which exceeds 0.025, so the stated design does not meet the target. (4) It confuses each isolated test’s nominal level with the probability of at least one rejection across all looks; it does not account for the global union event. (5) For example, adding an unplanned analysis, changing a planned information level, or changing from a one-sided to a two-sided rule can invalidate the calculation unless the error rate is recalculated.