Verifying Cumulative Type I Error

Why Verify Every Look?

A group sequential design permits formal testing at multiple planned analyses. Its efficacy boundaries are calibrated so that the probability of rejecting $H_0$ by each analysis agrees with the intended alpha-spending plan. A final cumulative Type I error rate close to $\alpha$ is not, by itself, enough to verify the design: an earlier look could have spent too much or too little, with later looks compensating for the discrepancy.

This tutorial focuses on checking a computed design. It uses the standard canonical joint-null model for the analysis statistics and the planned efficacy boundaries as inputs. The earlier material on boundary calibration concerns how boundaries are obtained; here the task is to evaluate the resulting cumulative null rejection probabilities and compare them with their targets.

Definition: At look $k$, let $Z_k$ be the standardized test statistic under $H_0$, and let $b_k$ be its upper efficacy boundary for a one-sided test. The first-crossing event at look $k$ is $$ A_k=\{Z_1<b_1,\ldots,Z_{k-1}<b_{k-1},\ Z_k\ge b_k\}. $$ The cumulative null rejection probability through look $k$ is $$ \alpha_k^{\mathrm{actual}}=P_{H_0}\left(\bigcup_{j=1}^{k}A_j\right). $$

Model, Assumptions, and the Verification Target

For the usual canonical group sequential model, under the null hypothesis the statistics have a joint multivariate normal distribution with mean zero, variance one at every look, and correlations $ \operatorname{Corr}(Z_i,Z_j)=\sqrt{I_i/I_j} $ for $i<j$, where $I_k$ is the information at look $k$. The information values must be positive and increase across looks. The analysis times and boundaries must be those actually intended for the design being checked. If the design uses a different joint-null model, its corresponding joint distribution must replace the canonical one.

For a design with $K$ looks, write the planned cumulative spending target at look $k$ as $a_k$, with $a_0=0$. The target incremental spending is $a_k-a_{k-1}$. The computed design is checked by calculating $P_{H_0}(A_k)$ for each look and accumulating these probabilities. The key comparison is between the cumulative value and $a_k$ at each look.

For numerical integration, the first-look probability is $P_{H_0}(Z_1\ge b_1)=1-\Phi(b_1)$. At a later look, integrate over histories that have not crossed an earlier boundary, and multiply by the conditional probability of crossing the current boundary. In the canonical joint-normal model, if $\mathbf Z_{k-1}=(Z_1,\ldots,Z_{k-1})^\mathsf{T}$ has correlation matrix $R$, let $r$ be the vector of correlations between $Z_k$ and those earlier statistics. Then $Z_k\mid\mathbf Z_{k-1}=\mathbf z$ is normal with mean $r^\mathsf{T}R^{-1}\mathbf z$ and variance $1-r^\mathsf{T}R^{-1}r$. This gives the integral

P_{H_0}(A_k)= \int_{\substack{z_j<b_j\\j=1,\ldots,k-1}} \left[ 1-\Phi\left( \frac{b_k-r^\mathsf{T}R^{-1}\mathbf z} {\sqrt{1-r^\mathsf{T}R^{-1}r}} \right) \right] f_{1:k-1}(\mathbf z)\,d\mathbf z ,

where $f_{1:k-1}$ is the joint density of the preceding statistics under $H_0$. The integration region explicitly excludes paths that already crossed. That restriction is essential: the probability of crossing the current boundary without an earlier crossing is not generally the marginal probability $P_{H_0}(Z_k\ge b_k)$.

Why Summing First Crossings Gives Cumulative Error

The first-crossing events are mutually exclusive. For $i<j$, event $A_i$ requires a crossing at look $i$, whereas $A_j$ requires no crossing at look $i$. Therefore $A_i\cap A_j$ is empty. Also, by look $k$, any path that has crossed at least one efficacy boundary has a unique first look at which it crossed. Thus the event of rejection by look $k$ is exactly the union of $A_1,\ldots,A_k$.

\begin{aligned} \alpha_k^{\mathrm{actual}} &=P_{H_0}\left(\bigcup_{j=1}^{k}A_j\right)\\ &=\sum_{j=1}^{k}P_{H_0}(A_j). \end{aligned}

The second equality follows from additivity of probability for disjoint events. Consequently, the increment in actual cumulative Type I error at look $k$ is $P_{H_0}(A_k)$, and the cumulative total is the sum of those increments. This is the accounting identity used in the verification.

Worked Example: A Three-Look Design That Matches Its Targets

Consider a hypothetical one-sided design with cumulative spending targets $a_1=0.0050$, $a_2=0.0150$, and $a_3=0.0250$. Suppose numerical integration under the specified joint-null model gives first-crossing probabilities $P_{H_0}(A_1)=0.0050$, $P_{H_0}(A_2)=0.0100$, and $P_{H_0}(A_3)=0.0100$. These are hypothetical integration results for illustrating the check.

At the first look, the cumulative probability is $0.0050$, which matches $a_1=0.0050$. At the second look,

\alpha_2^{\mathrm{actual}}=0.0050+0.0100=0.0150.

This matches $a_2=0.0150$. At the third look,

\alpha_3^{\mathrm{actual}}=0.0050+0.0100+0.0100=0.0250.

The corresponding target increments are $0.0050$, $0.0100$, and $0.0100$. Each actual increment agrees with its target increment, and the cumulative probability agrees with the spending target at every look. On these reported values, the design passes the check.

Compare Cumulative Values, Not Just the Final Total

The design’s spending targets are cumulative: $a_k$ means the intended null rejection probability by look $k$. An increment $a_k-a_{k-1}$ instead describes the intended probability of a first crossing at look $k$, when the design achieves its targets exactly. Both quantities are useful, but they answer different questions. A verification table should show the first-crossing probability, its cumulative sum, the target, and the discrepancy at every look.

Worked Example: A Correct Final Total Hides an Intermediate Discrepancy

Suppose a hypothetical three-look design has targets $a_1=0.0040$, $a_2=0.0120$, and $a_3=0.0250$. Numerical integration gives first-crossing probabilities $0.0041$, $0.0087$, and $0.0122$.

The cumulative probabilities are $0.0041$ at look 1, $0.0041+0.0087=0.0128$ at look 2, and $0.0041+0.0087+0.0122=0.0250$ at look 3. Subtracting each target from its cumulative probability gives discrepancies $+0.0001$, $+0.0008$, and $0.0000$, respectively.

The final cumulative value matches the final target exactly, but at look 2 the design’s null rejection probability exceeds its target by $0.0008$. If the prespecified absolute tolerance is $0.0002$, the look-2 check fails. A correct final total does not remove this intermediate failure: the later increment happens to offset the earlier excess. The conclusion is that the design does not meet the stated verification criterion at every look.

Numerical Precision and Diagnostic Checks

Integration results and spending targets are often stored with more digits than are shown in a report. Calculate cumulative sums and discrepancies using the unrounded values; round only for presentation. Set a numerical tolerance before interpreting small differences. The tolerance should reflect the accuracy of the integration and the design’s required precision, rather than being chosen after seeing the result.

Several checks help detect errors in the calculation or its inputs:

Check the event definition. Each reported increment must be a first-crossing probability, not the marginal probability of crossing at that look.
Check cumulative arithmetic. The total at look $k$ must equal the total at look $k-1$ plus $P_{H_0}(A_k)$. It cannot decrease as looks are added.
Check the model inputs. Use the intended information times, boundary values, sidedness, and null distribution. A mismatch in any of these can change the crossing probabilities.
Check probability ranges. Each first-crossing probability must be between zero and one, and cumulative rejection probabilities must be nondecreasing and no greater than one.
Check displayed precision. Small apparent failures can be caused by rounding, while rounded equality can conceal a discrepancy in the underlying values.

If a discrepancy is larger than the prespecified tolerance, inspect the joint-null model and the integration region before concluding that the boundaries are incorrectly calibrated. For example, omitting the no-earlier-crossing restrictions counts paths that already rejected, while using marginal crossing probabilities can substantially overstate later increments.

Two-Sided Tests and Alternative Checks

For a symmetric two-sided efficacy rule with critical value $c_k>0$, replace the crossing condition $Z_k\ge b_k$ with $|Z_k|\ge c_k$, and define the first-crossing event to require $|Z_j|<c_j$ at every earlier look. The same disjoint-event argument applies. In the conditional integral, the current-look crossing probability becomes the sum of its lower-tail and upper-tail probabilities. The total spending target must correspond to the specified two-sided testing convention; it should not be compared with a one-sided target.

Direct joint-normal integration evaluates the probabilities implied by the specified model and is often suitable for a deterministic design check. A boundary table alone is not a probability check, and a final-look calculation alone cannot establish that targets were met at earlier looks. Independent computational approaches can be useful for investigating a substantial discrepancy, but they must use the same event definitions, information schedule, and null model to make their results comparable.

This verification is conditional on the design assumptions and supplied inputs. It does not establish that the assumed information times or analysis model will describe a realized trial, and it does not verify power or other design properties. Its specific conclusion is whether the computed boundaries produce cumulative null rejection probabilities that agree with the specified spending targets at the planned looks, to the chosen numerical tolerance.

Key takeaway: Verify a spending design look by look: integrate each disjoint first-crossing probability, sum those increments through each analysis, and compare every cumulative sum with its planned target using unrounded values and a stated tolerance.

Worked Example: Applying a Numerical Tolerance

A hypothetical design has cumulative targets $0.00300$, $0.01000$, and $0.02500$. Its numerically integrated first-crossing probabilities are $0.00296$, $0.00708$, and $0.01493$. Suppose the prespecified absolute tolerance is $0.00010$.

The cumulative values are $0.00296$, $0.00296+0.00708=0.01004$, and $0.01004+0.01493=0.02497$. The cumulative discrepancies, actual minus target, are $-0.00004$, $+0.00004$, and $-0.00003$. Each absolute discrepancy is less than $0.00010$, so the design passes this stated numerical check at all three looks. This conclusion is about agreement to the tolerance, not exact equality of the probabilities.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Why are first-crossing events at distinct looks disjoint?
A two-look design has first-crossing probabilities $0.006$ and $0.009$. What is its cumulative null rejection probability by look 2?
The cumulative targets are $0.004$, $0.012$, and $0.025$; the actual cumulative values are $0.004$, $0.0128$, and $0.025$. Does a final match verify the design at every look?
Why is $P_{H_0}(Z_k\ge b_k)$ generally not the first-crossing probability at look $k$?
For a symmetric two-sided design, what event replaces $Z_k\ge b_k$ in the definition of crossing?

Solutions. (1) A first crossing at an earlier look requires crossing there, while a first crossing at a later look requires no crossing at that earlier look; these conditions cannot both hold. (2) The cumulative probability is $0.006+0.009=0.015$, or 1.5%. (3) No. The look-2 value exceeds its target by $0.0008$, so the look-2 discrepancy must be assessed against the prespecified tolerance even though the final values match. (4) The marginal event $Z_k\ge b_k$ includes paths that crossed an earlier boundary; the first-crossing probability excludes those paths. (5) Use $|Z_k|\ge c_k$, together with the condition that $|Z_j|<c_j$ at every earlier look.