Common Alpha-Spending Implementation Errors

Why Implementation Details Matter

An alpha-spending design controls the probability of crossing an efficacy boundary under $H_0$ across a sequence of analyses. Its guarantee depends not just on the chosen spending function, but on how the function is translated into boundaries. A program can use a familiar spending curve and still miss its target if it treats cumulative spending as a per-look allocation, indexes the curve by the wrong fraction, or calculates crossing probabilities using the wrong joint null distribution.

The discussion below uses an upper-sided efficacy rule: at look $i$, stop for efficacy if $Z_i \ge u_i$. Let $I_i$ be the information available at look $i$, with $0<I_1<\cdots<I_K$, and define the information fraction $t_i=I_i/I_K$. Let $A(t)$ be a nondecreasing spending function with $A(0)=0$ and $A(1)=\alpha$. The intended cumulative Type I error by look $i$ is $A(t_i)$.

Definition: Cumulative alpha spending at look $i$ is the probability, under $H_0$, of having crossed an efficacy boundary at or before that look: $$ P_0\left(\bigcup_{j=1}^{i}\{Z_j\ge u_j\}\right)=A(t_i). $$ The incremental spending between looks $i-1$ and $i$ is $A(t_i)-A(t_{i-1})$, where $A(t_0)=0$. It is the probability of first crossing at look $i$, not generally the marginal probability $P_0(Z_i\ge u_i)$.

This tutorial concerns implementation checks, not the choice of a particular spending shape. For a two-sided design, the crossing event must instead reflect the specified two-sided efficacy rule, such as $\{|Z_i|\ge u_i\}$; its boundaries must be calibrated to that event rather than obtained by mechanically reusing a one-sided calculation.

Error 1: Treating Cumulative Spending as a Per-Look Allocation

The boundary at look $i$ is calibrated against the probability of crossing by that look. Earlier crossings are part of that probability. Consequently, the increment $A(t_i)-A(t_{i-1})$ is not ordinarily the marginal upper-tail probability at look $i$, and the cumulative value $A(t_i)$ is not a fresh allowance to add to the error spent earlier.

The relationship can be derived directly. Define $C_i=\bigcup_{j=1}^{i}\{Z_j\ge u_j\}$, the event of any crossing by look $i$. Since $C_{i-1}$ is contained in $C_i$, the event of a first crossing at look $i$ is $C_i\setminus C_{i-1}$. Thus:

P_0(C_i\setminus C_{i-1}) =P_0(C_i)-P_0(C_{i-1}) =A(t_i)-A(t_{i-1}).

Among paths that have not crossed by look $i-1$, the conditional probability of crossing at look $i$ is therefore:

P_0(C_i\setminus C_{i-1}\mid C_{i-1}^{c}) =\frac{A(t_i)-A(t_{i-1})}{1-A(t_{i-1})}.

This conditional probability is also not, in general, the marginal tail probability $P_0(Z_i\ge u_i)$. Some paths that exceed $u_i$ at look $i$ may already have crossed earlier; the marginal tail includes those paths, while the incremental spending does not.

Worked Example: Incremental Spending Is Not the Marginal Tail

Consider a hypothetical one-sided design with $\alpha=0.025$, looks at information fractions $t_1=0.4$ and $t_2=0.8$, and a linear spending function $A(t)=0.025t$. Assume the canonical joint null model with $\operatorname{Corr}(Z_1,Z_2)=\sqrt{t_1/t_2}$.

The cumulative targets are $A(t_1)=0.010$ and $A(t_2)=0.020$. The second-look increment is $0.020-0.010=0.010$. At the first look, the boundary satisfies $P_0(Z_1\ge u_1)=0.010$, so $u_1=\Phi^{-1}(0.990)\approx2.326$.

A mistaken implementation might assign the second-look increment $0.010$ to the marginal tail too, setting $u_2=\Phi^{-1}(0.990)\approx2.326$. But the cumulative crossing probability would then be $$ P_0(Z_1\ge u_1\ \text{or}\ Z_2\ge u_2) =0.010+0.010-P_0(Z_1\ge u_1,Z_2\ge u_2). $$ The intersection probability is positive, so this union probability is less than $0.020$. The design underspends by look 2. Correct calibration instead solves $$ P_0(Z_1<u_1,\ Z_2<u_2)=1-A(t_2)=0.980, $$ using the joint null distribution. The second boundary must account for both the first-look boundary and the dependence between looks.

A related coding error is to add each cumulative value as if it were a new increment. For example, adding $A(t_1)$ and then $A(t_2)$ as two separate first-crossing targets would assign $0.010+0.020=0.030$ by look 2 in this example, rather than the intended cumulative $0.020$. Useful diagnostics include printing $A(t_i)$, $A(t_i)-A(t_{i-1})$, and the achieved cumulative crossing probability as separate quantities. They should not be stored under an ambiguous label such as “alpha at look $i$.”

Error 2: Using the Enrollment Fraction Instead of the Information Fraction

A spending function is indexed by information time: $t_i=I_i/I_K$. A fraction of participants enrolled, randomized, or followed is a valid substitute only when it correctly represents the fraction of statistical information. That may be approximately true in some designs, but it is not a general identity. Information depends on the model and the data contributing to the test statistic; for example, in an event-driven time-to-event study, the number of observed events can matter more directly than the number of participants enrolled.

Worked Example: Enrollment and Event Information Give Different Targets

Suppose a hypothetical event-driven study plans to analyze after 200 events. At an interim look, 120 of the expected participants have enrolled, corresponding to 60% of planned enrollment, but only 80 events have occurred. Assume, for this illustration, that information is proportional to the event count, so $t=80/200=0.40$. Let $\alpha=0.025$ and use the linear spending function $A(t)=0.025t$.

Using the information fraction gives the cumulative target $$ A(0.40)=0.025(0.40)=0.010. $$ Using the enrollment fraction instead would give $A(0.60)=0.025(0.60)=0.015$. This substitutes a different time point in the spending function and increases the intended cumulative spending by $0.005$, or 20% of the total $\alpha$. The latter target is not justified by the stated information model.

The assumption that information is proportional to event count is part of this hypothetical example, not a universal rule. In an actual design, the analysis plan should define how $I_i$ is obtained. The implementation should use that planned information measure consistently and should document whether the fraction is based on observed information, planned information, or a prespecified estimate.

A useful check is to compare $t_i$ with the planned information schedule at every look, not just at the final analysis. If enrollment, events, outcome variability, allocation, or follow-up make these quantities diverge, an enrollment fraction can shift the spending curve forward or backward. This is especially consequential for spending functions that allocate error unevenly over time. A correct-looking plot of $A(t)$ does not validate the values of $t_i$ supplied to the boundary code.

Error 3: Calibrating With the Wrong Null Covariance

Boundary calibration is a joint-probability calculation. Under the canonical joint normal model with independent information increments, under $H_0$ the standardized statistics have mean zero, variance one, and covariance

\operatorname{Cov}_0(Z_i,Z_j)=\sqrt{\frac{t_i}{t_j}},\qquad i\le j.

The formula uses information fractions and the null distribution. Treating the looks as independent, or supplying a covariance matrix inconsistent with the null score process, changes the probability of crossing at least one boundary. When the canonical model does not describe the design, calibration requires the appropriate joint null distribution for that design; the canonical covariance should not be imposed automatically.

Worked Example: Independence Changes a Two-Look Crossing Probability

For a numerical illustration, suppose two hypothetical looks occur at $t_1=0.5$ and $t_2=1$, and consider upper boundaries $u_1=u_2=2$. Under the canonical null model, the correlation is $$ \rho=\sqrt{0.5/1}=\sqrt{0.5}\approx0.7071. $$ The marginal upper-tail probability at either look is $q=1-\Phi(2)\approx0.02275$.

If the looks were incorrectly treated as independent, the probability of at least one crossing would be $$ 1-(1-q)^2=1-(0.97725)^2\approx0.04498. $$ Under the canonical model, the correct calculation is instead $$ P_0(Z_1\ge2\ \text{or}\ Z_2\ge2) =2q-P_0(Z_1\ge2,Z_2\ge2), $$ where the joint tail must be evaluated with correlation $0.7071$, for example by bivariate-normal numerical integration. Because this correlation is positive, the joint upper-tail probability exceeds $q^2$, so the crossing probability is lower than the independence calculation. Neither calculation should be assumed to equal a chosen spending target: the example illustrates why the specified joint null law must be used when solving for boundaries.

It is important to distinguish null calibration from power evaluation. Type I error control is defined under $H_0$, so the boundary-calibration probability must use the null distribution. Under the canonical model, a design alternative changes the means of the sequential statistics but retains the canonical covariance; other models may require their own carefully specified joint distribution. Replacing the null calibration law with an alternative distribution does not calculate the intended Type I error.

A Practical Audit of Boundary Calibration

The following sequence helps separate input errors from numerical errors. It assumes the planned analysis uses a joint null distribution for the sequential test statistics and a specified upper-efficacy crossing rule.

Verify the time inputs.
Check that each $t_i$ is the intended information fraction, satisfies $0<t_1<\cdots<t_K=1$, and is not an unexamined enrollment fraction.

Print cumulative targets and increments separately.
Confirm that $A(t_i)$ is cumulative and that the new first-crossing target is $A(t_i)-A(t_{i-1})$.

Inspect the joint null model.
Check its variances, covariances, ordering, and correspondence to the planned information times. Do not assume independence unless it is justified.

Recompute achieved cumulative error.
Using the final boundaries and the same null model, evaluate $P_0(\bigcup_{j=1}^{i}\{Z_j\ge u_j\})$ at each look and compare it with $A(t_i)$.

This audit complements numerical checks such as repeating a calculation at tighter integration tolerances, using a second implementation, and simulating under $H_0$. Simulation estimates can be noisy, especially for small crossing probabilities, so a close match in one simulation is not proof of correct calibration. Conversely, a large discrepancy should prompt checks of the inputs, event definition, covariance, and numerical convergence before changing the spending function.

Alternatives, Limitations, and a Common Pitfall

A fixed per-look test threshold or a Bonferroni allocation can be simpler to implement, but it is a different way of controlling error and does not automatically produce the boundaries specified by a chosen spending function. An alpha-spending design calibrates sequential boundaries to cumulative crossing probabilities at information times. Numerical integration or an appropriate recursion can evaluate those probabilities; neither method can correct a mistaken time fraction or covariance supplied to it. Simulation is valuable as an independent diagnostic, but it does not replace stating and checking the null model.

Every implementation also has model limitations. The canonical covariance formula depends on its independent-increment and information assumptions. If the analysis has a different joint null law, calibration must match that law. Likewise, a spending target at an information time only has the intended meaning if the information time is defined and estimated consistently with the design. These checks verify the calculation under its assumptions; they do not establish that the assumptions fit every study.

Common pitfall: “The increment at look 2 is 0.01, so I set the marginal tail at look 2 to 0.01.” The reasoning confuses a first-crossing probability among paths still under observation with a marginal probability over all paths. The correction is to solve for the boundary using the joint no-crossing probability through look 2, then verify the cumulative crossing probability under the null.

Key takeaway: A valid implementation keeps three objects distinct: the information fraction $t_i$, the cumulative target $A(t_i)$, and the joint null probability used to calibrate and verify the boundaries. Confusing any pair can produce boundaries that do not achieve the intended Type I error spending.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If $A(t_1)=0.008$ and $A(t_2)=0.020$, what is the incremental spending between looks 1 and 2? Is it necessarily $P_0(Z_2\ge u_2)$?
A linear spending function has $\alpha=0.04$. At a look, the information fraction is $0.25$, while the enrollment fraction is $0.50$. What cumulative target does the linear function assign at that look, and why?
Under the canonical model, two looks occur at $t_1=0.36$ and $t_2=0.81$. What is their null correlation?
Why can a boundary be incorrect even if the spending function itself is programmed correctly?

Solutions. (1) The increment is $0.020-0.008=0.012$. It is the probability of first crossing at look 2, not necessarily the marginal tail probability at look 2, because the marginal tail also includes paths that crossed earlier. (2) The information fraction is the required input, so $A(0.25)=0.04(0.25)=0.010$. Using enrollment would incorrectly give $0.020$ unless enrollment were known to represent the information fraction. (3) The correlation is $\sqrt{0.36/0.81}=\sqrt{4/9}=2/3\approx0.6667$. (4) The implementation may use the wrong time fractions, confuse cumulative and incremental spending, or evaluate crossing probabilities with a joint null distribution that does not match the design.