Calibrating Futility Boundaries Under a Design Alternative

Why Calibrate Futility Under a Design Alternative?

A futility rule is intended to stop a trial when the accumulating evidence makes the planned benefit unlikely to be demonstrated. Its operating characteristics therefore depend on what effect the design regards as worthwhile. A threshold chosen without specifying that effect has no unique beta interpretation: the probability of crossing the threshold changes with the assumed effect.

Here the calibration question is precise: under a specified design alternative, how should lower thresholds be chosen so that the probability of crossing a futility threshold by each scheduled look matches a prespecified cumulative beta-spending target? This is a probability-calibration problem. It is not, by itself, a complete calculation of the trial’s overall power or type II error when efficacy stopping can also occur.

Definition: Let $Z_i$ be the standardized test statistic at look $i$, and let $l_i$ be its lower futility threshold. Futility is crossed at look $i$ when $Z_i \le l_i$. Under the design alternative, the cumulative futility-crossing probability through look $i$ is $$ F_i=P_A\!\left(\bigcup_{j=1}^{i}\{Z_j\le l_j\}\right). $$ A cumulative beta-spending target $b_i$ specifies the desired value of $F_i$ under that alternative. The incremental target at look $i$ is $\beta_i=b_i-b_{i-1}$, with $b_0=0$.

Model and Calibration Assumptions

Use the canonical group sequential model for a one-sided test of a positive treatment effect. Let $t_i=I_i/I_K$ be the information fraction at look $i$, where $I_i$ is the information available at that look and $I_K$ is the planned final information. The looks are prespecified and ordered, so $0<t_1<\cdots<t_K=1$. Let $\Delta$ be the standardized effect under the design alternative at the planned final information. Under that alternative, the canonical model is

E_A(Z_i)=\Delta\sqrt{t_i}, \qquad \operatorname{Cov}_A(Z_i,Z_j)=\sqrt{\frac{t_i}{t_j}}\quad (i\le j).

Thus each $Z_i$ has variance 1, and the joint distribution across looks is multivariate normal. The covariance structure represents the correlation induced by accumulating information: an earlier statistic contributes to the later one. It is not appropriate to treat the looks as independent. In a particular application, this model is an approximation whose adequacy depends on the test-statistic construction and information-time assumptions.

The calibration also assumes the design alternative and the spending targets are chosen before observing interim outcomes. The targets must satisfy $0\le b_1\le b_2\le\cdots\le b_K\le1$; otherwise, they cannot represent cumulative probabilities. For the calculations in this tutorial, futility spending is defined by crossing the lower thresholds under the design alternative, without conditioning on whether an efficacy boundary would have stopped the trial first. This convention makes the futility-crossing calibration explicit. It does not assert that the resulting targets equal the probability of failing to reject in a design where efficacy stopping also competes with futility stopping.

From Cumulative Targets to First-Crossing Probabilities

A cumulative target includes paths that crossed a futility boundary earlier. Consequently, the probability newly allocated at look $i$ is not $b_i$, but the difference $b_i-b_{i-1}$. A path contributes to this increment only if it avoided all earlier futility thresholds and crosses the current one.

F_i = F_{i-1} + P_A(Z_1>l_1,\ldots,Z_{i-1}>l_{i-1},Z_i\le l_i).

Setting $F_i=b_i$ and $F_{i-1}=b_{i-1}$ gives the calibration equation

P_A(Z_1>l_1,\ldots,Z_{i-1}>l_{i-1},Z_i\le l_i) = b_i-b_{i-1} = \beta_i.

This equation is the central step. For a given set of earlier thresholds, the left side is a nondecreasing function of the current threshold $l_i$: raising $l_i$ makes it easier to cross the lower boundary. Under the continuous normal model, it can be evaluated using the joint distribution or a conditional distribution given the previous statistics. The current threshold is then found by solving for the value that makes the probability equal to the incremental target.

First-look result: At the first look there are no earlier futility thresholds to avoid. Since $Z_1\sim N(\Delta\sqrt{t_1},1)$, a positive first-look target $b_1$ gives $$ b_1=P_A(Z_1\le l_1) =\Phi(l_1-\Delta\sqrt{t_1}), \qquad l_1=\Delta\sqrt{t_1}+\Phi^{-1}(b_1), $$ where $\Phi$ is the standard normal cumulative distribution function.

Worked Example: Calibrating a First-Look Threshold

Suppose a hypothetical one-sided trial plans a look at $t_1=0.50$. The design alternative has final standardized effect $\Delta=2.00$, and the cumulative futility-spending target at this look is $b_1=0.025$. Find the lower threshold.

Under the design alternative, the mean of the first-look statistic is $$ \Delta\sqrt{t_1}=2.00\sqrt{0.50}=1.414. $$ The standard normal quantile for a lower-tail probability of 0.025 is $\Phi^{-1}(0.025)=-1.960$. Therefore, $$ l_1=1.414+(-1.960)=-0.546. $$

The calibrated rule is to cross futility if $Z_1\le-0.546$. Under the stated design alternative and model, the probability of doing so at this look is $$ P_A(Z_1\le-0.546) =\Phi(-0.546-1.414) =\Phi(-1.960) =0.025. $$ This probability is a futility-crossing probability under the design alternative, not the probability that the null hypothesis is true.

Calibrating a Later Look

At a later look, applying the one-look quantile formula to the cumulative target generally gives the wrong threshold. It counts paths that have already crossed futility as if they were newly crossing at the current look. The correct calculation subtracts earlier-crossing paths, or, equivalently, works with the probability of avoiding earlier futility thresholds and crossing now.

For two looks, the canonical model gives a particularly direct conditional calculation. If $t_1<t_2$, then under the design alternative,

Z_2\mid Z_1=x \sim N\!\left( \Delta\sqrt{t_2} +\sqrt{\frac{t_1}{t_2}}\bigl(x-\Delta\sqrt{t_1}\bigr), \ 1-\frac{t_1}{t_2} \right).

The conditional mean includes the observed departure of $Z_1$ from its design-alternative mean. The conditional variance is less than 1 because the two statistics share information. Integrating the conditional lower-tail probability over values $Z_1>l_1$ gives the new first-crossing probability at look 2.

Worked Example: Calibrating a Second Look After an Earlier Boundary

Consider a hypothetical two-look design with $t_1=0.50$, $t_2=1.00$, and $\Delta=2.00$. Set cumulative targets $b_1=0.020$ and $b_2=0.050$. Find the first threshold and specify how the second is calibrated.

At look 1, the mean is $2.00\sqrt{0.50}=1.414$. Since $\Phi^{-1}(0.020)\approx-2.054$, $$ l_1=1.414-2.054=-0.640. $$ The additional probability to be spent at look 2 is $$ \beta_2=b_2-b_1=0.050-0.020=0.030. $$

Here $Z_2\mid Z_1=x$ has mean $2.00+\sqrt{0.50}(x-1.414)=1.00+0.7071x$ and variance $1-0.50=0.50$. The second threshold must therefore solve $$ \int_{-0.640}^{\infty} \phi(x-1.414) \Phi\!\left( \frac{l_2-(1.00+0.7071x)}{\sqrt{0.50}} \right)\,dx =0.030, $$ where $\phi(x-1.414)$ is the density of $Z_1\sim N(1.414,1)$. The lower limit restricts the integral to paths that did not cross futility at look 1.

Numerical evaluation of this one-dimensional integral gives $l_2\approx0.249$. Substituting that threshold into the integral gives a new crossing probability of approximately 0.030. Adding the earlier crossing probability yields $$ F_2\approx0.020+0.030=0.050. $$ Using the marginal probability $P_A(Z_2\le l_2)$ alone would not perform this calculation: it would include paths already counted in $b_1$.

Checking the Spending Schedule and the Calibration

A spending schedule is a sequence of probability targets, not a sequence of thresholds. The targets must be cumulative and nondecreasing, while the threshold at each look is determined by both the current increment and the distribution of paths that remain eligible to cross. In particular, a smaller increment does not translate into a simple fixed change in the threshold across looks: the mean, information fraction, dependence on earlier statistics, and earlier boundaries all matter.

Worked Example: Checking a Three-Look Spending Plan

A hypothetical design has information fractions $t_1=0.40$, $t_2=0.70$, and $t_3=1.00$, with design alternative $\Delta=1.50$. A proposed cumulative spending schedule is $b_1=0.020$, $b_2=0.045$, and $b_3=0.080$. Check its increments and calculate the first threshold.

The increments are $$ \beta_1=0.020,\qquad \beta_2=0.045-0.020=0.025,\qquad \beta_3=0.080-0.045=0.035. $$ They are all nonnegative, and the cumulative targets are nondecreasing, so the schedule passes this basic coherence check.

At the first look, the design-alternative mean is $$ 1.50\sqrt{0.40}=0.949. $$ Using $\Phi^{-1}(0.020)\approx-2.054$, the first lower threshold is $$ l_1=0.949-2.054=-1.105. $$ At look 2, the calibration must allocate a further probability of 0.025 to paths with $Z_1>-1.105$. At look 3, it must allocate 0.035 to paths that avoided both previous futility thresholds. The targets alone do not give those later thresholds by marginal quantiles; the correlated joint distribution must be used.

Specify the probability model.
State the design alternative, the information fractions, and the canonical joint distribution assumed for the sequential statistics.

Check the cumulative targets.
Verify that each target lies between 0 and 1 and that the sequence is nondecreasing. Compute $\beta_i=b_i-b_{i-1}$.

Calibrate in order.
Calculate the first threshold from its normal quantile, then solve each later first-crossing equation using the thresholds already determined.

Verify the result.
Evaluate the cumulative crossing probabilities under the design alternative and check that they match the specified targets, subject to numerical tolerance.

Diagnostics, Comparisons, and Limitations

A useful diagnostic is to recompute the cumulative probabilities after obtaining the thresholds. At each look, add the probability of a new crossing to the probability accumulated earlier. The resulting values should reproduce $b_i$, up to numerical error. A material mismatch can indicate that the calculation used a marginal rather than first-crossing probability, that the covariance structure or information fractions were entered incorrectly, or that a solver did not find the intended root.

The calibration also has clear edge cases. If $b_i=b_{i-1}$, the incremental target is zero. Under a continuous model with positive probability of reaching the current look without an earlier crossing, a finite threshold will generally assign positive probability to a new crossing. The boundary must be treated as a limiting no-spending boundary or handled according to the design’s prespecified convention. A negative increment is not a difficult numerical case; it is an incoherent cumulative spending schedule and must be corrected before calibration.

The main alternative to spending-based calibration is to choose futility thresholds directly according to another design objective, such as a desired conditional-power rule or a particular operational stopping policy. Such thresholds need not yield a prespecified cumulative crossing probability at each look. Beta spending instead makes those probabilities explicit under a stated design alternative, but it does not make the choice of alternative or spending schedule automatic.

The central limitation is that calibration is conditional on the probability model and the specified design alternative. If the actual effect differs from that alternative, the futility-crossing probabilities will differ from the targets. In addition, the calculation here tracks lower-boundary crossing without conditioning on prior efficacy stopping. When efficacy and futility boundaries jointly determine whether the trial stops, the probability of a particular stopping outcome requires that joint stopping rule to be accounted for. Thus the calibrated beta-spending targets should not be casually interpreted as overall type II error or as the probability of a futility stop in every implementation.

Common mistake: “The cumulative target at look 2 is 0.05, so choose the look-2 threshold to make $P_A(Z_2\le l_2)=0.05$.” This uses the marginal distribution and counts paths that already crossed the first futility threshold. The correction is to allocate only $b_2-b_1$ to new crossings and calculate $P_A(Z_1>l_1,Z_2\le l_2)$, or its appropriate higher-look counterpart.

Check Your Understanding

Answer each question using the design-alternative calibration convention stated above. Solutions follow the questions.

In one sentence, distinguish a cumulative beta-spending target from an incremental target.
A design has $b_1=0.015$ and $b_2=0.040$. What probability is allocated to new futility crossings at look 2?
At the first look, $t_1=0.25$, $\Delta=2.00$, and $b_1=0.050$. Calculate $l_1$, to three decimal places.
Why is a decreasing sequence of cumulative beta targets invalid?
Does a calibrated cumulative futility-crossing target automatically equal the trial’s overall type II error when efficacy stopping is also possible? Explain briefly.

Solutions. (1) The cumulative target is the probability of at least one futility crossing by a look; the incremental target is the probability newly allocated to first crossing at that look. (2) $\beta_2=0.040-0.015=0.025$. (3) The first-look mean is $2.00\sqrt{0.25}=1.000$, and $\Phi^{-1}(0.050)\approx-1.645$, so $l_1=1.000-1.645=-0.645$. (4) A cumulative probability cannot decrease as additional looks are included, so the proposed increments would include a negative probability. (5) No. The calibration describes lower-boundary crossing under the design alternative under its stated convention; overall type II error also depends on the trial’s full stopping and final testing rules.