Verifying Attained Beta Under A Futility Design

Why Verify Attained Futility Spending?

A lower boundary is usually calculated so that the probability of stopping for futility under a specified design alternative matches a planned spending target. The calculated boundary is only a numerical result, however. Its attained probability must be checked: an integration error, a boundary indexing error, or an omitted efficacy restriction can make a reported boundary look plausible while missing its target.

The check is based on the probability of a first futility crossing at each look. A path that crosses a lower boundary at an earlier look is no longer at risk of crossing at a later look in a binding design. Likewise, a path that crosses an upper efficacy boundary has already stopped and cannot subsequently count as a futility crossing. Thus the probability at look \(j\) is computed among paths that continued past all earlier looks.

This is a check of the futility-spending targets, not a replacement for evaluating the design’s overall Type II error. In particular, failure to cross an efficacy boundary at the final analysis can contribute to Type II error without being a futility stop.

Model and Recursive Calculation

Assume a one-sided group sequential design with \(K\) scheduled looks at information fractions \(0<t_1<\cdots<t_K=1\). Let \(Z_j\) denote the standardized test statistic at look \(j\), with lower futility boundary \(l_j\) and upper efficacy boundary \(u_j\), where \(l_j<u_j\). The design is binding: crossing either boundary ends the trial. Under the design alternative, suppose the canonical joint normal model holds:

Here \(\delta\) is the standardized effect at the final information level. Equivalently, the vector of statistics is multivariate normal with the stated means and correlations. These assumptions are appropriate when the information and effect model used to plan the design is adequate and the canonical joint-normal approximation is reasonable. Verification under a different effect or information pattern would be verification of a different design alternative.

For \(i<j\), the conditional distribution needed for recursion is \(Z_j\mid Z_i=z\sim N\bigl(\mu_j+\rho_{ij}(z-\mu_i),\,1-\rho_{ij}^2\bigr)\), where \(\mu_i=\delta\sqrt{t_i}\) and \(\rho_{ij}=\sqrt{t_i/t_j}\). This follows from the conditional distribution of a multivariate normal vector. In particular, consecutive looks can be evaluated by integrating over the previous look’s continuation region.

Let \(g_{j-1}(z)\) be the joint density contribution from paths that have continued through look \(j-1\) and have \(Z_{j-1}=z\). It is a sub-density: its integral is the probability of continuation, not necessarily one. At the first look, \(g_1(z)=\phi(z-\mu_1)\) on \(l_1<z<u_1\), and is zero elsewhere. For later looks, the continued-path density is obtained by conditioning on the previous statistic and retaining only values inside the current continuation region.

The probability of a first futility crossing at look \(j\) is the mass of the continued paths at the previous look multiplied by the conditional probability of falling at or below \(l_j\). For \(j=1\), it is simply the lower-tail probability at the first look. The attained cumulative probability is the sum of these increments.

If \(b_j\) is the planned cumulative futility-spending target through look \(j\), the numerical verification compares \(B_j\) with \(b_j\), using the same target definition and design alternative used to calculate the boundaries. The signed discrepancy \(B_j-b_j\) indicates whether the computed probability is above or below its target.

Worked Verification Examples

Justification and Numerical Checks

The recursion works because first-futility events at different looks are disjoint. A trial cannot first stop for futility at both look \(i\) and look \(j>i\). At look \(j\), conditioning on the previous statistic partitions the probability into the density of paths that survived through look \(j-1\), multiplied by the conditional probability \(P(Z_j\le l_j\mid Z_{j-1})\). Integrating over the previous continuation interval accounts for every path still eligible to stop at look \(j\), and excludes paths that already crossed either boundary. Summing the disjoint first-crossing probabilities therefore gives the cumulative probability.

This argument also gives useful consistency checks. Each increment \(q_j\) must be nonnegative, and the cumulative sequence must satisfy \(B_j\ge B_{j-1}\). The mass that first stops for futility, first stops for efficacy, or continues must add to one, up to numerical tolerance. A computed futility probability exceeding one, a decreasing cumulative total, or a material failure of this mass accounting signals an implementation or integration problem.

Numerical integration error and target discrepancy are different quantities. Integration error describes uncertainty in the computed approximation to \(B_j\); the discrepancy \(B_j-b_j\) compares that probability with the planned target. A useful report gives both, for example an attained value to adequate precision, its target, their difference, and a numerical error bound or stability check. Reporting many decimal places without checking integration stability does not make the result more reliable.

Diagnostics, Alternatives, and Limitations

The recursive normal-integral calculation is generally preferable to a crude grid over the full multivariate normal region: it uses the conditional structure and directly computes each first-crossing increment. A multivariate normal probability routine can provide an independent check when configured for the same stopping region. Simulation can also approximate operating probabilities, but its estimates have Monte Carlo variability and may be too noisy to resolve small differences from a spending target unless many replicates are used. Neither alternative corrects an incorrectly specified alternative, boundary, or stopping rule.

Several failures are especially common:

• Counting paths more than once. Adding marginal probabilities \(P(Z_j\le l_j)\) ignores earlier stopping. Use first-crossing probabilities conditional on continuation.
• Ignoring prior efficacy stopping. For a binding design, a path that crossed an upper boundary earlier is not eligible for later futility. Integrate only over the earlier continuation interval.
• Confusing cumulative futility with overall beta. Futility stops contribute to failure to reject, but final non-rejection can also contribute. State which probability is being verified.
• Using the wrong alternative. Attained futility probability depends on the distribution of the statistics. A boundary calibrated under one design alternative need not attain the same spending under another.
• Misreading quadrature tolerance. A numerical tolerance is not a confidence interval and does not describe uncertainty about the true treatment effect. It controls the numerical approximation, subject to the method’s implementation.

The check is only as reliable as the canonical model and the inputs. In a real design, changes in information, effect assumptions, or the operational rule may change the attained probabilities. The earlier material on binding futility establishes why both efficacy and futility crossings determine continuation; this verification applies that rule to assess the computed spending, rather than recalibrating boundaries or replacing the design’s operating-characteristic analysis.