Sensitivity Of Futility Boundaries To Effect Size

Why the Assumed Effect Matters

A beta-spending futility boundary is calibrated under an assumed treatment effect. That effect determines how likely the trial is to enter the lower, futility region at each analysis. If the assumed effect changes, a boundary designed to spend a specified amount of beta may no longer have the intended crossing probability. The direction is intuitive: under a larger beneficial effect, the test statistic tends to be higher, so a higher lower boundary is needed to retain the same lower-tail probability.

This sensitivity is important when the effect used for design is uncertain or when a proposed design is being adapted to a different target effect. A useful assessment separates two questions: what happens if the old boundaries are retained, and what new boundaries result if the same spending targets are recalibrated under a different effect? Those questions have different answers.

Model and Calibration Convention

Assume a normally distributed group sequential test statistic oriented so that larger values favour treatment. Let \(t_i=I_i/I_K\) be the information fraction at analysis \(i\), where \(I_i\) is the information at that analysis and \(I_K\) is the planned final information. Thus \(0<t_1<\cdots<t_K=1\). Write \(\theta\) for the assumed standardized drift at the final analysis. Under this design alternative, the canonical joint model is

In particular, \(Z_i\sim N(\theta\sqrt{t_i},1)\). The parameter \(\theta\) is measured in standard-deviation units of the final-analysis statistic; it is not an effect in the original outcome units. The corresponding treatment effect in original units must be translated through the test's standard error and information model.

For clarity, use a beta-spending convention in which \(B_i\) is the target cumulative probability of crossing a futility boundary by look \(i\), evaluated under the design alternative. The exact event must match the stopping rule: for a binding futility rule it includes the specified interaction with efficacy stopping, while for a nonbinding rule its calculation treats futility according to the chosen nonbinding convention. The convention must be held fixed when comparing effect assumptions. Cumulative spending targets satisfy \(B_1\leq\cdots\leq B_K\). A futility crossing probability is not automatically the overall type II error, which also involves final nonrejection and any competing stopping rules.

The canonical model assumes the planned information fractions are appropriate, the test statistics have the stated joint normal distribution, and the treatment effect is represented by a constant drift over the analyses. If these assumptions are unsuitable—for example, because information is materially misestimated or the effect changes over time—the calculated sensitivity may not describe the trial's actual behaviour.

One-Look Derivation of the Boundary Shift

At one analysis with information fraction \(t\), let the futility cutoff be \(l\), with futility declared when \(Z\leq l\). To spend a target amount \(B\) under the design alternative, require \(P_\theta(Z\leq l)=B\). Standardizing the normal variable gives

Here \(\Phi\) is the standard normal cumulative distribution function. Holding \(B\) and \(t\) fixed, the boundary's local and exact one-look change is

Thus a larger assumed beneficial effect shifts the cutoff upward in \(Z\)-units. The shift is smaller at an early look with little information and larger at a late look. This formula is exact for one marginal analysis; it is not, by itself, a multi-look boundary formula.

Holding a Boundary Fixed Changes Its Spending

If instead the original cutoff is retained after changing the assumed effect, its crossing probability generally changes. For a one-look boundary \(l\), the probability under a new effect \(\theta\) is

Its derivative with respect to \(\theta\) is \(-\sqrt{t}\,\phi(l-\theta\sqrt{t})\), which is negative because \(t>0\) and the standard normal density \(\phi\) is positive. A fixed lower cutoff therefore has a smaller crossing probability under a larger beneficial drift. This is a statement about futility crossing under the model, not a general statement that every measure of power changes by the same amount.

Multi-Look Sensitivity Requires Joint Recalibration

With multiple analyses, the statistics are correlated. A path that has already crossed a lower boundary cannot be counted again as a new first crossing. Consequently, cumulative spending at look \(i\) depends on all earlier boundaries, not just on the marginal distribution \(Z_i\). In a simple nonbinding calculation without a competing efficacy stop, the calibration requirement can be written as

A binding rule or a convention that excludes paths stopped for efficacy requires the corresponding joint event instead. Either way, changing \(\theta\) changes the joint distribution's mean vector. The boundaries must be recalculated using the same crossing-event definition and the same spending targets; applying the one-look formula independently to each look need not satisfy the cumulative targets.

For fixed boundaries, increasing \(\theta\) shifts each component mean upward while leaving the canonical covariance matrix unchanged. The probability of crossing at least one fixed lower boundary by a given look therefore decreases. But when the boundaries are recalibrated sequentially, the earlier cutoffs also change. The resulting later cutoff is determined by a joint probability calculation, so the simple shift \((\theta_2-\theta_1)\sqrt{t_i}\) is not generally an exact multi-look adjustment.

How to Assess Sensitivity in Practice

A defensible sensitivity assessment keeps the design choices that are not under investigation fixed. In particular, hold the information fractions, cumulative spending targets, efficacy-boundary convention, and binding or nonbinding rule constant. Then vary the assumed effect over a scientifically plausible range and recalibrate the futility boundaries for each value. For each recalibration, verify the achieved cumulative futility-crossing probabilities under that same assumed effect.

The main alternative to recalibration is a fixed-boundary sensitivity analysis: keep the originally chosen cutoffs and calculate how crossing probabilities change across effect values. This answers what the existing design would do if its assumptions were wrong. Recalibration answers what boundaries would be required to meet the same spending targets under each alternative. Neither analysis replaces the other; they address different design questions.

Useful diagnostics include plotting each recalibrated cutoff against \(\theta\), checking that the calibration probabilities match their targets within numerical tolerance, and recomputing probabilities under several effects while holding the boundaries fixed. Unexpected direction or large discontinuities can signal a sign error, an inconsistent crossing-event definition, incorrect information fractions, or numerical integration problems. In multi-look calculations, verify the joint crossing probability rather than checking only the marginal tail probability at each analysis.

Sensitivity analysis has limits. It cannot protect against arbitrary departures from the assumed model, and it does not establish that any effect value is clinically plausible. A single drift parameter may also be inadequate when treatment effects vary over time or across populations. Finally, a boundary change alone does not summarize the design: report the corresponding crossing and overall operating characteristics, with their definitions, so readers can see what the design gains or loses.

Common Mistake

Wrong reasoning: “The assumed effect increased by 0.5, so I can add \(0.5\) to every futility boundary and preserve the beta spending.” This ignores the information scale: even for one look, the shift is multiplied by \(\sqrt{t}\). In a multi-look design, it also ignores correlation and the dependence of cumulative crossing probabilities on earlier boundaries.

Correction: Use the one-look shift formula only for a marginal one-look calibration. For a sequential design, keep the spending targets and stopping-rule definition fixed, then solve and verify the joint calibration again under the new effect.