Bias After Futility-Based Continuation

Why Futility-Based Continuation Changes Estimation

A futility rule can prevent a trial from using additional participants and resources when interim results make a worthwhile treatment effect look unlikely. But the rule also determines which trials reach the final analysis. If the final treatment-effect estimate is reported only for trials that continue, it is calculated from a selected set of interim results. Even when every stage-specific estimate is unbiased before selection, the final estimate among continued trials need not be unbiased.

The direction of the shift follows from the rule. When continuation requires the interim estimate to exceed a lower threshold, trials with relatively low interim estimates are more likely to stop. Among those that continue, the interim estimate therefore tends to be above its unselected mean. The pooled final estimate inherits some of that upward shift. This is a consequence of selection, not evidence that the treatment effect itself changed.

Two-Stage Model and Continuation Rule

Consider a fixed-effect treatment contrast \(\theta\), such as a risk difference measured in percentage points. Let \(\hat{\theta}_1\) be the estimate from stage 1 and \(\hat{\theta}_2\) the estimate from stage 2. Assume that, before applying the continuation rule,

where \(s_1\) and \(s_2\) are known standard errors for this calculation. Assume the stage estimates are independent, the same treatment effect \(\theta\) applies in both stages, and stage 2 is conducted only after continuation. The model treats the stage 2 estimate, conditional on continuation, as unbiased for \(\theta\) and independent of the stage 1 estimate. These assumptions can be reasonable for independent stages with a prespecified stage 2 design, but they are not automatic in every adaptive trial.

Suppose the trial continues when \(\hat{\theta}_1>c\), where \(c\) is a prespecified futility threshold on the effect-estimate scale. A rule based on a standardized interim statistic \(Z_1=\hat{\theta}_1/s_1\) continues when \(Z_1>f\); this is equivalent to \(c=fs_1\). The calculations below use an upper-tail continuation rule: higher interim estimates make continuation more likely.

If both stages are combined using inverse-variance weighting, the final estimate, when the trial continues, is

The weight \(w\) is the stage 1 share of total information. The expression is a precision-weighted average; without selection, its expectation is \(\theta\). With selection, the expectation of the stage 1 contribution changes.

Deriving the Conditional Bias

Standardize the stage 1 estimate as \(Z=(\hat{\theta}_1-\theta)/s_1\), so \(Z\sim N(0,1)\). Continuation is the event \(Z>a\), where

Write \(\phi\) and \(\Phi\) for the standard normal density and cumulative distribution function. The continuation probability is \(P(C)=1-\Phi(a)\). For a standard normal variable, integration of \(z\phi(z)\) over the upper tail gives \(\phi(a)\), because \(\phi'(z)=-z\phi(z)\). Dividing by the probability of that tail gives the conditional mean:

The function \(\lambda(a)\) is the upper-tail inverse Mills ratio. Since \(\hat{\theta}_1=\theta+s_1Z\), the conditional mean of the stage 1 estimate is \(\theta+s_1\lambda(a)\). Independence and unbiasedness give \(E(\hat{\theta}_2\mid C)=\theta\). Therefore,

For finite \(a\), \(\lambda(a)>0\). Thus, under this model and continuation rule, the final estimate among continued trials has positive conditional bias. The size of the bias depends on how much weight stage 1 receives, the stage 1 standard error, and the threshold relative to the true effect. It is not determined by the threshold alone.

How the Rule Changes the Distribution

The conditional mean does not describe the whole distribution. Given continuation, the stage 1 estimate follows a lower-truncated normal distribution: it retains only values above \(c\). The final estimate combines that truncated variable with an independent normal stage 2 estimate. It is therefore generally not normally distributed, even though both stage estimates were normal before selection.

The truncated stage 1 estimate has conditional variance

This follows from the standard normal upper-tail moments: \(E(Z\mid Z>a)=\lambda(a)\) and \(E(Z^2\mid Z>a)=1+a\lambda(a)\). Subtracting the squared conditional mean gives the variance formula. By independence, the conditional variance of the pooled estimate is

Selection shifts the stage 1 mean and changes its variance; the final distribution also inherits the stage 1 truncation. The variance expression is useful for describing spread under this model, but by itself it does not make a symmetric normal interval valid for the selected estimate.

Interpretation, Diagnostics, and Alternatives

The bias above is conditional on continuation. It is not the unconditional bias of every possible reporting procedure. If the final pooled estimate is reported only for trials that continue, its relevant expectation under the model is \(E(\hat{\theta}\mid C)\). If a different estimate is reported when the trial stops, the overall expectation depends on that separate rule and on the probability of stopping. These are different summaries and should not be conflated.

The result also clarifies why futility-based continuation does not guarantee an unbiased estimate among successful continuers. A futility threshold is a decision rule, not a correction to the distribution. Comparing the rule with alternatives helps make the estimand and reporting choice explicit:

Before using the calculation, check that the continuation rule truly has the assumed direction and threshold, the effect scale and standard errors are consistent, and the stage estimates are reasonably approximated by independent normal variables. Also verify that the stage 2 design is not itself altered in a way that changes its conditional expectation or uncertainty. For binary outcomes with sparse data, estimated standard errors, substantial missingness, or a data-dependent stage 2 allocation, this simple model may not adequately represent the sampling distribution.

A common mistake is to argue, “Both stage estimates are unbiased, so their weighted average remains unbiased among trials that continue.” That reasoning omits the selection event. In general, \(E(\hat{\theta}_1\mid C)\ne E(\hat{\theta}_1)\): here the conditional mean is \(\theta+s_1\lambda(a)\), not \(\theta\). The stage 2 estimate remains unbiased under the independence assumption, but its presence only reduces the shift according to its weight; it does not remove it.