Confidence Intervals After Futility Stopping

Why Futility Stopping Changes an Interval

A futility stop is triggered by data that are unfavorable to the prospect of demonstrating benefit. Consequently, the estimate available at that stop is not distributed like an estimate from a study whose sample size was fixed in advance: it is observed only when it satisfies the stopping rule. A fixed-sample interval can therefore have poor coverage among trials that stop for futility. The introductory material on conditional versus unconditional confidence coverage distinguishes these coverage targets; here the focus is on constructing an interval for the parameter conditional on the futility stop.

The construction below uses a one-look normal model with known variance and a fixed futility boundary. This is a useful setting because the selection effect can be written exactly. The same calculation should not be applied unchanged when the stopping event depends on multiple looks, estimated standard errors, or additional design features.

Model and stopping rule: Let \(\theta\) be the treatment effect, in its natural measurement units, and let \(I\) be the information at the interim analysis. Assume the standardized statistic \(Z\) has distribution \(Z\sim N(\theta\sqrt{I},1)\), equivalently the estimate \(\hat{\theta}=Z/\sqrt{I}\) has distribution \(N(\theta,1/I)\). The variance is known, the normal model is correct, \(I\) is fixed, and the futility boundary \(c_F\) is specified before examining the data. The trial stops for futility when \(Z\le c_F\).

Write \(m=\theta\sqrt{I}\), the mean of \(Z\). At a futility stop the observed value \(z\) satisfies \(z\le c_F\). The relevant sampling distribution is therefore the distribution of \(Z\) given \(Z\le c_F\), not its original untruncated normal distribution. The boundary is part of the information used for inference.

The Conditional Distribution and Its Inversion

Under a proposed value of \(\theta\), the probability of observing a statistic no larger than \(z\), conditional on having stopped for futility, is the probability of \(Z\le z\) divided by the probability of the stopping event. Since \(z\le c_F\), the event \(Z\le z\) is contained in \(Z\le c_F\).

F_{\theta}(z\mid Z\le c_F) = \frac{P_{\theta}(Z\le z)}{P_{\theta}(Z\le c_F)} = \frac{\Phi(z-\theta\sqrt{I})}{\Phi(c_F-\theta\sqrt{I})}, \qquad z\le c_F,

Here \(\Phi\) is the standard normal cumulative distribution function. The denominator depends on \(\theta\): a parameter value that makes a futility stop unlikely also changes the distribution of observations among the trials that did stop. Omitting this denominator discards the selection introduced by the rule.

Definition: An equal-tailed \(100(1-\alpha)\%\) conditional confidence interval after a futility stop is obtained by inverting the conditional distribution. Its lower endpoint \(\theta_L\) and upper endpoint \(\theta_U\) solve

F_{\theta_L}(z\mid Z\le c_F)=1-\frac{\alpha}{2}, \qquad F_{\theta_U}(z\mid Z\le c_F)=\frac{\alpha}{2}.

For fixed \(z<c_F\), the conditional distribution shifts toward the boundary as \(\theta\) increases, so its cumulative probability at \(z\) decreases. Thus the larger conditional tail probability gives the lower parameter endpoint, and the smaller tail probability gives the upper endpoint. Numerical root-finding is usually needed; the endpoints are not generally obtained by adding and subtracting a fixed normal critical value from \(\hat{\theta}\).

Why the Interval Has Conditional Coverage

The coverage statement follows directly from the conditional probability integral transform. For a continuous distribution, when data are drawn from that distribution, its cumulative probability evaluated at the observed value is uniformly distributed between zero and one. Apply this fact to the conditional distribution of \(Z\) given \(Z\le c_F\), under the true parameter \(\theta_0\). The interval includes \(\theta_0\) exactly when the observed conditional cumulative probability lies between the two tail cutoffs.

\begin{aligned} P_{\theta_0}(\theta_L\le\theta_0\le\theta_U\mid Z\le c_F) &=P_{\theta_0}\left( \frac{\alpha}{2} \le F_{\theta_0}(Z\mid Z\le c_F) \le 1-\frac{\alpha}{2} \,\middle|\, Z\le c_F\right)\\ &=1-\alpha. \end{aligned}

This is a conditional coverage guarantee: among repetitions that stop at this specified look under this specified rule, the interval contains the true effect with probability \(1-\alpha\), under the model assumptions. It is not a statement that the parameter has probability \(1-\alpha\) of being inside the interval after the data have been observed. It also does not, by itself, establish coverage across all possible stopping stages.

Worked Calculations

Worked Example: A 90% Conditional Interval After a Futility Stop

Suppose a hypothetical trial has information \(I=100\), stops if \(Z\le c_F=-1\), and observes \(z=-1.5\). Construct an equal-tailed 90% conditional interval for \(\theta\), measured in the original effect units.

Here \(\alpha=0.10\), so the endpoint equations use conditional probabilities \(0.95\) and \(0.05\). Let \(m=10\theta\). Substitution into the conditional distribution gives

\frac{\Phi(-1.5-m)}{\Phi(-1-m)}=0.95 \quad\text{for } \theta_L, \qquad \frac{\Phi(-1.5-m)}{\Phi(-1-m)}=0.05 \quad\text{for } \theta_U.

Solving these equations numerically gives \(m\approx-2.94\) for the lower endpoint and \(m\approx4.57\) for the upper endpoint. Since \(\theta=m/10\), the interval is approximately \((-0.294,\ 0.457)\) effect units.

The observed estimate is \(\hat{\theta}=z/\sqrt{I}=-1.5/10=-0.150\) effect units. The interval is asymmetric around that estimate. Its upper endpoint is relatively high because, under large positive effects, a stop at \(Z\le-1\) is unusual; conditioning on that rare event changes the distribution used to assess the observed value. Under the stated model, the interval has 90% coverage conditional on stopping at this futility boundary.

Worked Example: A Conditional Tail Probability for a Proposed Effect

Use the same hypothetical design, with \(I=100\), \(c_F=-1\), and observed \(z=-1.5\). Evaluate the conditional cumulative probability at the proposed effect \(\theta=0.20\) effect units.

The standardized mean under this proposed effect is \(m=\theta\sqrt{I}=0.20(10)=2\). Therefore

F_{0.20}(-1.5\mid Z\le-1) = \frac{\Phi(-1.5-2)}{\Phi(-1-2)} = \frac{\Phi(-3.5)}{\Phi(-3)} \approx \frac{0.0002326}{0.001350} \approx 0.1723.

Conditional on stopping, about 17.2% of statistics under \(\theta=0.20\) would be at or below the observed value. This calculation illustrates why the ordinary unconditioned probability \(\Phi(-3.5)\) is not the relevant conditional tail probability: it does not account for the fact that only values at or below \(-1\) lead to a futility stop. The calculation alone is not a confidence interval; interval endpoints require inverting the conditional probability at the chosen tail cutoffs.

Worked Example: Checking Fixed-Sample Coverage Among Futility Stops

Consider the same one-look design with \(I=100\) and \(c_F=-1\), but now suppose the true effect is \(\theta=0\). A conventional fixed-sample 90% interval uses the known standard error \(1/\sqrt{I}=0.1\) and the critical value 1.645. It contains the true effect precisely when \(|Z|\le1.645\).

Among trials that stop for futility, \(Z\le-1\). The fixed-sample interval covers zero only when \(-1.645\le Z\le-1\). Since \(Z\sim N(0,1)\) under the true effect,

P_0(\text{fixed interval covers }0\mid Z\le-1) = \frac{\Phi(-1)-\Phi(-1.645)}{\Phi(-1)} = \frac{0.1587-0.0500}{0.1587} \approx 0.685.

Thus its conditional coverage is about 68.5%, not 90%. The interval has been designed for the unselected normal sampling distribution, while the reported data have been selected by the stopping rule. This example does not say that every fixed-sample interval must have exactly this degree of undercoverage; it demonstrates the problem for these particular values.

Choosing and Checking the Interval

A conditional interval answers a specific question: what interval has the requested coverage among repetitions that stop for futility at this look? Other procedures target different coverage properties. A fixed-sample interval ignores the stopping event and can fail conditionally, as shown above. An interval based on the stopped sampling law can instead target coverage across possible stopping outcomes. Conditional and unconditional coverage are distinct goals; neither label alone establishes that an interval is appropriate for a particular reporting purpose.

The conditional construction is especially useful when inference is explicitly to be interpreted among futility-stopped trials and the stop event is fully specified. It has limitations. It relies on the assumed normal model, known variance, fixed information, and correctly implemented boundary. If the standard error is estimated, the statistic need not have the stated normal law. If stopping depends on several interim statistics, conditioning only on the final inequality \(Z\le c_F\) does not describe the full selection event. The conditional distribution must match the actual rule and observed stopping stage.

Several checks help identify errors in an analysis:

Check the event. Confirm that the observed statistic satisfies \(z\le c_F\), and that the denominator represents the probability of the actual futility event.
Check the direction of the endpoints. The lower effect endpoint uses conditional cumulative probability \(1-\alpha/2\); the upper endpoint uses \(\alpha/2\).
Check the scale. The root is for \(m=\theta\sqrt{I}\), so divide by \(\sqrt{I}\) to report an effect in its original units.
Check numerical computation. For extreme boundaries, both normal probabilities in the ratio can be very small. Use a numerically stable normal-tail calculation and verify that the endpoint equations reproduce their target probabilities.
Check the stated coverage target. Do not describe conditional coverage at one futility stop as unconditional coverage over every possible trial outcome.

Common mistake: “The estimate is normal with standard error \(1/\sqrt{I}\), so the usual estimate plus or minus a normal critical value is a 90% interval even after stopping.” The correction is that the reported estimate is observed only when \(Z\le c_F\). For conditional inference, use the truncated conditional distribution, including its parameter-dependent denominator, and invert its tail probabilities.

Check Your Understanding

Use the one-look model and equal-tailed construction above. The questions emphasize the conditioning event and the meaning of the resulting coverage.

For a futility rule \(Z\le c_F\) and an observed \(z\le c_F\), write the conditional cumulative distribution at \(z\).
In a 95% equal-tailed conditional interval, which conditional probabilities define the lower and upper effect endpoints?
Suppose \(I=25\). What is the standard error of \(\hat{\theta}\) under the stated model?
Why is a fixed-sample interval not guaranteed to have its nominal coverage among trials stopped for futility?

Solutions. (1) It is \(\Phi(z-\theta\sqrt{I})/\Phi(c_F-\theta\sqrt{I})\); the denominator is the probability of stopping. (2) The lower endpoint solves for \(0.975\), and the upper endpoint solves for \(0.025\). (3) The standard error is \(1/\sqrt{25}=0.2\) effect units. (4) The stop rule selects observations with \(Z\le c_F\), so their conditional distribution is truncated and differs from the unconditioned distribution used to calibrate the fixed-sample interval.