Evaluating Interim Futility Rules

Why Evaluate Futility Rules by Simulation?

An interim futility rule can reduce the number of participants exposed to an ineffective treatment and avoid spending resources on a trial unlikely to succeed. The trade-off is that interim data are uncertain: a trial that appears unpromising can still produce a successful final result if it continues. Evaluating a rule therefore requires more than calculating its value for one interim result. Simulation estimates how often the rule stops across repeated trials, how many participants are expected to be enrolled, and how often stopping sacrifices a result the continued trial would have achieved.

This tutorial focuses on futility rules based on conditional power or predictive power. These are decision summaries calculated at an interim analysis. Their value depends on the assumed model, the effect used or integrated over, and the stopping threshold. Simulation should represent the proposed data-collection and analysis process, including timing, missingness, and the final success criterion where relevant. The earlier material on simulating event-driven trials and unequal follow-up is especially relevant when the interim is defined by accrued information rather than a fixed participant count.

Two Interim Measures of Future Success

For a compact derivation, consider a one-sided normal test with a positive treatment effect. Let $t$ be the fraction of planned final information available at interim, with $0<t<1$. Let $Z_I$ be the standardized interim statistic, $Z_F$ the final statistic, and $c$ the prespecified critical value for final success. Suppose the effect is represented by a fixed standardized parameter $\delta$, and assume independent normally distributed information increments with known variance. Under this model,

Z_F=\sqrt{t}\,Z_I+\sqrt{1-t}\,Z_{\mathrm{new}}, \qquad Z_{\mathrm{new}}\sim N\!\left(\delta\sqrt{1-t},1\right).

The first term carries the interim information into the final statistic; the second represents information collected after interim. Conditional on $Z_I=z_I$ and a specified $\delta$, the future increment is independent of the interim data. Therefore $Z_F$ has conditional mean $\sqrt{t}z_I+\delta(1-t)$ and conditional variance $1-t$.

Definition: Conditional power is the probability that the final analysis meets its success criterion, conditional on the observed interim data and on a specified assumption about the effect (and any other unknown quantities). In this normal example, for the one-sided criterion $Z_F>c$, $$ CP(z_I;\delta)= \Phi\!\left( \frac{\sqrt{t}z_I+\delta(1-t)-c}{\sqrt{1-t}} \right), $$ where $\Phi$ is the standard normal cumulative distribution function.

The assumed effect might be the design alternative, a clinically relevant effect, or an estimate from interim data. Those choices produce different conditional power values and must be fixed in advance. A plug-in estimate is not the same as knowing the true effect: it can be noisy, particularly with little interim information.

Predictive power averages the probability of final success over a specified distribution for uncertain quantities, rather than fixing the effect at one value. One possible construction uses a probability distribution for the effect updated with interim data. This calculation is predictive in the sense that it averages over both remaining trial outcomes and effect uncertainty; it should not be confused with conditional power evaluated at a single assumed effect. The distribution and updating procedure are part of the rule and must be prespecified.

Worked Example: Conditional Power Under a Fixed Effect

Suppose a normally distributed endpoint is analyzed at $t=0.50$, the interim statistic is $z_I=0.80$, the assumed standardized effect is $\delta=0.40$, and the final one-sided critical value is $c=1.96$. Calculate conditional power.

The assumptions are the independent normal information increments above, a fixed effect of 0.40 standardized units, and final success defined by $Z_F>1.96$. Substituting into the conditional-power formula gives

CP= \Phi\!\left( \frac{\sqrt{0.50}(0.80)+0.40(1-0.50)-1.96}{\sqrt{1-0.50}} \right) = \Phi\!\left(\frac{0.5657+0.2000-1.960}{0.7071}\right) = \Phi(-1.689) \approx 0.0456.

Under these assumptions, the probability of final success conditional on this interim result is about 4.56%. If the futility rule stops when conditional power is below 10%, this interim result meets the stopping criterion. That calculation alone does not establish the rule’s overall stopping rate or its effect on expected enrollment; those require repeated-trial evaluation.

From a Futility Rule to Operating Characteristics

A rule needs an explicit metric, threshold, and direction. For example, “stop for futility if conditional power is below 10%” means stop when $CP<0.10$. A predictive-power rule might use the same threshold, but its power measure may have a different value because it averages over effect uncertainty. A reproducible simulation also specifies the interim timing, the final success criterion, the data-generating scenarios, and the analysis used to calculate the interim metric.

Under a simulated true effect, the stopping probability is the proportion of trials that meet the futility criterion. The expected sample size is the average number of participants enrolled after applying the rule. To quantify the chance of stopping a truly effective trial, specify what “truly effective” means—for example, a particular positive treatment effect of clinical interest—and estimate the stopping probability under that scenario. A more direct measure of potentially forgone success is the joint probability that a trial stops for futility and would have met the final success criterion had it continued. This last quantity requires simulating the post-interim outcomes even for trials designated to stop; those outcomes are a counterfactual, not observed trial data.

Specify scenarios and the rule.
Set the true effects and other data-generating conditions, the interim timing, the final criterion, the futility metric, and its threshold.

Simulate and analyze each interim.
Generate interim data, apply the planned interim analysis, calculate conditional or predictive power, and record whether the rule stops the trial.

Track enrollment and continuation.
Record the sample size at stopping or, if the trial continues, the planned final sample size. For counterfactual success, also generate the post-interim outcomes in stopped replicates.

Summarize Monte Carlo results.
Estimate stopping probability, mean sample size, and—if simulated—joint stopping and continued-trial success. Report simulation uncertainty as well as the estimates.

Worked Example: Predictive Power With Effect Uncertainty

Continue with $t=0.50$, $z_I=0.80$, and $c=1.96$, but now suppose the standardized effect has a normal prior distribution $\delta\sim N(0.40,0.20^2)$. Assume the interim likelihood is $Z_I\mid\delta\sim N(\delta\sqrt{t},1)$. Find the posterior predictive probability of final success under this model.

The normal prior and likelihood give a normal posterior. Its variance and mean are

v_1=\left(\frac{1}{0.20^2}+t\right)^{-1} =\frac{1}{25+0.50}=0.03922, \qquad m_1=v_1\left(\frac{0.40}{0.20^2}+\sqrt{0.50}(0.80)\right) =0.4143.

Given the interim result, the final statistic has mean $\sqrt{t}z_I+(1-t)m_1$. Its predictive variance includes both future sampling variation and the remaining uncertainty about $\delta$:

\begin{aligned} E(Z_F\mid z_I)&=\sqrt{0.50}(0.80)+0.50(0.4143)=0.7729,\\ \operatorname{Var}(Z_F\mid z_I)&=(1-t)+(1-t)^2v_1 =0.50+0.25(0.03922)=0.5098. \end{aligned}

Thus the predictive standard deviation is $\sqrt{0.5098}=0.7140$, and

P(Z_F>1.96\mid z_I) = \Phi\!\left(\frac{0.7729-1.96}{0.7140}\right) = \Phi(-1.663) \approx 0.0482.

This is about 4.82% under the stated prior, likelihood, and success criterion. It is close to, but not identical to, the fixed-effect conditional power in the previous example. The calculation illustrates why a predictive-power rule must state the uncertainty distribution it uses; changing that distribution can change the stopping decision.

Simulation Estimates and Their Uncertainty

If $M$ independent simulated trials are run and $S$ stop for futility, the estimated stopping probability is $\hat p_{\mathrm{stop}}=S/M$. Its approximate Monte Carlo standard error is

SE(\hat p_{\mathrm{stop}}) \approx \sqrt{\frac{\hat p_{\mathrm{stop}}(1-\hat p_{\mathrm{stop}})}{M}}.

If trial $j$ enrolls $N_j$ participants after applying the rule, the estimated expected sample size is $\bar N=M^{-1}\sum_{j=1}^{M}N_j$. Its Monte Carlo standard error is the sample standard deviation of the $N_j$ values divided by $\sqrt{M}$. These standard errors quantify simulation noise, not uncertainty about whether the assumed trial model is correct. The earlier material on validating simulation code, checking simulated distributions, and assessing estimator bias and coverage gives complementary checks on the simulation and analysis.

Worked Example: Stopping Rate and Sample Size From Simulated Trials

Suppose 10,000 hypothetical trials are simulated under a specified clinically effective true effect. Each trial has a planned maximum of 200 participants, an interim at 100 participants, and a plug-in conditional-power rule that stops when the interim estimate gives conditional power below 10%. The simulation records 3,680 stops. In addition, post-interim outcomes are generated for every stopped replicate to assess whether continuation would have met the final criterion; 480 replicates both stop and meet that counterfactual criterion.

The estimated stopping probability is

\hat p_{\mathrm{stop}}=\frac{3680}{10000}=0.368, \qquad SE(\hat p_{\mathrm{stop}}) \approx \sqrt{\frac{0.368(0.632)}{10000}} =0.00482.

The simulation estimates a 36.8% probability of stopping under this particular effective-trial scenario; the Monte Carlo standard error is 0.482 percentage points. Assuming all non-stopped trials enroll 200 participants, expected enrollment is

\bar N= \frac{3680(100)+6320(200)}{10000} =163.2\text{ participants}.

The estimated probability of stopping and thereby forgoing a final success that would have occurred on continuation is $480/10000=0.0480$, or 4.80% of all simulated trials. Among stopped trials, the corresponding fraction is $480/3680=0.1304$, or 13.0%. These are different summaries: the first uses all trials as its denominator, while the second conditions on stopping. The outcomes used for the counterfactual calculation are simulated continuations, not outcomes that would be observed after a real futility stop.

Assumptions, Diagnostics, and Alternatives

The normal calculations above assume a correctly specified normal information model, independent interim and future information increments conditional on the effect, a correctly represented information fraction, and a fixed final critical value. They are useful for deriving and checking a rule, but an actual trial may require a more detailed data-generating model. For example, unequal follow-up, delayed events, dropout, or an interim triggered by event accrual can affect the amount and composition of information. In those settings, simulate participant-level follow-up and the planned analysis rather than relying on a simple normal approximation unless it has been justified.

Check the simulated design. Verify interim timing, accumulated sample size or information, allocation, follow-up, and maximum enrollment in each replicate against the protocol.
Check the interim calculation. Reproduce the conditional or predictive power from selected simulated datasets using an independent calculation. Confirm the effect scale and whether the rule uses a design effect, an interim estimate, or an uncertainty distribution.
Check boundary behavior. Test interim values clearly on either side of the futility threshold, and verify that equality is treated as specified. A coding error in the inequality can reverse the decision.
Check Monte Carlo precision. Increase $M$ when estimated probabilities are noisy, especially for rare stopping or joint-success events. A small Monte Carlo standard error does not correct model misspecification.
Check the effective-trial scenario. Report stopping rates across a range of true effects, including effects considered clinically important. A rule may behave differently under a modest benefit than under the design alternative.

Conditional power is transparent and can be evaluated under a deliberately chosen effect assumption, but a plug-in effect estimate can be unstable at an interim. Predictive power accounts for uncertainty by integrating over a specified distribution, but its result can depend strongly on that distribution. A fixed design without interim futility avoids this particular early-stop risk but does not gain the same opportunity to reduce enrollment. These are alternatives with different assumptions and consequences, not interchangeable calculations. Futility stopping should not be interpreted as evidence that the treatment has no effect. Its purpose is to implement a prespecified decision under uncertainty.

A further distinction is whether a futility rule is binding in the trial’s conduct and analysis. Regardless of that label, simulate the rule as it will actually be applied and assess the complete design’s operating characteristics under the planned final analysis. Do not infer error-rate properties solely from the futility threshold. The next topic considers interim efficacy boundaries; here, the emphasis is the operating impact of stopping for low prospects of success.

Common Mistake

Common mistake: “Conditional power is 4.6%, so the treatment has a 4.6% chance of working.” This confuses a model-based probability of meeting a specified final statistical criterion, conditional on the interim data and chosen effect assumptions, with the probability that a treatment is effective. Correction: state the criterion, assumptions, and conditioning explicitly; then use simulation across defined true-effect scenarios to quantify how often the rule stops and what outcomes might be forgone.

Check Your Understanding

Answer each question, then compare with the solutions below.

In the normal information model, which quantities determine conditional power at interim, and what does the effect assumption represent?
If 250 of 2,500 simulated trials stop for futility, calculate the estimated stopping probability and its approximate Monte Carlo standard error.
A simulation records 60 trials that stop and would have succeeded if continued, among 2,500 total trials. What are the estimated overall joint probability and the estimated fraction among stopped trials if 250 stopped?
Why should a predictive-power rule state the distribution used for uncertain effects?

Solutions. (1) Conditional power depends on interim information $t$, interim statistic $z_I$, final threshold $c$, and the assumed effect $\delta$, through $\Phi((\sqrt{t}z_I+\delta(1-t)-c)/\sqrt{1-t})$. The effect assumption is the specified value used to represent the effect over the remaining information, not a guarantee about the true effect. (2) The stopping estimate is $250/2500=0.100$. Its approximate standard error is $\sqrt{0.100(0.900)/2500}=0.00600$, or 0.600 percentage points. (3) The overall joint probability estimate is $60/2500=0.0240$, or 2.40%; among stopped trials it is $60/250=0.240$, or 24.0%. (4) Predictive power averages over uncertainty in the effect, so changing its distribution changes the averaging and may change both the calculated power and the stopping decision.