Predictive Probability Rules for Early Efficacy

Why Use a Predictive Rule for Early Efficacy?

In a single-arm phase 2 trial, an interim response count can look promising without establishing whether the planned terminal success criterion is likely to be met. A predictive probability rule asks a forward-looking question: given the outcomes observed so far and the specified model, how likely is the trial to meet its terminal success criterion if it continues as planned? The rule can use that probability to stop early when the prespecified threshold is exceeded.

This is a design rule, not a replacement definition of terminal success. A high predictive probability means the model assigns high probability to the required future trial outcome; it does not mean that the terminal criterion has already been observed. The rule is meaningful only after specifying the terminal criterion, planned sample size, interim analysis, model and threshold.

Definition: A predictive-probability rule for early efficacy specifies a threshold \(\gamma\) before the trial. At an interim analysis, calculate the posterior predictive probability of meeting the terminal success criterion, conditional on the observed data and the planned trial. Stop for early efficacy if that probability is greater than \(\gamma\). The rule is applied only at the interim looks specified in the design.

Model and Predictive Probability

For a binary response endpoint, let \(N\) be the planned total number of participants with observed outcomes, \(m\) the number with observed outcomes at an interim analysis, and \(x\) the number of responses among those \(m\). Let \(p\) denote the response probability. The standard beta-binomial setup assumes conditionally independent participant outcomes with response probability \(p\), and a beta prior for that probability:

Y_i\mid p \sim \operatorname{Bernoulli}(p), \qquad p\sim \operatorname{Beta}(a,b), \qquad X_m\mid p \sim \operatorname{Binomial}(m,p).

Here \(a>0\) and \(b>0\) are fixed prior parameters. After observing \(x\) responses among \(m\) participants, the posterior distribution is \(\operatorname{Beta}(A,B)\), where \(A=a+x\) and \(B=b+m-x\). Assume that terminal success is defined as at least \(c\) responses among all \(N\) participants. There are \(r=N-m\) planned outcomes remaining. If \(K\) is the number of responses among those remaining participants, the trial meets its terminal criterion when \(x+K\ge c\).

P(\text{terminal success}\mid x,m) = P(K\ge c-x\mid x,m) = \sum_{k=\max(0,c-x)}^{r} {r\choose k} \frac{B(A+k,B+r-k)}{B(A,B)}.

In this expression, \(B(u,v)\) is the beta function, and a sum with a lower limit greater than \(r\) is zero. If \(c-x\le 0\), the current responses already guarantee the count criterion, so the predictive probability is one. If \(c-x>r\), there are not enough remaining participants to reach the criterion, so it is zero. The usual case for an early look has \(0<c-x\le r\).

Why the Beta-Binomial Formula Applies

Conditional on \(p\), the number \(K\) of responses among the \(r\) remaining participants has a binomial distribution. But \(p\) is uncertain after the interim data. The predictive probability averages the conditional binomial probability over the posterior distribution of \(p\). For each possible future response count \(k\), this gives the beta-binomial probability.

\begin{aligned} P(K=k\mid x,m) &=\int_0^1 P(K=k\mid p)\,\pi(p\mid x,m)\,dp\\ &={r\choose k}\frac{1}{B(A,B)} \int_0^1 p^{A+k-1}(1-p)^{B+r-k-1}\,dp\\ &={r\choose k}\frac{B(A+k,B+r-k)}{B(A,B)}. \end{aligned}

The second line uses the binomial probability \({r\choose k}p^k(1-p)^{r-k}\) and the beta posterior density. The integral is the beta function with parameters \(A+k\) and \(B+r-k\), which yields the final expression. Summing these probabilities over the future response counts that satisfy \(x+k\ge c\) gives the predictive probability of terminal success.

The calculation assumes the planned total \(N\) and terminal criterion \(c\) are fixed, the interim data are correctly summarized by \(x\) and \(m\), and the beta-binomial model is an appropriate representation of uncertainty in the response probability and future outcomes. The rule also presumes that future participants have the same response probability \(p\) as those already observed. Changes in eligibility, treatment delivery or outcome measurement can undermine that assumption.

Worked Example: A High Predictive Probability Triggers the Rule

Suppose a hypothetical single-arm trial plans \(N=10\) participants and defines terminal success as at least \(c=6\) responses. Use a \(\operatorname{Beta}(1,1)\) prior. At an interim analysis, \(m=4\) participants have observed outcomes, all four responses. The design uses \(\gamma=0.80\). Determine whether the rule stops for early efficacy.

The posterior is \(\operatorname{Beta}(1+4,1+0)=\operatorname{Beta}(5,1)\). Six participants remain, and at least two of them must respond. Thus \(r=6\) and the required future response count is \(K\ge2\). It is convenient to calculate the complement:

\begin{aligned} P(K\ge2\mid x=4,m=4) &=1-P(K=0)-P(K=1)\\ &=1-\frac{(1)_6}{(6)_6} -\frac{6(5)_1(1)_5}{(6)_6}\\ &=1-\frac{720}{332640}-\frac{3600}{332640}\\ &=0.9870. \end{aligned}

Here \((u)_j=u(u+1)\cdots(u+j-1)\) denotes the rising factorial. Since \(0.9870>0.80\), the prespecified rule stops the trial for early efficacy. In context, conditional on this model and these four observed responses, the posterior predictive probability of reaching at least six responses in ten participants is about 98.70%.

Applying the Threshold Consistently

The threshold \(\gamma\) distinguishes a design rule from an informal judgment that the interim results “look good.” It must be chosen before seeing the interim data. A larger threshold requires stronger predictive evidence before stopping; a smaller one makes early stopping easier. The threshold is a design choice, not a probability calculated from the data, and the numerical value alone does not establish that a design has acceptable error rates or desirable operating characteristics.

Worked Example: The Same Criterion Does Not Always Trigger Stopping

Use the same hypothetical design: \(N=10\), terminal success at \(c=6\), prior \(\operatorname{Beta}(1,1)\), and threshold \(\gamma=0.80\). At a different interim analysis, suppose \(m=4\) and \(x=2\). Apply the rule.

The posterior is \(\operatorname{Beta}(3,3)\), and six outcomes remain. At least four future responses are required. The beta-binomial distribution is symmetric in this case, so \(P(K\ge4)=P(K\le2)\). Its first three probabilities are:

\begin{aligned} P(K=0)&=\frac{(3)_6}{(6)_6} =\frac{20160}{332640}=0.06061,\\ P(K=1)&=\frac{6(3)_1(3)_5}{(6)_6} =\frac{45360}{332640}=0.1364,\\ P(K=2)&=\frac{15(3)_2(3)_4}{(6)_6} =\frac{64800}{332640}=0.1948. \end{aligned}

Therefore \(P(K\ge4)=0.06061+0.1364+0.1948=0.3918\). Since \(0.3918\) is not greater than \(0.80\), this rule does not stop for early efficacy. Not stopping under this rule is not a conclusion that the treatment is ineffective; it says only that the prespecified predictive-probability condition has not been met.

Worked Example: A Predictive Probability Near the Threshold

Consider a separate hypothetical design with \(N=8\), terminal success defined as at least \(c=6\) responses, prior \(\operatorname{Beta}(1,1)\), and threshold \(\gamma=0.59\). At the interim, \(m=4\) participants have been observed, of whom \(x=3\) responded. Calculate the probability and decision.

The posterior is \(\operatorname{Beta}(4,2)\). Four outcomes remain, and at least three must be responses. Calculate the probability of zero, one or two future responses, then take the complement:

\begin{aligned} P(K\ge3\mid x=3,m=4) &=1-\{P(K=0)+P(K=1)+P(K=2)\}\\ &=1-\frac{120+384+720}{3024}\\ &=\frac{1800}{3024}=0.5952. \end{aligned}

Because \(0.5952>0.59\), the rule stops for early efficacy. The margin over the threshold is small, which makes the comparison especially sensitive to implementation details such as rounding. The decision should be based on the unrounded probability: compare \(0.5952\) with \(0.59\), rather than rounding both values to \(0.6\).

Design Choices, Diagnostics and Limitations

A predictive rule is conditional on the model and the planned continuation of the trial. Before using it, verify that the endpoint is binary as specified, \(x\) and \(m\) count participants with observed outcomes, the remaining number is \(N-m\), and the success event is exactly the event intended at the terminal analysis. Check that the prior parameters and terminal criterion are the ones specified in the design. A predictive calculation based on the wrong remaining sample size answers a different question.

The operating characteristics of the complete rule matter as much as its interim probability. Repeatedly checking the rule and stopping when it is crossed can change the chance of a false efficacy conclusion and the chance of detecting a genuinely promising treatment compared with a fixed-sample design. Evaluate the full design under clinically relevant response probabilities, including probabilities corresponding to an ineffective treatment and to a promising treatment. This evaluation should include the specified prior, planned looks, stopping rule and terminal decision. A threshold such as 0.80 is not, by itself, a guarantee of any particular false-positive rate or power.

The main alternatives answer different questions. A fixed-sample design waits for the terminal analysis and avoids this particular early-stopping rule, though it may use more participants when the terminal result is already predictable. A posterior probability that \(p\) exceeds a clinical target concerns the unknown response rate directly; it is not the same as the probability that the future trial data will satisfy a terminal count criterion. Predictive-probability futility rules use a low probability to consider stopping for a different reason; that is a distinct decision rule.

Common pitfall: “The predictive probability is 0.90, so there is a 90% probability that the treatment is efficacious and terminal success has already been established.” This confuses a future-data event with a claim about the treatment or a result already observed. The correct interpretation is: under the specified model, prior, observed interim data and planned continuation, the probability of meeting the stated terminal criterion is 0.90. Whether stopping is appropriate depends on the prespecified design and its operating characteristics.

Other limitations follow from the assumptions. A prior that is poorly justified can materially affect the posterior and predictive probability, especially early in a small trial. A single common response probability may be inadequate if patients differ substantially or the treatment effect changes across subgroups or over time. Missing or delayed outcome data can also mean that the interim count does not represent the intended set of observed outcomes. These concerns call for checking the design assumptions and data definitions; they are not repaired by changing the threshold after seeing results.

Specify the terminal event.
State the planned total \(N\) and the precise criterion for terminal success, such as at least \(c\) responses.

Specify the interim rule.
Choose the look or looks, prior and predictive-probability threshold \(\gamma\) before observing interim results.

Calculate and compare.
Use the interim data to obtain the posterior predictive probability of the terminal event. Stop for early efficacy only if that probability exceeds \(\gamma\).

Evaluate the whole design.
Check model and data assumptions, then assess the stopping rule together with the terminal decision under relevant response probabilities.

Key takeaway: An early-efficacy predictive rule compares a model-based probability of future terminal success with a prespecified threshold. It is a rule for deciding when to stop, not evidence that the terminal criterion has already been met.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A trial plans \(N=12\) participants and defines success as at least 7 responses. At an interim analysis, 5 participants have observed outcomes and 3 responded. How many outcomes remain, and how many additional responses are needed?
In a beta-binomial analysis, what does a predictive probability of 0.85 mean if the terminal criterion is at least 6 responses among 10 participants?
A design stops for early efficacy when the predictive probability is greater than \(\gamma=0.80\). The computed probability is exactly 0.80. Does the specified rule stop?
Why is the posterior probability that \(p\) exceeds a response-rate target not generally interchangeable with the predictive probability of meeting a terminal response-count criterion?

Solutions. (1) There are \(12-5=7\) outcomes remaining. Since \(7-3=4\), at least four of those outcomes must be responses to reach seven total. (2) Conditional on the specified model, prior, observed interim data and planned continuation, the probability of reaching at least six responses in ten participants is 0.85. It does not say that terminal success has already occurred or that \(p\) has an 85% probability of exceeding a target. (3) No. The rule requires the probability to be greater than 0.80; equality does not satisfy that strict inequality. (4) The posterior probability about \(p\) concerns the unknown response rate, whereas the predictive probability concerns future trial outcomes and a terminal data criterion. The latter averages over uncertainty in \(p\) and the randomness of future responses.