Predictive Probability Rules for Arm Dropping

Why Predictive Probability Is Different

At an interim analysis, an arm may not yet meet a success criterion, but additional patients could provide evidence that changes the final assessment. A predictive probability rule asks: given the data observed so far, how likely is it that this arm will meet a specified success criterion at its planned sample size? It therefore accounts for both uncertainty about the arm’s response rate and the outcomes that remain to be observed.

This is not the same as asking whether the arm currently has a high posterior probability of being effective. A current posterior probability concerns the response rate given the data already observed. A predictive probability concerns a future event: whether the completed trial’s analysis will satisfy a specified criterion. The distinction matters when an arm is uncertain but still has substantial enrollment remaining.

Definition: A predictive probability of success at an interim analysis is the posterior predictive probability, conditional on the observed data and the design assumptions, that the data available at the planned final analysis will satisfy a prespecified success criterion. A predictive-probability arm-dropping rule removes an arm when this probability crosses a prespecified dropping threshold.

Binary-Outcome Model and Calculation

Consider one treatment arm with a binary outcome, such as whether a patient responds by a specified assessment time. Let \(\theta\) be the probability of response. Assume that, conditional on \(\theta\), observed and future patient outcomes are independent Bernoulli variables with the same response probability. Use a Beta prior, \(\theta\sim\operatorname{Beta}(a_0,b_0)\), with \(a_0,b_0>0\). If \(x\) responses are observed among \(n\) patients, the posterior distribution is \(\operatorname{Beta}(A,B)\), where \(A=a_0+x\) and \(B=b_0+n-x\).

Suppose \(m\) additional patients are planned for this arm. Let \(Y\) be the number of responses among those \(m\) patients. For a particular future value \(y\), the final posterior distribution is \(\operatorname{Beta}(A+y,B+m-y)\). For illustration, define final success as having posterior probability at least \(\gamma\) that the response rate exceeds a benchmark \(p_0\). The final data meet the criterion when \(P(\theta>p_0\mid\text{current and future data})\geq\gamma\).

P(\theta>p_0\mid x,n,Y=y) =1-I_{p_0}(A+y,B+m-y), \qquad S(y)=\mathbf{1}\!\left\{1-I_{p_0}(A+y,B+m-y)\geq\gamma\right\}.

Here \(I_{p_0}(a,b)\) is the cumulative distribution function of a \(\operatorname{Beta}(a,b)\) random variable evaluated at \(p_0\), and \(S(y)\) equals 1 when the final success criterion is met and 0 otherwise. The predictive probability of success is the predictive probability that \(S(Y)=1\).

P(\text{final success}\mid x,n) =\sum_{y=0}^{m} S(y)\, \binom{m}{y} \frac{B(A+y,B+m-y)}{B(A,B)}.

The terms following \(S(y)\) are the beta-binomial predictive probabilities for the future number of responses. The formula averages the final success indicator over every possible future response count, weighting each count by its probability given the current posterior.

Why the Predictive Distribution Has This Form

Conditional on a particular response rate \(\theta\), the future response count has a binomial distribution. Because \(\theta\) is not known, the predictive probability of \(Y=y\) averages that binomial probability over the current posterior distribution of \(\theta\). This integration gives the beta-binomial expression.

\begin{aligned} P(Y=y\mid x,n) &=\int_0^1 P(Y=y\mid\theta)\,\pi(\theta\mid x,n)\,d\theta\\ &=\binom{m}{y}\int_0^1 \theta^y(1-\theta)^{m-y} \frac{\theta^{A-1}(1-\theta)^{B-1}}{B(A,B)}\,d\theta\\ &=\binom{m}{y} \frac{B(A+y,B+m-y)}{B(A,B)}. \end{aligned}

The last equality follows from the definition \(B(u,v)=\int_0^1 t^{u-1}(1-t)^{v-1}\,dt\). The predictive probabilities sum to 1 over \(y=0,\ldots,m\), as required for a probability distribution. The calculation assumes a fixed number \(m\) of additional patients and that the future outcomes follow the same response model as the observed outcomes.

Worked Example: Predictive Probability With a Planned Final Criterion

Suppose a hypothetical arm has a \(\operatorname{Beta}(1,1)\) prior. At an interim analysis, 8 of 20 patients have responded. Ten more patients are planned. Define final success as posterior probability at least \(0.90\) that \(\theta>0.30\). Calculate the predictive probability of final success.

The interim posterior is \(\operatorname{Beta}(1+8,1+20-8)=\operatorname{Beta}(9,13)\). Thus \(A=9\), \(B=13\), and \(m=10\). For each future count \(y\), the final posterior is \(\operatorname{Beta}(9+y,23-y)\). Evaluating its posterior probability above 0.30 shows that the final criterion is met for \(y\geq5\), but not for \(y\leq4\). For example, the relevant beta-tail probabilities at \(y=4\) and \(y=5\) are approximately 0.893 and 0.947, respectively, so 5 is the first passing count.

The predictive probability is therefore \(P(Y\geq5\mid x=8,n=20)\). The beta-binomial probabilities for \(y=0,\ldots,10\), rounded to three significant figures, are approximately 0.0146, 0.0596, 0.128, 0.187, 0.207, 0.180, 0.123, 0.0660, 0.0264, 0.00712, and 0.000986. Summing the probabilities for \(y=5,\ldots,10\) gives \(0.180+0.123+0.0660+0.0264+0.00712+0.000986=0.404\), with the unrounded calculation giving approximately 0.403.

The arm has about a 0.403 predictive probability of meeting the stated final criterion under this model and planned sample size. This value is not the current posterior probability that \(\theta>0.30\); it is the probability that the final posterior analysis will pass its 0.90 threshold.

Using a Prespecified Dropping Threshold

A predictive probability becomes an arm-dropping rule only after the trial specifies a decision threshold. For example, let \(\eta\) be the minimum predictive probability required to continue an arm. A rule might drop the arm if its predictive probability of final success is below \(\eta\). The choice of \(\eta\) is design-specific; there is no universal value that is appropriate for every trial.

Decision rule: Given a prespecified final success criterion and dropping threshold \(\eta\), drop an arm at the interim analysis if \(P(\text{final success}\mid\text{current data})<\eta\). Otherwise, retain it under this rule. The trial protocol should specify how equality at the threshold is handled.

Worked Example: Applying a Dropping Threshold

Use the predictive probability from the preceding hypothetical trial, approximately 0.403. Suppose the protocol prespecifies \(\eta=0.45\) and says to drop an arm when its predictive probability is below that value.

The comparison is \(0.403<0.45\). The arm therefore meets the rule for dropping. This conclusion follows from the prespecified cutoff, not from a claim that the arm is ineffective or that its current posterior probability of benefit is below 0.45. A different prespecified cutoff could produce a different decision from the same interim data.

Worked Example: A Low Predictive Probability Under Weak Interim Evidence

In a second hypothetical setting, suppose the interim posterior is \(\operatorname{Beta}(2,8)\), there are 10 patients remaining, and final success again requires posterior probability at least 0.90 that \(\theta>0.30\). For the final posterior \(\operatorname{Beta}(2+y,18-y)\), the criterion is first met at \(y=7\): the posterior probabilities for \(y=6\) and \(y=7\) are approximately 0.818 and 0.916. Thus only \(Y\geq7\) leads to final success.

The predictive probabilities can be obtained from the beta-binomial formula with \(A=2\), \(B=8\), and \(m=10\). In the order \(y=0,\ldots,10\), the probabilities are approximately 0.211, 0.248, 0.209, 0.149, 0.0930, 0.0514, 0.0250, 0.0104, 0.00351, 0.000867, and 0.000119. Therefore, \(P(Y\geq7\mid\text{interim data})=0.0104+0.00351+0.000867+0.000119=0.0149\).

With a prespecified dropping threshold of \(\eta=0.10\), the comparison \(0.0149<0.10\) calls for dropping the arm. Even though future responses could in principle satisfy the final criterion, the model assigns that event a low predictive probability.

Diagnostics, Comparisons, and Limitations

A predictive-probability calculation is conditional on its model and design assumptions. Before acting on the result, check that the analysis matches the planned decision:

Outcome model: Confirm that the binary response definition and assessment time are the same for observed and future patients, and that a common response probability is a defensible working assumption.
Remaining sample size: Use the number of future patients actually planned for the arm. A different \(m\) changes both the predictive distribution and the final posterior criterion.
Success criterion: Verify the benchmark \(p_0\), posterior threshold \(\gamma\), and any comparison defining success. Small changes can change which future outcomes count as success.
Prior and data: Check that the prior is appropriate for the endpoint and that the interim response count and denominator are correct. A prior with substantial influence can materially affect the predictive probability when data are sparse.
Operational consistency: If enrollment, follow-up, or the allocation plan changes, the assumed future data distribution may no longer match the design used for the calculation.

A posterior futility rule, as in the earlier material on Posterior Futility Rules for Arm Removal, assesses current posterior evidence against a futility condition. A predictive probability rule instead averages over possible future data and asks whether the completed analysis is likely to meet a chosen final success condition. A posterior superiority rule asks whether current evidence supports superiority under its specified comparison. These rules answer different questions; their numerical thresholds are not interchangeable.

The predictive calculation also has limitations. It does not by itself account for every possible operational change, such as a variable number of remaining patients, changing outcome ascertainment, or dependence between patients. Nor does a low predictive probability establish that the treatment has no effect. It says that, under the specified model, prior, remaining sample size, and final criterion, passing that criterion is unlikely. Predictive probabilities should be prespecified and assessed as part of the design rather than selected after inspecting interim results.

A One-Patient Check on the Logic

Worked Example: A Discrete Predictive Probability

Suppose the interim posterior is \(\operatorname{Beta}(2,2)\) and one patient remains. Let final success mean posterior probability at least 0.65 that \(\theta>0.50\). If the patient responds, the final posterior is \(\operatorname{Beta}(3,2)\); its probability above 0.50 is \(11/16=0.6875\), so the criterion passes. If the patient does not respond, the final posterior is \(\operatorname{Beta}(2,3)\); its probability above 0.50 is \(5/16=0.3125\), so the criterion fails.

The predictive response probability for the one remaining patient is the posterior mean \(2/(2+2)=0.5\). Since success occurs exactly when that patient responds, the predictive probability of final success is \(0.5\times1+0.5\times0=0.50\). With a prespecified dropping threshold of 0.40, this arm is retained under the rule because \(0.50\) is not below \(0.40\). This example also shows why predictive probabilities can change in jumps: with very few remaining patients, only a small number of future outcomes are possible.

Key takeaway: First define what will count as final success. Then calculate the posterior predictive distribution of the remaining outcomes, identify which future outcomes would pass that criterion, and sum their predictive probabilities. Compare the result with the prespecified dropping threshold without interpreting it as a direct probability that the treatment is effective.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In one sentence, distinguish the current posterior probability \(P(\theta>p_0\mid\text{current data})\) from the predictive probability of final success.
With current posterior \(\operatorname{Beta}(A,B)\) and \(m\) planned future patients, what distribution does the future response count \(Y\) have after integrating over \(\theta\)?
An arm’s predictive probability of final success is 0.12 and the prespecified dropping threshold is 0.15. What does the rule decide, if dropping occurs when the predictive probability is below the threshold?
Why can changing the planned number of remaining patients change the predictive probability even when the interim data do not change?

Solutions. (1) The current posterior probability concerns whether the response rate exceeds \(p_0\) given observed data; the predictive probability concerns whether future data, combined with those observed, will satisfy the final analysis criterion. (2) \(Y\) has a beta-binomial distribution with probabilities \(\binom{m}{y}B(A+y,B+m-y)/B(A,B)\) for \(y=0,\ldots,m\). (3) Since \(0.12<0.15\), the arm is dropped under the stated rule. (4) Changing the number of future patients changes the possible future response counts and their predictive probabilities, and it can also change the final posterior for each possible count. Both changes can alter the probability of meeting the final criterion.