Why the Decision Threshold Matters
A trial may provide evidence that treatment improves an outcome without providing strong evidence that the improvement is large enough to matter to patients or clinicians. Those are different claims, so they should not be represented by the same final decision rule. A rule requiring an effect greater than zero asks whether the data support any benefit. A rule requiring an effect greater than a positive clinical-importance margin asks whether they support a benefit of a specified minimum size.
Predictive success is the probability, given the data available now, that future trial data will make the final posterior decision rule pass. It averages over uncertainty about the effect and over the possible future observations. The decision rule must therefore be stated precisely before predictive success is calculated.
For a transparent binary-outcome example, suppose a treatment is being assessed against a fixed, established benchmark response probability \(p_0=0.30\). Let \(p\) be the treatment response probability and let the effect be the absolute difference \(\theta=p-p_0\). This setup treats the benchmark as known; it is not a substitute for modeling a concurrently randomized control group when its response probability is uncertain.
In the benchmark example, the positive-effect rule concerns \(p>0.30\). If the minimum clinically important absolute improvement is 0.10, the meaningful-effect rule concerns \(p>0.40\). Both probabilities refer to the treatment response probability, but they answer different clinical questions. A cutoff such as \(\gamma=0.95\) specifies how much posterior probability the final rule requires; it is not the probability of future success.
Model and Predictive Probability
Assume that each observed response is a Bernoulli outcome with common response probability \(p\), and that responses are conditionally independent given \(p\). Use the prior \(p\sim\operatorname{Beta}(2,2)\). If \(x\) responses are observed among \(n\) participants, conjugacy gives \(p\mid x\sim\operatorname{Beta}(2+x,2+n-x)\). Let \(Y\) be the number of responses among \(m\) future participants, with the same response probability and independent outcomes conditional on \(p\).
To predict \(Y\), uncertainty about \(p\) must be integrated out using its current posterior. With \(a=2+x\) and \(b=2+n-x\), the predictive probability of \(y\) future responses is
This is a beta-binomial distribution. For each possible \(y\), the final posterior is \(\operatorname{Beta}(a+y,b+m-y)\). If the final go rule requires \(P(p>p_0+\delta\mid x,Y=y)\geq\gamma\), predictive success is the sum of the predictive probabilities of those future outcomes that pass that rule:
The indicator is one when the final posterior probability meets the cutoff and zero otherwise. This calculation distinguishes the probability of passing the final rule from the current posterior probability of benefit. Current evidence is summarized by the posterior now; predictive success also accounts for the possible future data and how each possible final data set would change that posterior.
Worked Examples
Worked Example: Current Evidence for Any Benefit
Suppose, hypothetically, that two participants have been observed and \(x\) responded. Under the \(\operatorname{Beta}(2,2)\) prior, calculate the current posterior probability of exceeding the benchmark \(p_0=0.30\) for \(x=0,1,2\).
For \(x=0\), the posterior is \(\operatorname{Beta}(2,4)\). Its upper-tail probability is \(P(p>0.30\mid x=0)=(1-0.30)^4[1+4(0.30)]=0.52822\).
For \(x=1\), the posterior is \(\operatorname{Beta}(3,3)\). The beta CDF at \(t\) is \(10t^3-15t^4+6t^5\), so \(P(p>0.30\mid x=1)=1-[10(0.30)^3-15(0.30)^4+6(0.30)^5]=0.83692\).
For \(x=2\), the posterior is \(\operatorname{Beta}(4,2)\). Its CDF at \(t\) is \(5t^4-4t^5\), giving \(P(p>0.30\mid x=2)=1-[5(0.30)^4-4(0.30)^5]=0.96922\).
Thus, the current probabilities of any benefit are approximately 0.5282, 0.8369, and 0.9692. These are posterior probabilities based on two observed outcomes, not predictive success probabilities. To calculate predictive success, we still need a final posterior cutoff, a clinical threshold, and a model for the remaining data.
Worked Example: Comparing Final Rules After Two More Participants
Continue the hypothetical example with \(n=2\), \(x=2\), and \(m=2\). Suppose the final go criterion is a posterior probability of at least \(\gamma=0.95\). Compare the positive-effect rule \(p>0.30\) with a clinically meaningful-effect rule \(p>0.40\).
The current posterior is \(\operatorname{Beta}(4,2)\). The predictive probabilities for \(Y=0,1,2\) follow from the beta-binomial formula: \(P(Y=0\mid x)=2(3)/[6(7)]=1/7\), \(P(Y=1\mid x)=2(4)(2)/[6(7)]=8/21\), and \(P(Y=2\mid x)=4(5)/[6(7)]=10/21\). They sum to \(1/7+8/21+10/21=1\).
The final total number of responses is \(k=x+y\), ranging from 2 to 4. After four participants, the posterior for \(p\) is \(\operatorname{Beta}(2+k,6-k)\). For integer beta parameters, the identity \(P(p>t)=P\{\operatorname{Binomial}(a+b-1,t)\leq a-1\}\) for \(p\sim\operatorname{Beta}(a,b)\) gives the following final tail probabilities.
| Final responses \(k\) | Posterior \(P(p>0.30)\) | Posterior \(P(p>0.40)\) |
|---|---|---|
| 0 | 0.3294 | 0.1586 |
| 1 | 0.6471 | 0.4199 |
| 2 | 0.8740 | 0.7102 |
| 3 | 0.9712 | 0.9037 |
| 4 | 0.9962 | 0.9812 |
For example, when \(k=3\), the posterior is \(\operatorname{Beta}(5,3)\). At \(t=0.30\), the tail is the probability of at most four successes in seven binomial trials, \(0.9712\); at \(t=0.40\), it is \(0.9037\). Consequently, at cutoff 0.95, the positive-effect rule passes for final \(k=3\) or \(4\), while the meaningful-effect rule passes only for \(k=4\).
Starting from \(x=2\), the positive-effect rule therefore has predictive success \(P(Y\geq1\mid x)=8/21+10/21=18/21=0.8571\). The meaningful-effect rule has predictive success \(P(Y=2\mid x)=10/21=0.4762\). The same current data and future-data model yield different predictive success because the final clinical claim is more demanding.
Worked Example: Fewer Responses at Interim
Now suppose, hypothetically, that \(n=2\), \(x=1\), and two participants remain, with the same prior, benchmark, margin, and 0.95 final cutoff. The current posterior is \(\operatorname{Beta}(3,3)\). Its beta-binomial predictive probabilities are \(P(Y=0\mid x)=3(4)/[6(7)]=2/7\), \(P(Y=1\mid x)=2(3)(3)/[6(7)]=3/7\), and \(P(Y=2\mid x)=3(4)/[6(7)]=2/7\).
The final total response count is \(k=1,2,\) or \(3\). From the table, only \(k=3\) meets the positive-effect rule, so its predictive success is \(P(Y=2\mid x)=2/7=0.2857\). No possible final count reaches the meaningful-effect rule’s required \(k=4\). Its predictive success is therefore zero. This zero is a consequence of the specified remaining sample size and final rule, not proof that a meaningful effect is impossible in the population.
Justification, Diagnostics, and Alternatives
The beta-binomial calculation follows directly by averaging a binomial likelihood over the current posterior. Conditional on \(p\), \(P(Y=y\mid p)=\binom{m}{y}p^y(1-p)^{m-y}\). Integrating this quantity against the \(\operatorname{Beta}(a,b)\) posterior gives
The integral is the beta-function ratio shown in the predictive formula. This derivation clarifies why simply inserting a single estimated value of \(p\) into a binomial distribution is not the same procedure: averaging over the posterior includes parameter uncertainty.
The assumptions should be checked against the trial design. The calculation assumes a fixed benchmark, a common response probability across observed and future participants, and conditionally independent Bernoulli responses. Changes in eligibility, follow-up, treatment delivery, or outcome ascertainment could undermine the common-probability assumption. If the comparison response is uncertain, the effect is a contrast between uncertain quantities and the joint posterior uncertainty must be included rather than treating the control rate as fixed. If responses are clustered or otherwise dependent, the independent Bernoulli model may also be inadequate.
Several checks help detect calculation or interpretation problems:
- Predictive probabilities over all possible future counts must be nonnegative and sum to one.
- The final posterior must use all observed and future data, not only the future responses.
- For the same \(\gamma\), a meaningful threshold above the benchmark cannot make the final rule easier to pass than the positive-effect rule. Its predictive success should therefore be no larger.
- Changing \(m\), the prior, \(\delta\), or \(\gamma\) can change predictive success. Report these choices so the result can be interpreted and checked.
A common mistake is to read a high current posterior probability of any benefit as a high probability of eventual clinical success. For instance, a current probability above 0.95 for \(p>0.30\) does not imply that the final probability for \(p>0.40\) will reach 0.95. The correction is to apply the actual final rule to each possible future data set, then sum the predictive probabilities of the data sets that pass.
Another alternative is to calculate success using a single assumed value of \(p\), sometimes called a plug-in calculation. That can be useful as a scenario calculation, but it does not average over the current posterior uncertainty and is not the posterior predictive success defined here. Likewise, the current posterior probability \(P(p>p_0+\delta\mid x)\) answers what is supported now, whereas predictive success answers whether the final rule is expected to be met after additional observations. Neither quantity is automatically a frequentist operating characteristic; that question evaluates a trial procedure across repeated data sets under specified true values.
The approach is limited by the chosen model, prior, fixed benchmark, clinical margin, and final cutoff. A clinically important margin should be justified on clinical grounds rather than selected to make predictive success look favorable. Predictive success is a planning or monitoring quantity conditional on current data and model assumptions, not a guarantee of final success and not a causal conclusion by itself.
Check Your Understanding
Answer each question using the definitions and assumptions above; concise solutions follow.
- If \(p_0=0.30\) and the clinical margin is \(\delta=0.08\), what treatment response probability threshold defines a clinically meaningful effect?
- With \(n=2\), \(x=2\), and the \(\operatorname{Beta}(2,2)\) prior, what is the current posterior distribution for \(p\)?
- In the worked example with \(x=2\), \(m=2\), and cutoff 0.95, why is meaningful-effect predictive success \(10/21\), not \(18/21\)?
- Name two assumptions that would need reconsideration if a randomized control group had an uncertain response probability.
Solutions. (1) The threshold is \(p>p_0+\delta=0.30+0.08=0.38\). (2) The posterior is \(\operatorname{Beta}(2+2,2+2-2)=\operatorname{Beta}(4,2)\). (3) Only final \(k=4\) reaches posterior probability at least 0.95 for \(p>0.40\), and from \(x=2\) that occurs only when \(Y=2\), whose predictive probability is \(10/21\). The value \(18/21\) includes \(Y=1\), which passes the positive-effect rule but not the meaningful-effect rule. (4) The benchmark is no longer known and fixed, so uncertainty in the control response must be modeled jointly with treatment response; also, independence and comparable response probabilities across participants and arms must be assessed under the actual design.