From a Probability of Success to a Probability for Future Data
A probability of trial success is only meaningful after success has been specified. For a development decision, the rule might concern a minimum difference in response counts, a clinically relevant treatment effect, or a formal statistical criterion. The calculation must then account for both what is uncertain now and the random outcomes a future trial may produce. This tutorial works through that calculation for binary and continuous endpoints, including cases where the future sample size is uncertain.
Let \(D\) denote the evidence currently available, \(\theta\) the unknown model parameters, \(N\) the future design, and \(Y_{\mathrm{new}}\) the future data. Write \(S(Y_{\mathrm{new}},N)=1\) when the prespecified success rule is met, and \(0\) otherwise. The posterior predictive probability of success is the probability of that event, conditional on current evidence.
The posterior predictive calculation is not the probability that a treatment effect is clinically worthwhile, nor is it the probability that a hypothesis is true. It is a probability for a future observable outcome under a specified model, current evidence, design, and success rule. If a future design depends on interim data or parameter estimates, its dependence must be represented in the joint predictive calculation rather than treated as independent by convenience.
Worked Example: Binary Endpoints With Uncertain Response Rates
Worked Example: Binary Endpoints With Uncertain Response Rates
Hypothetical data. Suppose current evidence gives independent posteriors \(p_T\mid D\sim\operatorname{Beta}(2,2)\) and \(p_C\mid D\sim\operatorname{Beta}(2,2)\) for treatment and control response probabilities. A future trial will enroll four patients per arm. Define success as observing at least two more responses in the treatment arm than in the control arm. This deliberately simple count-based rule is used to illustrate the predictive calculation; it is not a recommended regulatory success criterion.
If \(K\) is the number of responses among \(n\) future patients and \(p\) has posterior distribution \(\operatorname{Beta}(\alpha,\beta)\), then integrating the binomial probability over the beta posterior gives
This is the beta-binomial predictive distribution. With \(n=4\) and \(\alpha=\beta=2\), the predictive probabilities for \(K=0,1,2,3,4\) are respectively \(120/840,192/840,216/840,192/840,120/840\), or approximately \(0.1429, 0.2286, 0.2571, 0.2286, 0.1429\). Each arm has this distribution under the stated independent posterior model.
Success requires \(K_T-K_C\geq 2\). The successful count pairs are \((2,0),(3,0),(3,1),(4,0),(4,1),(4,2)\). Independence of the two arms’ posterior predictive counts gives
Thus, under these hypothetical posteriors, trial design, and success rule, the probability of satisfying the rule is about 21.1%. This low probability does not mean that the posterior probability of a beneficial treatment effect is 21.1%; the calculated event concerns future response counts, and the two probabilities are different quantities.
Continuous Endpoints: Integrating Treatment-Effect and Variance Uncertainty
For a continuous endpoint, a useful starting point is the predictive distribution of the future estimated treatment difference. Suppose the future difference in sample means, \(\widehat{\delta}_{\mathrm{new}}\), is normally distributed conditional on the treatment effect \(\delta\), residual standard deviation \(\sigma\), and equal per-arm sample size \(n\):
This conditional model assumes independent treatment and control observations, a common residual variance, normally distributed outcomes (or a setting where the normal approximation for the mean difference is adequate), and no additional sources of variation omitted from the model. The success probability for any draw of \(\delta\) and \(\sigma\) can be calculated from this distribution, then averaged over their joint posterior.
Worked Example: Continuous Endpoint With Joint Posterior Uncertainty
Hypothetical model summary. Suppose a joint posterior is approximated by three equally weighted types of parameter values: \((\delta,\sigma)=(0.4,0.8)\) with probability \(0.25\), \((0.7,1.0)\) with probability \(0.50\), and \((0.9,1.2)\) with probability \(0.25\). The treatment effect and residual standard deviation are measured in endpoint units. A future trial has \(n=25\) patients per arm. Define success as a future estimated treatment difference of at least \(0.5\) units.
Conditional on each posterior value, the standard error of the future difference is \(SE=\sigma\sqrt{2/25}\), and \(P(S=1\mid\delta,\sigma,n)=1-\Phi((0.5-\delta)/SE)\), where \(\Phi\) is the standard normal cumulative distribution function. For the first posterior value, \(SE=0.8\sqrt{2/25}=0.2263\), so the success probability is \(1-\Phi((0.5-0.4)/0.2263)=1-\Phi(0.4419)\approx0.3294\).
For the second value, \(SE=1.0\sqrt{2/25}=0.2828\), giving \(1-\Phi((0.5-0.7)/0.2828)=1-\Phi(-0.7071)\approx0.7602\). For the third value, \(SE=1.2\sqrt{2/25}=0.3394\), giving \(1-\Phi((0.5-0.9)/0.3394)=1-\Phi(-1.1785)\approx0.8807\).
The posterior predictive probability is therefore approximately 68.3%. The integration includes uncertainty in both the treatment effect and residual variability: larger \(\sigma\) increases future sampling variability, while the posterior effect values shift the center of the future distribution. The example’s success rule concerns the future estimated difference alone. A rule requiring a confidence interval or test to meet a criterion would need to model that statistic and its future variance estimate as well.
Uncertain Future Sample Size
When the future sample size is not fixed, calculate success probability at each possible design and average over the stated distribution of designs. The probability assigned to each sample size must be justified by the planning scenario; it is not automatically uniform, and uncertainty in sample size should not be confused with variability in the number of evaluable outcomes.
Worked Example: Averaging Over Two Future Sample Sizes
Hypothetical model summary. Let the posterior treatment effect be \(0.4\) endpoint units with probability \(0.5\) and \(0.8\) units with probability \(0.5\). Assume a known residual standard deviation of \(1\) unit, equal allocation, and that the future per-arm sample size is \(10\) or \(100\), each with probability \(0.5\). Success means the future estimated difference is at least \(0.5\) units. The conditional standard error is \(\sqrt{2/n}\).
For \(n=10\), \(SE=\sqrt{2/10}=0.4472\). At \(\delta=0.4\), the success probability is \(1-\Phi((0.5-0.4)/0.4472)=1-\Phi(0.2236)\approx0.4115\). At \(\delta=0.8\), it is \(1-\Phi((0.5-0.8)/0.4472)=1-\Phi(-0.6708)\approx0.7487\). Averaging over the two posterior effect values gives \(0.5(0.4115)+0.5(0.7487)=0.5801\).
For \(n=100\), \(SE=\sqrt{2/100}=0.1414\). The two conditional success probabilities are \(1-\Phi(0.7071)\approx0.2398\) and \(1-\Phi(-2.1213)\approx0.9831\), respectively. Their posterior average is \(0.5(0.2398)+0.5(0.9831)=0.6115\). Finally, average over the two equally likely designs:
The overall predictive probability is about 59.6%. Substituting the mean sample size, \(55\) per arm, would instead give a standard error of \(\sqrt{2/55}=0.1907\) and a posterior-averaged probability of approximately \(0.621\). These calculations differ because success probability is a nonlinear function of sample size: averaging probabilities over designs is generally not the same as calculating a probability at the average design.
Calculation Choices, Diagnostics, and Limitations
The examples use exact integration or a small discrete posterior mixture. In a larger model, a direct simulation follows the same logic: draw parameters from their posterior, draw a future design if it is uncertain, simulate future data under that draw, apply the success rule, and estimate the predictive probability by the proportion of simulated trials that succeed. For \(M\) simulated trials with \(m\) successes, the estimate is \(m/M\); its approximate Monte Carlo standard error is \(\sqrt{\hat{q}(1-\hat{q})/M}\), where \(\hat{q}=m/M\). This measures simulation error, not uncertainty or misspecification in the model.
A plug-in calculation fixes parameters at point estimates and computes future sampling probability conditional on those values. It can be useful as a quick scenario, but it omits parameter uncertainty and can give a materially different result, as the binary example illustrates. A posterior predictive calculation integrates parameter uncertainty under the specified posterior. A frequentist conditional-power calculation conditions on a chosen effect estimate and a specified sampling model; it answers a different question unless its conditioning assumptions and the Bayesian posterior calculation happen to align.
Check the calculation by examining its components before relying on the final probability:
- Verify the event. State the endpoint, analysis population, estimand, and success threshold or decision rule. A probability without a clear event can be misinterpreted.
- Check the conditional sampling model. Confirm that allocation, independence, outcome distribution, residual variance structure, and any approximation used are appropriate to the planned trial.
- Inspect posterior coverage and dependence. Parameter draws should represent the joint posterior, including correlation between treatment effects and nuisance parameters when relevant. Independently sampling correlated quantities can distort predictions.
- Check design uncertainty. Ensure the distribution of future sample sizes reflects the actual planning scenarios. If sample size depends on interim information, represent that dependence explicitly.
- Assess computational stability. For simulation, increase \(M\), check Monte Carlo error, and compare with an exact calculation or a second implementation when feasible.
The result is conditional on the model, posterior, and success rule. It does not repair biased or incomplete evidence, account automatically for model uncertainty, or guarantee that the assumed future design will be implemented. If outcome distributions are skewed, dropout is substantial, treatment effects vary across subgroups, or the final success rule uses a complex analysis, simulate that full analysis rather than substituting a simplified endpoint rule.
A Short Derivation and a Common Pitfall
The averaging step follows from the law of total probability. Conditional on parameters and design, the future data have a sampling distribution, so the chance of success is \(P(S=1\mid\theta,n,D)\). Averaging first over the design distribution and then over the posterior yields the overall chance of success:
The factorization in the second line assumes that, given \(D\), the future design \(N\) and parameters \(\theta\) are independent. If they are not independent, replace \(P(n\mid D)\pi(\theta\mid D)\) with their joint distribution \(P(n,\theta\mid D)\).
A further interpretive safeguard is to report the assumptions and event alongside the number. For example: “Under the stated posterior model and a future trial with 25 patients per arm, the estimated probability that the future mean difference is at least 0.5 endpoint units is 68.3%.” That statement is more informative than reporting “probability of success = 68.3%” without context.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In the binary example, why does the predictive distribution of future response counts differ from a binomial distribution evaluated at the posterior mean response probability?
- For a continuous endpoint, what changes in the calculation if success requires the future estimated effect to exceed a threshold and its lower confidence bound to exceed zero?
- A future sample size is 20 or 80 per arm with probabilities 0.7 and 0.3. How should the overall predictive success probability be formed?
- What does a Monte Carlo standard error quantify, and what important uncertainty does it not quantify?
Solutions. (1) The beta-binomial distribution averages binomial probabilities over uncertainty in the response probability. A binomial distribution at the posterior mean conditions on one fixed probability and omits that uncertainty. (2) For each posterior draw, simulate or derive the joint predictive distribution of the estimated effect and its standard error, then apply both parts of the success rule to the simulated or analytically evaluated future result. (3) Calculate the posterior predictive success probability separately at each sample size, then combine them as \(0.7P(S=1\mid D,N=20)+0.3P(S=1\mid D,N=80)\). (4) It quantifies numerical error from using a finite number of Monte Carlo draws. It does not quantify uncertainty from limited current evidence, model misspecification, or an inappropriate success rule.