Deriving Predictive Success by Averaging Over Future Data

Why Average Over Future Data?

At an interim analysis, the trial has produced data but has not yet reached its final decision point. The final decision will depend on outcomes that have not yet been observed. Predictive success answers a forward-looking question: given the data available now, what is the probability that the future data will lead to the prespecified final success criterion being met?

That probability must account for two sources of uncertainty. The unknown trial parameters are uncertain given the interim data, and future outcomes are uncertain even if the parameters were known. The Bayesian calculation averages the probability of meeting the final criterion over both sources, using the current posterior distribution for the unknown parameters.

Definition: Let $D$ denote the observed interim data, $\theta$ the unknown model parameters, and $\tilde Y$ the future trial data. Let $S(D,\tilde Y)=1$ when the final decision rule is met after combining the interim and future data, and $S(D,\tilde Y)=0$ otherwise. The predictive probability of success is $$ \operatorname{PPOS}(D) = P\{S(D,\tilde Y)=1\mid D\}. $$ It is the current posterior-predictive probability of eventual success under the stated model, prior, future-data plan, and final decision rule.

The final rule may be a posterior go criterion, such as requiring the final posterior probability of an effect exceeding a clinical margin to be at least a specified threshold. In that case, success is determined by applying the rule to the posterior after the future data have been observed. Predictive success is not simply the current posterior probability that the treatment effect is clinically meaningful.

The Averaging Identity

Suppose the model specifies a posterior distribution $p(\theta\mid D)$ and a predictive distribution for the future data conditional on the parameter, $p(\tilde y\mid\theta,D)$. The latter describes how future outcomes would be generated under the planned trial, conditional on the parameter and information already observed. The posterior-predictive distribution of future data is obtained by averaging that conditional distribution over the current posterior:

p(\tilde y\mid D) = \int p(\tilde y\mid\theta,D)\,p(\theta\mid D)\,d\theta.

For discrete future outcomes, the integral is replaced by a sum. Since success is a function of the future data and the observed interim data, predictive success is the probability of the set of future outcomes that meet the rule:

\operatorname{PPOS}(D) = \int S(D,\tilde y)\,p(\tilde y\mid D)\,d\tilde y.

Substituting the posterior-predictive distribution gives the form that makes the two averages explicit:

\operatorname{PPOS}(D) = \int \left[ \int S(D,\tilde y)\,p(\tilde y\mid\theta,D)\,d\tilde y \right] p(\theta\mid D)\,d\theta.

The inner integral is the probability of meeting the final rule if the parameter were fixed at $\theta$, with future outcomes still random. The outer integral averages that probability across plausible parameter values under the current posterior. For discrete $\theta$ and $\tilde Y$, the same identity is

\operatorname{PPOS}(D) = \sum_{\theta}p(\theta\mid D) \sum_{\tilde y}S(D,\tilde y)\,p(\tilde y\mid\theta,D).

This equality follows from conditioning on $\theta$. For each possible future outcome, its probability given $D$ is the posterior-weighted average of its probability given $\theta$ and $D$. Summing those probabilities over outcomes for which $S=1$ gives the predictive probability of success. The identity also shows why replacing $\theta$ with one estimate is not the same calculation: a plug-in value omits the outer posterior average.

Worked Calculations

Worked Example: A Binary Future Outcome

Hypothetical example. At an interim analysis, let $B$ mean that the treatment effect is in the beneficial state and $N$ mean it is not. Suppose the current posterior probabilities are $P(B\mid D)=0.60$ and $P(N\mid D)=0.40$. One future outcome $Y$ will be observed. The model gives $P(Y=1\mid B,D)=0.80$ and $P(Y=1\mid N,D)=0.20$. The final rule is to declare success if $P(B\mid D,Y)\geq 0.75$.

First calculate the posterior probability of $B$ after each possible future outcome. For $Y=1$, Bayes' rule gives

P(B\mid D,Y=1) = \frac{0.80(0.60)} {0.80(0.60)+0.20(0.40)} = \frac{0.48}{0.56} = 0.8571.

This exceeds 0.75, so $Y=1$ meets the final rule. For $Y=0$, the likelihoods are $P(Y=0\mid B,D)=0.20$ and $P(Y=0\mid N,D)=0.80$, giving

P(B\mid D,Y=0) = \frac{0.20(0.60)} {0.20(0.60)+0.80(0.40)} = \frac{0.12}{0.44} = 0.2727.

Thus, $Y=0$ does not meet the rule. The posterior-predictive probability of $Y=1$ is

P(Y=1\mid D) = 0.80(0.60)+0.20(0.40) = 0.56.

Only $Y=1$ leads to success, so $\operatorname{PPOS}(D)=0.56$. Under the stated model and rule, the probability of ultimately meeting the final criterion is 56%. This is not the current probability that the treatment is beneficial, which is 60%; the two probabilities answer different questions.

Worked Example: Averaging Across Three Parameter States

Hypothetical example. Suppose the current posterior assigns probabilities 0.20, 0.50, and 0.30 to parameter states $\theta_1,\theta_2,\theta_3$, respectively. A future binary outcome $Y$ has probabilities $P(Y=1\mid\theta_1,D)=0.10$, $P(Y=1\mid\theta_2,D)=0.50$, and $P(Y=1\mid\theta_3,D)=0.90$. State $\theta_3$ alone represents an effect exceeding a clinically meaningful threshold. The final rule is to require posterior probability at least 0.45 that the effect exceeds that threshold.

If $Y=1$, the predictive probability of that outcome is

P(Y=1\mid D) = 0.20(0.10)+0.50(0.50)+0.30(0.90) = 0.02+0.25+0.27 = 0.54.

The posterior probability of the clinically meaningful state after $Y=1$ is $0.30(0.90)/0.54=0.50$. That meets the 0.45 threshold. If $Y=0$, its predictive probability is $1-0.54=0.46$, and the posterior probability of the clinically meaningful state is

P(\theta_3\mid D,Y=0) = \frac{0.30(0.10)} {0.20(0.90)+0.50(0.50)+0.30(0.10)} = \frac{0.03}{0.18+0.25+0.03} = \frac{0.03}{0.46} = 0.0652.

That does not meet the final rule. Since only $Y=1$ leads to success, predictive success is $0.54(1)+0.46(0)=0.54$, or 54%. The calculation averages the likelihood of each future outcome over all three posterior states before deciding how much predictive probability belongs to the success set.

Worked Example: Two Future Responses and a Posterior Go Rule

Hypothetical example. Suppose the current posterior probability of a beneficial state $B$ is 0.40, with probability 0.60 for state $N$. Two future participants will each have a binary response. Conditional on the state, their outcomes are independent, with response probabilities 0.80 in state $B$ and 0.30 in state $N$. The final rule is to declare success if the final posterior probability of $B$ is at least 0.70. Let $K$ be the number of responses among the two participants.

For $K=2$, the likelihoods are $0.80^2=0.64$ under $B$ and $0.30^2=0.09$ under $N$. The predictive probability of $K=2$ and the corresponding final posterior probability are

P(K=2\mid D)=0.40(0.64)+0.60(0.09)=0.31, \qquad P(B\mid D,K=2)=\frac{0.40(0.64)}{0.31}=0.8258.

Two responses meet the final threshold. For $K=1$, the likelihoods are $2(0.80)(0.20)=0.32$ under $B$ and $2(0.30)(0.70)=0.42$ under $N$. Therefore,

P(K=1\mid D)=0.40(0.32)+0.60(0.42)=0.38, \qquad P(B\mid D,K=1)=\frac{0.40(0.32)}{0.38}=0.3368.

One response does not meet the rule. With no responses, the likelihoods are $0.20^2=0.04$ under $B$ and $0.70^2=0.49$ under $N$, giving

P(K=0\mid D)=0.40(0.04)+0.60(0.49)=0.31, \qquad P(B\mid D,K=0)=\frac{0.40(0.04)}{0.31}=0.0516.

The predictive probabilities for $K=0,1,2$ sum to $0.31+0.38+0.31=1.00$, as required. Only $K=2$ meets the final criterion; hence predictive success is $0.31$, or 31%.

Assumptions and Practical Checks

The averaging identity is a probability identity, but its usefulness depends on specifying the model and decision rule correctly. Before calculating, check that:

The posterior matches the data used. $p(\theta\mid D)$ must reflect the interim data and the prior assumptions used for the trial analysis.
The future-data model matches the plan. The conditional distribution $p(\tilde y\mid\theta,D)$ must represent the remaining participants, follow-up, allocation, and any planned changes to the design.
The final rule is explicit. For every possible future outcome, it must be possible to determine whether the final criterion is met. If the rule depends on a final posterior probability, that probability must be calculated using $D$ together with that outcome.
Dependence is handled appropriately. Conditional independence may be used only when justified by the model. If future outcomes are dependent, their joint predictive distribution must preserve that dependence.

A useful check for finite outcome spaces is that the predictive probabilities of all possible future outcomes sum to one, and that predictive success lies between zero and one. Another is to list the success-inducing outcomes explicitly and verify that each one does, and each excluded one does not, meet the final rule. If the calculation produces an implausible result, common causes include omitting interim data from the final update, averaging over the wrong parameter distribution, or using a success threshold different from the prespecified one.

How Predictive Success Differs From Alternatives

Quantity or approach	What it answers	Key distinction
Current posterior probability	How plausible is a parameter condition given the data observed so far?	It concerns the current unknown parameter, not whether future data will meet the final rule.
Predictive success	How probable is it that future data will lead to the final rule being met?	It averages over posterior uncertainty and future outcome uncertainty.
Plug-in calculation	What would the success probability be at one selected parameter value?	It conditions on a single value and generally does not average over the current posterior.
Prior predictive assurance	Before trial data are observed, how probable is success under the prior and design?	It averages over prior uncertainty, whereas interim predictive success conditions on $D$ and uses $p(\theta\mid D)$.

These quantities can differ substantially. A trial can have strong current posterior evidence yet modest predictive success if little information remains and the final rule is demanding. Conversely, current evidence may be inconclusive while a substantial amount of future information makes eventual success plausible. Predictive success is therefore a decision-support summary, not a replacement for the current posterior or for the prespecified final decision rule.

Limitations and Common Pitfall

Predictive success is conditional on the model, prior, planned future data collection, and final criterion. It does not guarantee that a trial will succeed, and it does not provide a model-free probability. If the outcome model poorly represents the trial, or the planned future enrollment differs from what occurs, the calculated probability may be misleading. A single predictive-success value can also hide sensitivity to prior assumptions or to the precise final threshold.

Common mistake: “The current posterior probability of benefit is 0.60, so predictive success is 60%.” This equates a probability about the unknown parameter with a probability about future data meeting a decision rule. The correction is to calculate the posterior-predictive distribution of future outcomes, apply the final rule to each possible outcome, and add or integrate the probabilities of the outcomes that satisfy it.

For discrete outcomes, exact enumeration is often transparent. For more complex outcome spaces, the same averaging identity still defines the target, although numerical integration may be needed. Regardless of the computational method, the interpretation remains conditional: it is the probability of meeting the stated final criterion given the interim data and the assumptions used to predict the rest of the trial.

Check Your Understanding

Answer each question using the distinction between parameter uncertainty, future-outcome uncertainty, and the final decision rule.

If the current posterior probability of a beneficial effect is 0.80, must predictive success also be 0.80? Explain.
In a finite outcome space, predictive success is 0.42 for one future outcome that meets the rule and 0.58 for outcomes that do not. What is predictive success?
Why does a calculation at a single estimated parameter value generally differ from posterior predictive success?
What must be done if future outcomes are dependent rather than conditionally independent under the model?

Solutions. (1) No. The 0.80 is a current posterior probability about the parameter; predictive success is the posterior-predictive probability that future data lead to the final rule being met. (2) Predictive success is 0.42, the predictive probability of the success-inducing outcome set. (3) The single-value calculation omits averaging over $p(\theta\mid D)$, so it does not account for current parameter uncertainty. (4) Use the joint conditional predictive distribution that represents the dependence; multiplying marginal probabilities as if outcomes were independent would be inappropriate.