Simulating Future Trial Outcomes to Estimate Predictive Success

From a Current Posterior to Simulated Trial Completions

A trial in progress has data already observed and participants or follow-up still to come. Predictive success asks: given what is known now, how likely is the completed trial to meet its prespecified success rule? The answer must account for both uncertainty about the trial’s underlying parameters and randomness in the remaining outcomes. Simulation provides a direct way to do this when the future-data distribution or final rule is inconvenient to calculate exactly.

The procedure is not to draw a parameter once, generate many possible outcomes conditional on that one value, and treat the result as the overall predictive probability. Instead, each simulated completion begins with a draw from the current posterior distribution. That draw represents one plausible value of the unknown parameter given the interim data. Future outcomes are then generated conditional on it, and the final rule is applied to the completed dataset.

Definition: Let $D$ denote the interim data, $\theta$ the unknown parameter or vector of parameters, and $Y_{\mathrm{future}}$ the remaining trial data. If $S(D,Y_{\mathrm{future}})$ equals 1 when the completed dataset meets the prespecified success rule and 0 otherwise, the predictive probability of success is $$ q=P\{S(D,Y_{\mathrm{future}})=1\mid D\} =\int \sum_{y} S(D,y)\,p(y\mid\theta,D)\,p(\theta\mid D)\,d\theta. $$ For continuous future data, the sum is replaced by an integral. The posterior distribution $p(\theta\mid D)$ represents current parameter uncertainty, and $p(y\mid\theta,D)$ represents the future-data model conditional on the parameter and interim information.

A simulation estimates this average by repeatedly drawing $\theta$ from $p(\theta\mid D)$, drawing future data from the model conditional on that $\theta$, and recording whether the final rule is met. The final analysis used within each simulated completion must be the analysis specified for the real completed trial. If success is defined by a final posterior probability, for example, the posterior must be calculated from the simulated completed dataset using the specified prior and model; a simulated parameter value is not itself a final posterior probability.

The Simulation Procedure

Before simulating, specify the interim posterior, the remaining trial design, the future outcome model, and the final decision rule. The simulation is only as relevant as these ingredients. The remaining sample size, allocation, follow-up, and any design features that affect outcomes must match the trial plan. If several parameters are uncertain, draw them jointly from their current posterior so their posterior dependence is retained.

Draw interim parameters.
For replicate $b$, draw $\theta^{(b)}\sim p(\theta\mid D)$. For multiple parameters, use a joint posterior draw.

Generate remaining outcomes.
Draw $Y_{\mathrm{future}}^{(b)}\sim p(y\mid\theta^{(b)},D)$, using the planned number of participants and the appropriate outcome and follow-up model.

Construct the completed dataset.
Combine the observed interim data $D$ with the simulated future outcomes. Do not replace observed outcomes with new draws.

Apply the final rule.
Calculate the specified final statistic or posterior quantity from that completed dataset and set $I_b=1$ if the rule is met, and $I_b=0$ otherwise.

Average the indicators.
For $B$ replicates, estimate predictive success by $\widehat q_{\mathrm{MC}}=B^{-1}\sum_{b=1}^{B} I_b$, the fraction of simulated completions that meet the rule.

The method relies on the posterior being appropriate for the interim data and on the future-data model representing how the remaining trial will actually be conducted. It also assumes the success rule is specified before judging each simulated completion. Simulation does not repair a misspecified model, an incorrectly coded rule, or a design that differs from the one being evaluated.

Worked Examples

Worked Example: Binary Outcomes and a Prespecified Final Score

Consider a hypothetical trial with binary outcomes and 25 participants remaining. After the interim data, suppose the response probability has posterior distribution $\theta\mid D\sim\mathrm{Beta}(4,4)$. For each replicate, draw $\theta^{(b)}$ from this posterior, then generate $Y^{(b)}\sim\mathrm{Binomial}(25,\theta^{(b)})$, where $Y^{(b)}$ is the number of responses among the remaining participants.

Suppose the trial’s prespecified final rule uses the evidence score $R(y)=P(B\leq y+7)$, where $B\sim\mathrm{Binomial}(25,0.5)$, and declares success when $R(y)\geq0.90$. This is a deliberately specified score rule for the example; other trials may use a final posterior probability or another prespecified criterion. For the relevant counts,

R(7)=P(B\leq14)=0.7878,\qquad R(8)=P(B\leq15)=0.8852,\qquad R(9)=P(B\leq16)=0.9461.

The first count meeting the 0.90 criterion is therefore $y=9$, so a simulated completion succeeds exactly when $Y^{(b)}\geq9$. In particular, a replicate with $Y^{(b)}=8$ fails; it is not enough that its score is close to 0.90.

Suppose eight illustrative posterior-predictive replicates produce counts $8,13,16,10,18,7,6,15$. Five counts are at least 9: $13,16,10,18,$ and $15$. Thus $$ \widehat q_{\mathrm{MC}}=\frac{5}{8}=0.625. $$ This small illustrative run estimates predictive success as 62.5%. It is not a precise estimate; the important point here is that every replicate is classified using the stated final rule, including the strict threshold implied by the score.

In an actual implementation, each $Y^{(b)}$ would be generated conditional on its own posterior draw $\theta^{(b)}$. Drawing $\theta$ from $\mathrm{Beta}(4,4)$ and then drawing a binomial count incorporates uncertainty in the response probability as well as ordinary outcome variation.

Worked Example: Continuous Outcomes and a Final Mean Criterion

In a hypothetical single-arm study, eight participants have already contributed outcomes with observed sum $1.00$ outcome units. Twelve participants remain. Suppose the interim posterior for the population mean is $\mu\mid D\sim N(0.18,0.08^2)$, and the future outcomes are conditionally independent with $X_i\mid\mu\sim N(\mu,0.40^2)$. The final rule is that the mean across all 20 participants must be at least $0.10$ outcome units.

For one replicate, draw $\mu^{(b)}=0.22$ outcome units from the interim posterior and simulate the 12 remaining observations. Suppose their simulated sum is $1.24$ outcome units. The completed total is $1.00+1.24=2.24$, so the completed sample mean is

\bar{x}_{\mathrm{final}}^{(b)} =\frac{2.24\text{ outcome units}}{20} =0.112\text{ outcome units}.

Since $0.112\geq0.10$, this replicate meets the final rule and has $I_b=1$. If instead the simulated future sum were $0.90$, the completed mean would be $1.90/20=0.095$ outcome units, and the replicate would fail. The comparison uses the completed dataset, not the mean of only the 12 simulated outcomes.

This example assumes the remaining participants’ outcomes are independent normal observations with the stated common variance, conditional on $\mu$, and that no additional design feature changes the endpoint or analysis. If the final rule were a posterior probability threshold rather than a sample-mean threshold, each replicate would require the prespecified final posterior analysis instead.

Worked Example: Simulating Remaining Event Counts

Suppose a hypothetical event-driven trial has observed 2 events and fixed remaining exposure of 10 person-years. Let the current posterior for the event rate be $\lambda\mid D\sim\mathrm{Gamma}(3,20)$, using the shape-rate convention, so the posterior mean is $3/20=0.15$ events per person-year. Assume future events follow a Poisson process conditional on $\lambda$, giving $K\mid\lambda\sim\mathrm{Poisson}(10\lambda)$. The final rule is success if the total event count is at least 5.

Because 2 events have already occurred, a simulated completion succeeds when $2+K\geq5$, or equivalently $K\geq3$. For one replicate, draw $\lambda^{(b)}=0.12$ events per person-year. The conditional future event count then has mean $10(0.12)=1.20$ events. If the simulated count is $K^{(b)}=3$, the completed trial has 5 events and succeeds. If $K^{(b)}=2$, it has 4 events and fails.

This simulation assumes fixed future exposure and a constant event rate conditional on $\lambda$, with independent increments in the Poisson process. If exposure depends on recruitment, censoring, or participant follow-up, those features must also be represented when generating each future dataset; simply simulating a Poisson count for fixed exposure would then not reflect the planned trial.

Why the Averaging Works

The predictive success probability averages the conditional chance of success over the current posterior. This follows from conditioning on $\theta$: first consider the future-data distribution at a fixed parameter value, then average those probabilities using the posterior distribution of that parameter. The simulation performs the same two-stage operation by drawing a parameter and then drawing data.

\begin{aligned} E(I_b\mid D) &=E\!\left[E(I_b\mid\theta,D)\mid D\right]\\ &=\int P\{S(D,Y_{\mathrm{future}})=1\mid\theta,D\}\, p(\theta\mid D)\,d\theta\\ &=q. \end{aligned}

Thus each success indicator has expectation $q$, and the average of the indicators targets the predictive probability. With independent simulation replicates, the average is unbiased for $q$. This justification depends on both stages being sampled from the specified distributions: replacing the posterior draw by a fixed estimate changes the target to a conditional success probability at that single parameter value.

Diagnostics, Alternatives, and Limitations

Several checks can reveal whether the simulation is implementing the intended calculation:

Check the posterior draws. Confirm that draws come from the current posterior and that joint parameter dependence is retained. Implausible ranges or impossible parameter combinations indicate a problem.
Check the simulated outcomes. Confirm the remaining sample size, allocation, exposure, and outcome constraints. For binary data, for example, simulated counts must be integers from 0 through the number of remaining participants.
Check the final analysis. Reconstruct the completed dataset and apply the stated final rule to a few replicates by hand. A boundary value is especially useful for detecting whether the inequality is $>$ or $\geq$.
Check observed data handling. Interim outcomes remain fixed in every completion. Resimulating them would evaluate a different predictive question.

Analytic predictive calculations are often preferable when they are available and manageable. The beta-binomial and normal-normal results from earlier material can yield exact or directly evaluable predictive probabilities under their assumptions. Numerical integration is another alternative when the parameter space is low-dimensional and the conditional success probability can be evaluated reliably. Simulation is especially useful when a final rule involves several parameters, multiple endpoints, event processes, or a complicated analysis that is straightforward to rerun on a completed dataset.

Simulation has limitations. Its estimate depends on the posterior model and future-data assumptions, and a large number of replicates cannot correct a wrong model or a misapplied decision rule. It also introduces Monte Carlo variability: with a finite number of replicates, the estimated fraction will vary across runs. The estimate should therefore be reported as a simulation estimate, not as an exact probability. Detailed assessment of Monte Carlo error is a separate diagnostic task.

Common Mistakes

Using only a plug-in parameter. Simulating future data at the posterior mean ignores uncertainty in $\theta$. Draw from the posterior on each replicate, then generate outcomes conditional on that draw.
Checking the parameter rather than the completed trial. A replicate succeeds only if its simulated completed dataset meets the final rule. A favorable sampled value of $\theta$ is not by itself a successful trial.
Using the interim rule instead of the final rule. Predictive success concerns the specified completed-trial criterion. Recalculate the final statistic or posterior using the full simulated dataset.
Counting a near-threshold result as success. Apply the exact inequality and numerical cutoff. In the binary example, a score of $0.8852$ is below $0.90$, so $Y=8$ does not meet the rule.
Ignoring the design. A model for fixed exposure or complete follow-up is inadequate if the trial’s future exposure or observation process is uncertain and materially affects the outcome distribution.

Key takeaway: Predictive success is the posterior-predictive probability that the completed trial will satisfy its prespecified final rule. Estimate it by drawing parameters from the interim posterior, simulating the remaining outcomes, and applying that rule to every simulated completion.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Why must a simulation draw a parameter from the current posterior rather than use only its posterior mean?
A trial has 4 observed events and will simulate 6 more. Its final rule is at least 8 events. What condition on the simulated event count indicates success?
In the binary example, why does a simulated count of 8 fail the 0.90 rule even though its score is close to 0.90?
What does the fraction of successful simulated completions estimate?
Name one reason that a simulation could be internally correct but still give a misleading predictive success estimate.

Solutions. (1) Posterior draws represent current uncertainty in the unknown parameter; a plug-in mean omits that uncertainty. (2) The simulated count must be at least 4, because $4+K\geq8$ is equivalent to $K\geq4$. (3) Its score is $0.8852$, which is below the prespecified cutoff $0.90$; the rule is not approximate. (4) It estimates the posterior-predictive probability that the completed trial meets its final success rule. (5) For example, the future-data model may omit informative dropout, use the wrong exposure, or otherwise fail to represent the planned trial, even if the code follows that incorrect model exactly.