Sensitivity of Predictive Success to the Final Success Criterion

Why the Final Criterion Matters

Predictive success is always success under a specified rule. It is not an intrinsic property of the interim data alone. A forecast asks how likely future trial data are to lead to a final analysis that meets the chosen criterion. If the rule changes—for example, from requiring a 90% to a 95% posterior probability, or from requiring an effect above zero to one above a positive margin—the predictive probability can change even though the interim evidence and the assumed data-generating model do not.

This distinction matters when a forecast is used to inform trial continuation or planning. A value such as “predictive success is 65%” is incomplete unless the final success criterion is stated. The probability refers to the chance, conditional on current information and the model, that the future data will satisfy that particular rule. It does not describe the probability that a treatment works, nor does changing the rule add information to the trial.

Definition and Normal-Model Setup

To derive how the criterion affects the forecast, consider a continuous endpoint with a normal model and known sampling variance. Let \(\theta\) be the treatment effect, measured in outcome units, and suppose the interim posterior is \(\theta\mid D\sim N(m,v)\). The future observations \(Y_1,\ldots,Y_r\), conditional on \(\theta\), are independent with \(Y_i\mid\theta\sim N(\theta,\sigma^2)\). Assume the future observations are collected in the same way as specified by this model, the interim posterior correctly summarizes the current information, and no unmodelled changes in the endpoint or data collection occur. These assumptions allow the future sample mean to have sampling variance \(\sigma^2/r\).

Suppose the final rule is to declare success when the final posterior probability that the effect exceeds a margin \(\delta\) is greater than \(q\), where \(0<q<1\). The margin has the same units as \(\theta\); \(\delta=0\) is a superiority rule, while a positive \(\delta\) requires the effect to exceed that positive value. The final posterior probability is calculated using all data, interim and future.

Turning the Final Rule into a Forecast

The normal-normal update gives the final posterior variance \(v_F\) and mean \(m_F\):

Because the final posterior is normal, its probability above \(\delta\) is \(\Phi((m_F-\delta)/\sqrt{v_F})\), where \(\Phi\) is the standard normal cumulative distribution function. Let \(z_q\) be the \(q\) quantile of that distribution, so \(\Phi(z_q)=q\). The final rule is met exactly when

Thus, changing \(q\) or \(\delta\) moves the cutoff that the final posterior mean must exceed. The interim posterior mean \(m\) and the predictive distribution of future data have not changed; the event counted as success has changed.

Under the posterior predictive distribution, the final posterior mean has distribution \(m_F\mid D\sim N(m,v-v_F)\). One way to justify its variance is to write \(A=1/v\) for current precision and \(B=r/\sigma^2\) for future-data precision. The weight on \(\bar{Y}_{\mathrm{new}}\) in \(m_F\) is \(v_FB\). Since \(\bar{Y}_{\mathrm{new}}\mid D\) has mean \(m\) and variance \(v+1/B\), the predictive variance of \(m_F\) is

The equality follows by substituting \(v=1/A\) and \(v_F=1/(A+B)\). The mean of \(m_F\mid D\) is \(m\), since the future sample mean has predictive mean \(m\). Therefore predictive success has the analytic expression

This derivation also clarifies that the forecast is not the current posterior probability of exceeding \(\delta\). It averages over possible final posterior means generated by future data, then counts the proportion of those possibilities that satisfy the final rule.

Monotonicity, Diagnostics, and Alternatives

For a fixed model and interim posterior, raising \(q\) raises \(z_q\), and raising \(\delta\) raises the cutoff \(\delta+z_q\sqrt{v_F}\). Since the predictive probability is the upper-tail probability beyond that cutoff, either change can only leave predictive success unchanged or lower it; in this continuous normal example, it lowers it strictly. This gives a useful diagnostic: if a calculation reports higher predictive success after making the final criterion more stringent while everything else is held fixed, check the inequality direction, tail calculation, and input values.

A second check is to separate the calculation into two stages: first compute the final posterior cutoff, then compute the probability that the predictive final posterior mean exceeds it. The cutoff is in effect units, and the standardization denominator \(\sqrt{v-v_F}\) is also in effect units. Their ratio must be dimensionless. A frequent computational error is to use \(\sqrt{v_F}\) as the predictive standard deviation of \(m_F\). That is the final posterior standard deviation for \(\theta\), not the predictive standard deviation of the final posterior mean; the latter is \(\sqrt{v-v_F}\).

An analytic calculation is convenient when conjugacy and the normal assumptions apply. Simulation is a useful alternative when the future-data model or final rule makes the predictive probability difficult to calculate directly: repeatedly draw future data from the posterior predictive distribution, apply the same final rule to every simulated data set, and estimate the probability by the fraction meeting the rule. Simulation does not remove the need to specify \(q\), \(\delta\), or the model, and a simulation estimate also has Monte Carlo error.

The sensitivity studied here is distinct from changing the prior while holding the final rule fixed. In this tutorial, the interim posterior and future-data model are fixed while the event called success changes. It is also distinct from comparing posterior probability at the interim analysis with predictive success: the former summarizes current evidence, while the latter concerns whether future data will lead to a final analysis meeting a rule.

Limitations and Common Pitfall

The normal calculation assumes known sampling variance, a normal interim posterior, and a normal future sampling distribution. It may be inappropriate for binary outcomes, time-to-event outcomes, uncertain control response, adaptive future sample sizes, or a design whose future observations depend on interim decisions. Those settings require a predictive distribution and final posterior calculation matched to the actual model and design. The general definition—probability of meeting the stated final criterion over future data—still applies, but the closed-form expression may not.

Common mistake: “Predictive success is 64%, so there is a 64% probability the treatment effect is positive.” This confuses the final decision event with the parameter event. In the example, 63.9% is the probability, conditional on the interim information and model, that future data will produce a final posterior probability above 95% for \(\theta>0\). It is not the current posterior probability that \(\theta>0\), nor a probability that a fixed parameter will change.