Sensitivity Analysis for Predictive Probability

Why Examine Prior Sensitivity?

A predictive probability of success combines the interim evidence with uncertainty about the treatment effect. Its value can therefore change when the prior distribution changes, even if the interim data, trial design, and success criterion remain fixed. Sensitivity analysis makes this dependence visible: rather than reporting one number as if it were determined by the data alone, it shows how plausible choices about prior location, spread, or skeptical weight alter the trial-success assessment.

The analysis below uses a normal canonical-score model and one future look. The same logic applies to a predictive probability defined by success at a specified later look, provided the corresponding future-score distribution and success boundary are used. The prior distributions are assumptions for illustration, not claims about an actual treatment effect.

Definition: The predictive probability of success at an interim look is the probability, conditional on the interim data and a specified model and prior, that the trial will meet its defined success criterion at a future analysis. It averages the future success probability over the posterior distribution of unknown parameters. It is not the probability that \(H_0\) is true, nor a guarantee that the trial will succeed.

Model and Predictive Probability

Let \(\theta\) denote the treatment effect on a canonical scale, so that its contribution to a score with information \(I\) is \(\theta\sqrt{I}\). At the interim analysis, information is \(I\), the observed interim statistic is \(Z_I\), and the corresponding estimate is \(\hat{\theta}=Z_I/\sqrt{I}\). Assume the normal likelihood

\hat{\theta}\mid\theta \sim N\left(\theta,\frac{1}{I}\right).

This model assumes the canonical normal approximation is adequate, the information values are specified, and the future score increment is independent of the observed interim increment conditional on \(\theta\). We consider a one-sided final success rule \(Z_F>c\), where \(c\) is fixed in advance. In an actual group sequential design, the appropriate future boundary must reflect that design’s specified success rule; the numerical example uses a single final boundary.

Let \(J\) be the remaining information and \(I_F=I+J\) the final information. Write \(W\) for the future score increment standardized to have variance 1 conditional on \(\theta\). Under the model,

W\mid\theta \sim N(\theta\sqrt{J},1), \qquad Z_F=\frac{\sqrt{I}\,Z_I+\sqrt{J}\,W}{\sqrt{I_F}}.

The first term in the final statistic is the observed interim score; the second is the future score contribution. Therefore success at the final analysis is equivalent to \(W>k\), where

k=\frac{c\sqrt{I_F}-\sqrt{I}\,Z_I}{\sqrt{J}}.

How a Normal Prior Enters

For a normal prior \(\theta\sim N(m_0,\tau^2)\), the posterior is normal. Combining the prior precision \(1/\tau^2\) with the interim likelihood precision \(I\) gives posterior variance \(v\) and mean \(m\):

v=\frac{1}{I+1/\tau^2}, \qquad m=v\left(I\hat{\theta}+\frac{m_0}{\tau^2}\right).

Given the interim data, the future increment \(W\) has a posterior predictive distribution. Its mean is the posterior average of \(\theta\sqrt{J}\). Its variance includes both the conditional future-score variance and the uncertainty in \(\theta\):

W\mid\text{interim data}\sim N\left(\sqrt{J}\,m,\ 1+Jv\right).

Consequently, the predictive probability of final success is

P(Z_F>c\mid\text{interim data}) = 1-\Phi\left(\frac{k-\sqrt{J}\,m}{\sqrt{1+Jv}}\right),

where \(\Phi\) is the standard normal cumulative distribution function. This expression makes the sensitivity mechanisms explicit: changing prior location can change \(m\), while changing prior spread can change both \(m\) and \(v\). Neither change is guaranteed to move the predictive probability in the same direction in every setting.

Worked Sensitivity Analyses

For the examples, suppose \(I=50\), \(J=50\), \(I_F=100\), \(Z_I=1.50\), and the final one-sided success boundary is \(c=1.96\). Thus \(\hat{\theta}=1.50/\sqrt{50}=0.2121\), in canonical effect units. These are hypothetical values. The observed interim score contributes \(\sqrt{50}(1.50)=10.61\) to the unstandardized final score, and the future score must satisfy

W>k=\frac{1.96\sqrt{100}-1.50\sqrt{50}}{\sqrt{50}} =1.2719.

Worked Example: A Skeptical, Tightly Concentrated Prior

Take the prior \(\theta\sim N(0.04,0.0004)\). Its standard deviation is \(0.0200\), so it concentrates strongly near a small positive effect. Calculate the posterior and predictive probability using the interim result above.

The prior precision is \(1/0.0004=2500\), and the likelihood precision is \(I=50\). The posterior variance and mean are

v=\frac{1}{2500+50}=0.0003922, \qquad m=\frac{2500(0.04)+50(0.2121)}{2550}=0.04337.

The predictive future-score mean is \(\sqrt{50}(0.04337)=0.3067\), and its standard deviation is

\sqrt{1+50(0.0003922)}=\sqrt{1.0196}=1.010.

Thus the standardized distance from the predictive mean to the success threshold is \((1.2719-0.3067)/1.010=0.9557\). The predictive probability is \(1-\Phi(0.9557)\approx0.1696\). Under this tightly concentrated prior and the stated model, the probability of crossing the final boundary is about 17.0%.

Worked Example: Increasing Prior Spread

Keep the prior mean at \(0.04\), but increase the prior variance to \(0.04\), giving prior standard deviation \(0.20\). This broad prior allows substantially more treatment-effect uncertainty. Recalculate rather than assuming that greater uncertainty must increase predictive probability.

The prior precision is now 25. The posterior quantities are

v=\frac{1}{25+50}=0.01333, \qquad m=\frac{25(0.04)+50(0.2121)}{75}=0.1547.

The predictive mean is \(\sqrt{50}(0.1547)=1.094\), and the predictive standard deviation is \(\sqrt{1+50(0.01333)}=1.291\). Therefore

P(Z_F>1.96\mid\text{data}) = 1-\Phi\left(\frac{1.2719-1.094}{1.291}\right) = 1-\Phi(0.1378) \approx 0.445.

The estimate moves toward the interim estimate because the broader prior gives the likelihood more influence; posterior uncertainty also increases. In this example, both effects contribute to a larger predictive probability, about 44.5%. This comparison changes the prior variance while holding the prior mean, interim result, and design fixed.

Worked Example: Changing Prior Location

Now hold the prior variance at \(0.04\) and compare prior means \(m_0=0\) and \(m_0=0.10\). The prior spread, data, and design remain fixed.

For \(m_0=0\), the posterior variance remains \(0.01333\), and \(m=(25(0)+50(0.2121))/75=0.1414\). The predictive mean is \(1.000\), and the predictive standard deviation is \(1.291\). The success probability is \(1-\Phi((1.2719-1.000)/1.291)=1-\Phi(0.2106)\approx0.4165\).

For \(m_0=0.10\), the posterior mean is \(m=(25(0.10)+50(0.2121))/75=0.1747333\). The predictive mean is \(\sqrt{50}(0.1747333)\approx1.2356\), with the same standard deviation \(1.291\). The probability is \(1-\Phi\left(\frac{1.2719-\sqrt{50}(0.1747333)}{\sqrt{1+50(1/75)}}\right)=1-\Phi(0.02816)\approx0.4888\).

Here a higher prior location raises the predictive probability from about 41.7% to 48.9%. The direction follows from the particular boundary and interim evidence; prior-location sensitivity should be recalculated, not inferred from a universal rule.

Skeptical Mixtures Require Updated Weights

A skeptical mixture prior assigns probability to more than one prior component, for example a near-null component and a component allowing larger effects. The prior mixture weight is not generally the weight to use when averaging success probabilities after observing interim data. Bayes’ rule updates the component weights using the interim likelihood.

Worked Example: Varying a Skeptical Mixture Weight

Suppose the prior is a mixture with weight \(0.80\) on \(N(0,0.04)\) and weight \(0.20\) on \(N(0.20,0.04)\). The observed estimate is \(\hat{\theta}=0.2121\), with sampling variance \(1/I=0.0200\). Under either component, the marginal distribution of \(\hat{\theta}\) has variance \(0.04+0.02=0.06\). The ratio of the second component’s marginal density to the first is

\frac{f_1(\hat{\theta})}{f_0(\hat{\theta})} = \exp\left(\frac{0.2121^2-(0.2121-0.20)^2}{2(0.06)}\right) \approx 1.453.

The posterior weight on the larger-effect component is therefore \((0.20\times1.453)/(0.80+0.20\times1.453)=0.2665\), not \(0.20\). Within each component, the posterior variance is \(1/(25+50)=0.01333\). The component posterior means are \(0.1414\) for the near-null component and \(0.2081\) for the larger-effect component.

Their predictive success probabilities are approximately \(0.4165\) and \(0.5614\), respectively, using the same threshold and predictive standard deviation \(1.291\). Using the unrounded component probabilities and updated weight, the mixture predictive probability is about \(0.4551\), or 45.5%. Changing the initial mixture weight would change both the updated weights and the overall assessment; it is not correct simply to average component probabilities using the prior weights after seeing the interim data.

Derivation, Diagnostics, and Alternatives

The normal predictive formula follows from the law of total expectation and variance. Conditional on \(\theta\), the future score has mean \(\theta\sqrt{J}\) and variance 1. Averaging over the posterior gives \(E(W\mid\text{data})=\sqrt{J}m\). For the variance, the law of total variance gives

\operatorname{Var}(W\mid\text{data}) = E\{\operatorname{Var}(W\mid\theta,\text{data})\mid\text{data}\} + \operatorname{Var}\{E(W\mid\theta,\text{data})\mid\text{data}\} =1+Jv.

Because the posterior and conditional score distributions are normal, this mean and variance specify the predictive distribution completely. The probability is then its upper-tail area beyond \(k\). This derivation also shows why using only a posterior mean as if it were the known effect discards the \(Jv\) contribution to predictive uncertainty.

A useful sensitivity analysis varies one prior feature at a time before examining selected combinations. Record each prior specification, its posterior location and spread, the updated mixture weights when relevant, and the resulting predictive probability. Check that the same interim data, information, success boundary, and likelihood are used in every comparison. Large changes across reasonable prior choices indicate that the assessment is prior-sensitive and should be reported as such.

Common warning signs include a mixture analysis whose weights are not updated, a probability calculated from the prior rather than the posterior, and a calculation that changes the success boundary while attributing the result to a prior change. Also check the canonical normal approximation and information assumptions: non-normal or discrete outcomes, inaccurate information values, dependence not represented by independent increments, or a misspecified effect model can make the displayed calculations unreliable. Sensitivity analysis does not repair a poor likelihood or an incorrect design model.

Predictive probability differs from conditional power, which calculates future success probability at a fixed assumed effect rather than averaging over a posterior distribution. A plug-in calculation at the interim estimate is another fixed-effect assessment; it omits posterior uncertainty as well. These alternatives can be useful reference points, but they answer different questions. Predictive probability is conditional on the observed interim data and the chosen prior, so it is not a frequentist operating characteristic such as a type I error rate and should not be interpreted as a probability that the treatment is effective.

The normal-prior formula is convenient, but its limitation is equally important: a sharply concentrated prior can dominate the data, while a broad prior can leave substantial posterior predictive uncertainty. Mixture priors add sensitivity to both component distributions and component weights. A thorough report therefore identifies the prior choices that materially affect the conclusion, rather than presenting only a preferred specification.

Common pitfall: “The interim estimate is favorable, so the predictive probability must be high.” This ignores the future information, boundary, prior, and remaining uncertainty. The correction is to calculate the posterior predictive distribution under the stated model and evaluate its probability of exceeding the actual success threshold.

Check Your Understanding

Use the model and notation in this tutorial. Answers follow the questions so that each calculation can be checked directly.

For the normal prior model, which posterior quantities determine the predictive mean and variance of \(W\)?
Why is the future-score predictive variance \(1+Jv\), rather than just 1?
With the worked-example design, what success threshold must \(W\) exceed?
In a two-component prior mixture, why can the posterior mixture weight differ from the prior mixture weight?
State one reason predictive probability and conditional power can disagree.

Solutions. (1) The posterior mean \(m\) and variance \(v\) determine them: the mean is \(\sqrt{J}m\), and the variance is \(1+Jv\). (2) The conditional future-score variance is 1, and uncertainty in \(\theta\) adds \(\operatorname{Var}(\theta\sqrt{J}\mid\text{data})=Jv\). (3) \(W>1.2719\), found from \((1.96\sqrt{100}-1.50\sqrt{50})/\sqrt{50}\). (4) Interim data have different marginal likelihoods under the components, so Bayes’ rule changes their relative weights. (5) Conditional power fixes \(\theta\) at an assumed value, whereas predictive probability averages over the posterior distribution of \(\theta\); posterior uncertainty and prior choice can therefore alter the latter.