Why nuisance-parameter uncertainty matters
A predictive probability of success is a probability, conditional on interim data, that a trial will meet its planned success criterion at the end. The earlier material on predictive probability establishes the basic averaging idea: future outcomes are uncertain, and their conditional success probabilities are evaluated using information available at the interim analysis. A further uncertainty arises when a nuisance parameter, such as the residual standard deviation, is not known.
A common shortcut is to estimate the nuisance parameter once and calculate the future success probability as if that estimate were the true value. This can be useful as a sensitivity calculation, but it does not account for uncertainty in the estimate. A posterior predictive calculation instead averages the conditional success probability over the posterior distribution of the nuisance parameter. Since the conditional probability is generally a nonlinear function of that parameter, the average need not equal the probability evaluated at a posterior mean or another single estimate.
This expression averages over the posterior distribution of the nuisance parameter while retaining the stated model for the treatment effect. It does not, by itself, average over treatment-effect uncertainty; that is a distinct modelling choice. If nuisance parameters and treatment-effect parameters are jointly uncertain or dependent in the posterior, the integration must use their joint posterior distribution rather than treating them as independent.
One-final-look normal model
Consider a trial with a single remaining final analysis. Conditional on the interim data and a fixed treatment-effect assumption, suppose the final estimator \(Y_T\) has distribution
where \(m_I\) is its conditional mean, \(\sigma\) is an unknown residual standard deviation, and \(c>0\) is a known design constant determined by the future information. For example, if the remaining independent observations have variance \(\sigma^2\) and the estimator's future contribution has variance \(\sigma^2/n_F\), then \(c=1/n_F\). Suppose success is defined as \(Y_T\ge h\), where \(h\) is a prespecified threshold on the estimator's scale. Conditional on \(\sigma\), the success probability is therefore
The predictive probability is \(E\{q(\sigma)\mid D_I\}\), not automatically \(q\{E(\sigma\mid D_I)\}\). The latter is a plug-in calculation. Both require the same conditional sampling model, success criterion, and interim conditioning; they differ in how they handle uncertainty about \(\sigma\).
Worked Example: Averaging over two plausible standard deviations
Worked Example: Averaging over two plausible standard deviations
Suppose a hypothetical trial has \(m_I=0.196\), success threshold \(h=0.168\), and \(c=60/10000\). At the interim analysis, the posterior distribution assigns probability \(0.5\) to \(\sigma=0.8\) and probability \(0.5\) to \(\sigma=1.2\). These values and probabilities are hypothetical. Calculate the integrated predictive probability and compare it with plugging in the posterior mean of \(\sigma\).
For \(\sigma=0.8\), the future standard deviation is \(\sigma\sqrt{c}=0.8\sqrt{60}/100\approx0.06197\). Thus
Equivalently, the probability of falling below the threshold is \(\Phi(-0.4519)\approx0.3257\). Because success is \(Y_T\ge h\), the success probability is the upper-tail probability, \(0.6743\), not \(0.3257\).
For \(\sigma=1.2\), the future standard deviation is \(1.2\sqrt{60}/100\approx0.09295\), giving
The integrated success probability is the posterior-weighted average:
The posterior mean of \(\sigma\) is \(0.5(0.8)+0.5(1.2)=1.0\). Plugging in \(\sigma=1.0\) gives \(q(1.0)=\Phi\{0.028/(\sqrt{60}/100)\}=\Phi(0.3615)\approx0.6410\). This differs from the integrated result, although both are near \(0.64\). The direction and size of the difference depend on the model and values; there is no general rule that integration must increase or decrease predictive probability.
Posterior integration for an unknown variance
A useful continuous example arises when the posterior distribution of the variance has an inverse-gamma form. Let \(V=\sigma^2\) and suppose, conditional on the interim data, \(V\) has density
with shape \(a>0\) and scale \(b>0\). Under the normal future-estimator model, integrating over \(V\) gives a Student \(t\) predictive distribution. This provides a direct calculation for a threshold event rather than requiring a separate normal-tail calculation for every possible variance.
To justify the result, write \(V=2b/W\), where \(W\mid D_I\) has a chi-square distribution with \(2a\) degrees of freedom. Conditional on \(V\), write \(Y_T=m_I+\sqrt{cV}Z\), with \(Z\sim N(0,1)\) independent of \(W\). Substitution gives
The last ratio has a Student \(t\) distribution with \(2a\) degrees of freedom, proving the stated predictive distribution. It has heavier tails than a normal distribution with a fixed variance, reflecting the additional uncertainty about \(V\). The derivation uses the conditional normal model and inverse-gamma posterior; it does not assert that every variance posterior produces a Student \(t\) distribution.
Worked Example: Integrating an inverse-gamma variance posterior
Suppose, hypothetically, \(m_I=0.180\), \(h=0.160\), \(c=0.006\), and the posterior variance \(V\mid D_I\) is inverse-gamma with \(a=5\) and \(b=0.25\). Here \(2a=10\), and the predictive \(t\) scale is \(\sqrt{cb/a}=\sqrt{0.006(0.25)/5}=\sqrt{0.0003}\approx0.01732\).
The standardized threshold is \((h-m_I)/0.01732=(-0.020)/0.01732\approx-1.155\). Therefore
For comparison, the posterior mean variance is \(b/(a-1)=0.25/4=0.0625\), so the corresponding plug-in standard deviation is \(0.25\). Plugging that value into the normal conditional calculation gives \(\Phi\{(0.180-0.160)/(0.25\sqrt{0.006})\}=\Phi(1.033)\approx0.8492\). The plug-in probability is lower here; the integrated probability accounts for the full posterior spread in variance. The posterior mean variance exists because \(a>1\), a condition needed for this particular plug-in comparison.
Worked Example: A smaller standard deviation and a different threshold
Worked Example: A smaller standard deviation and a different threshold
Consider another hypothetical final estimator with \(m_I=0.196\), threshold \(h=0.160\), and \(c=60/10000\). Suppose the posterior gives equal probability to \(\sigma=0.5\) and \(\sigma=1.2\). Calculate the success probability at each value and integrate.
At \(\sigma=0.5\), the future standard deviation is \(0.5\sqrt{60}/100\approx0.03873\). The standardized distance above the threshold is \((0.196-0.160)/0.03873\approx0.9295\). Thus \(q(0.5)=\Phi(0.9295)\approx0.8237\). Equivalently, the lower-tail probability is \(\Phi(-0.9295)\approx0.1763\).
At \(\sigma=1.2\), the future standard deviation is approximately \(0.09295\), so the standardized distance is \(0.036/0.09295\approx0.3873\). Hence \(q(1.2)=\Phi(0.3873)\approx0.6508\). The integrated predictive probability is
The posterior mean standard deviation is \(0.85\). A normal plug-in calculation gives \(\Phi\{0.036/(0.85\sqrt{60}/100)\}\approx\Phi(0.5467)\approx0.7077\). The \(0.7373\) integrated result and the \(0.7077\) plug-in result differ because averaging a nonlinear tail probability is not equivalent to evaluating it at an average standard deviation.
Practical computation and diagnostics
For a general nuisance parameter \(\eta\), the calculation can be organized in three steps. The final success event and conditional model are specified first; then conditional success probabilities are computed at values drawn from or integrated over the posterior; finally, those probabilities are averaged using posterior weights. This order helps prevent accidental use of prior weights after interim data have changed the relative plausibility of nuisance-parameter values.
State the distribution of the final statistic given interim data and nuisance parameter, and define success using the trial's actual final criterion.
Use \(\pi(\eta\mid D_I)\), including any dependence between nuisance parameters that the model entails.
Evaluate \(\int P(S\mid D_I,\eta)\pi(\eta\mid D_I)d\eta\), or its posterior-weighted sum for discrete values.
If useful, report a result using a posterior mean or another specified estimate, but identify it as a plug-in approximation rather than the integrated predictive probability.
A useful diagnostic is to inspect conditional probabilities across the posterior range of the nuisance parameter. If those probabilities vary little where most posterior mass lies, the plug-in approximation may be close. If they change sharply, or the posterior has substantial mass in regions with very different probabilities, integration can matter appreciably. Numerical integration should also be checked for adequate coverage of posterior tails and stability as the integration grid or simulation size is increased.
Common failure modes include using the wrong tail for the success event, integrating against the prior rather than the interim posterior, and substituting a point estimate before calculating the nonlinear success probability. Another error is to average standard errors and then treat the average as fixed. The relevant operation is to average conditional probabilities, not to replace the variance distribution by an average standard error. A plug-in calculation is not necessarily conservative: the curvature of \(q(\eta)\), the location of the posterior mass, and the form of the success event determine the direction of the difference.
What the calculation does and does not establish
Nuisance-parameter integration is appropriate when the uncertainty distribution is defensible and the conditional model is credible. It can represent uncertainty about residual variance, event rates, or another parameter that affects the distribution of future trial data. For example, an event-rate posterior may imply uncertainty in the amount of information available by a given time; that uncertainty belongs in the predictive calculation if it changes the future sampling distribution.
The calculation remains conditional on the chosen prior, likelihood, interim data, treatment-effect assumption, and analysis plan. A precise integrated value does not guarantee that those inputs are appropriate. In particular, a posterior for variance based on a poorly fitting model may not capture heavy tails, changing variance, or dependence between observations. Likewise, a calculation that holds the treatment effect fixed answers a different question from one that also averages over treatment-effect uncertainty. Keep that distinction explicit rather than allowing nuisance-parameter averaging to be mistaken for a full uncertainty analysis.
The main alternative is to report a plug-in conditional probability at a selected nuisance estimate. It is simpler and can be useful for transparency or as one scenario, but it suppresses uncertainty around that estimate. A full posterior predictive integration is more coherent when posterior uncertainty is material, though it requires a justified posterior and potentially numerical computation. Neither approach compensates for a misspecified final success criterion or an incorrect future-data model.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- In the normal model, what quantity is averaged to obtain predictive probability when \(\sigma\) is uncertain?
- Why does evaluating the conditional probability at the posterior mean of \(\sigma\) generally differ from integrating over its posterior?
- For a success event \(Y_T\ge h\), explain why a lower-tail probability must be complemented.
- Under the inverse-gamma variance model, what are the degrees of freedom and scale of the marginal predictive \(t\) distribution?
- Name two assumptions or diagnostics that should be checked before trusting a nuisance-integrated predictive probability.
Solutions. (1) Average \(P(S\mid D_I,\sigma)\) with respect to \(\pi(\sigma\mid D_I)\). (2) The conditional success probability is generally nonlinear in \(\sigma\), so the average of its values need not equal its value at the average \(\sigma\). (3) Success is the upper-tail event, so \(P(Y_T\ge h)=1-P(Y_T<h)\); using the lower tail alone reverses the event. (4) The degrees of freedom are \(2a\), and the scale is \(\sqrt{cb/a}\). (5) Examples include checking the conditional future-data model and final boundary, verifying that the posterior is conditioned on the interim data, and checking that numerical integration covers posterior tails and is stable.