Prediction Intervals for Decision-Relevant Outcomes

Why Predictive Uncertainty Matters

A development decision often concerns outcomes that have not yet been observed: how many patients might respond in a planned cohort, how many safety events might occur, or what outcome an individual patient might experience. A model’s estimated average is not a complete answer. Future outcomes vary between patients, and the model parameters used to predict them are themselves uncertain. A prediction interval represents both sources of uncertainty for a specified future outcome or group of outcomes.

The interval depends on the prediction target. Predicting the mean response probability is different from predicting the number of responses in a new cohort. The first concerns a model quantity; the second also includes the variability of future patient outcomes. State the target, its units, and the population and time frame before calculating an interval.

Combining Parameter Uncertainty and Patient Variability

Let \(\theta\) denote uncertain model parameters and \(Y_{\text{new}}\) a future outcome. A predictive distribution is formed by averaging the conditional distribution of the future outcome over the uncertainty distribution for \(\theta\):

Here \(p(\theta)\) represents the distribution used to characterize parameter uncertainty, such as a posterior distribution or a distribution obtained by a suitable resampling procedure. The conditional distribution \(P(Y_{\text{new}}\leq y\mid\theta)\) represents patient-level variability under the model. The parameter distribution must be appropriate to the analysis; a range of plausible values alone is not a probability distribution and cannot be averaged as one without assigning weights.

A useful justification follows from the law of total variance. For a future outcome with finite variance, write its deviation from its overall mean as its deviation from the conditional mean plus the deviation of the conditional mean from the overall mean. The cross-product has expectation zero because the conditional deviation has mean zero given \(\theta\). Therefore:

The first term is average patient-level variability conditional on the parameters. The second is variability in the model’s conditional mean as the parameters vary. This identity explains why an interval for a predicted mean alone is usually narrower than an interval for an individual future outcome.

For a future count \(K\) of responses among \(n\) patients, suppose each patient responds independently conditional on response probability \(p\), so \(K\mid p\sim\operatorname{Binomial}(n,p)\). If \(p\) is uncertain, then:

The first variance term reflects the binomial variation in patient outcomes; the second reflects uncertainty in \(p\). When the same uncertain \(p\) applies to the whole future cohort, patients’ outcomes are generally dependent marginally, even though they are independent conditional on \(p\). Sampling a separate \(p\) for every patient would represent a different model.

Worked Safety-Event Prediction

Safety outcomes are often expressed as counts over a defined number of patients and follow-up period. The prediction target should specify both: an event count without a follow-up duration is incomplete if risk accumulates over time. The conditional outcome model must also match the event process. For the following illustration, use a binomial model for whether each patient experiences at least one event during a fixed follow-up period.

Continuous Outcomes and a Prediction-Interval Derivation

For a continuous patient outcome, a common starting model is \(Y_{\text{new}}\mid\mu\sim N(\mu,\sigma^2)\), where \(\mu\) is an uncertain population mean and \(\sigma\) describes patient-level variation. If \(\mu\sim N(m,\tau^2)\), and the future patient’s residual variation is independent of uncertainty in \(\mu\), then \(Y_{\text{new}}\) is normally distributed with mean \(m\) and variance \(\tau^2+\sigma^2\). This follows because the outcome can be written as the sum of the uncertain mean and an independent normal residual.

Construction Choices, Diagnostics, and Limitations

When a predictive distribution is not available in closed form, it can be approximated by simulation. Draw parameter values from the selected uncertainty distribution; for each draw, simulate the requested future outcome from the conditional patient-outcome model; then calculate the desired quantiles from the simulated outcomes. For cohort predictions, simulate the full cohort under each parameter draw, preserving shared parameters, dependence, and any relevant structure. A prediction interval based on too few simulated outcomes can have unstable endpoints, so simulation precision should be checked.

A prediction interval is only as credible as the assumptions used to construct it. Check that the target population, dose, follow-up, and outcome definition match the decision context. Examine whether the conditional outcome model fits the relevant data and whether its residual variability is appropriate. For example, a binomial model requires a defined binary outcome and a suitable account of heterogeneity; a normal model may be unsuitable for a strongly skewed, bounded, or count outcome. If the model omits patient subgroups or dependence that matter for the target, the resulting interval may be misleadingly narrow or otherwise poorly calibrated.

A useful comparison is between a full prediction interval, a parameter-only interval, and a plug-in prediction based on a single parameter estimate. The parameter-only interval omits future patient variation. A plug-in prediction can include patient variation conditional on the chosen estimate, but omits uncertainty in that estimate. Neither generally answers the same question as a prediction interval that includes both sources. Analytic calculations are convenient when their assumptions are appropriate; simulation is more flexible for complex models but requires careful implementation and adequate Monte Carlo precision.

The interval also inherits limitations from its uncertainty distribution and model structure. A posterior predictive distribution depends on the likelihood and prior assumptions; a resampling-based parameter distribution depends on the resampling procedure and available data. Neither corrects a misspecified patient-outcome model by itself. An interval for one individual should not be substituted for a prediction of a cohort count, and a cohort-level interval should not be read as an individual-level range. Always report what is being predicted, for whom, over what period, and in what units.

Common Pitfall

Prediction intervals support decisions by showing a range of future outcomes compatible with the specified model and uncertainty assumptions. They do not establish that an outcome is acceptable, that the model is correct, or that a particular development choice is optimal. Their decision value comes from comparing the predicted outcomes with the consequences and criteria relevant to the choice.