Model-Selection Uncertainty in Clinical Predictions

Why Mechanism Uncertainty Matters for Clinical Predictions

Selecting a mechanism is not only a question about which model best describes existing data. The choice can also affect predictions used to compare doses, estimate response rates, or plan a clinical study. If several biological mechanisms remain plausible, predictions from the best-fitting model alone can hide meaningful uncertainty: a different plausible mechanism may imply a different response probability or dose threshold.

The key is to distinguish two sources of predictive uncertainty. Within-model uncertainty reflects uncertainty in parameters and future outcomes when a particular mechanism is assumed. Between-model uncertainty reflects disagreement among the candidate mechanisms about the prediction. These components answer different questions and should not be conflated with uncertainty in the estimated parameters of just one selected model.

Suppose the candidate mechanisms are indexed by \(m=1,\ldots,K\), and \(D\) denotes the evidence used to compare them. Let \(w_m=P(M=m\mid D)\) be the posterior probability of mechanism \(m\), with \(w_m\geq 0\) and \(\sum_{m=1}^K w_m=1\). For a clinical quantity \(Z\), define its conditional predictive mean and variance as \(\mu_m=E(Z\mid D,M=m)\) and \(v_m=\operatorname{Var}(Z\mid D,M=m)\). These quantities average over parameter uncertainty within model \(m\), and, when \(Z\) is a future outcome, over future-outcome variability as well.

The quantity \(Z\) must be defined consistently across mechanisms. For example, it might be the response indicator for a patient with specified characteristics at a specified dose, or a dose threshold defined by the same clinical criterion under every mechanism. The model weights and predictions are conditional on the data and on the candidate model set. They do not account automatically for a mechanism omitted from that set or for a change in the target population.

Why the Variance Decomposition Holds

Let \(M\) be the uncertain model index. The law of total variance states that the variance of \(Z\) conditional on the evidence is the expected conditional variance plus the variance of the conditional mean:

Because \(M\) takes the values \(1,\ldots,K\) with probabilities \(w_m\), the first term is \(\sum_m w_m v_m\), and the second is \(\sum_m w_m(\mu_m-\bar{\mu})^2\). This gives the decomposition above. In particular, between-model variance is zero when every model with positive weight gives the same predictive mean. It can be large even when each individual model gives a precise prediction.

The decomposition assumes that the candidate models define the same \(Z\), refer to the same target population and prediction conditions, and are assigned coherent posterior weights. Each \(v_m\) must include the uncertainty appropriate to the chosen target. For a future patient outcome, it includes outcome variability; for an underlying response probability, it need not. Being explicit about the target prevents a common misreading of the components.

Response Probability Versus Future Patient Outcomes

For a future patient, let \(Y\) be a binary response indicator, where \(Y=1\) denotes response. If the model-averaged response probability is \(P(Y=1\mid D)=\bar{q}\), then the predictive variance of the binary outcome is \(\bar{q}(1-\bar{q})\). That predictive variance includes both uncertainty in the probability and the variability of the individual outcome. It is not the same as the variance of \(Q\) in the previous example.

This distinction can be seen by conditioning first on model \(m\). If \(q_m=P(Y=1\mid D,M=m)\) is treated as fixed within each model, then \(\operatorname{Var}(Y\mid D,M=m)=q_m(1-q_m)\). The total-variance decomposition separates the average within-model Bernoulli variance from the between-model spread in response probabilities. If \(q_m\) is itself uncertain because of parameter uncertainty, that uncertainty must also be integrated within the model when calculating \(v_m\).

Dose Thresholds Require Their Own Predictive Distribution

A dose threshold is usually a derived quantity: for example, the minimum dose at which a model predicts that a specified response probability reaches a prespecified target. Because the dose-response relationship can differ by mechanism, each model induces its own posterior distribution of the threshold. The model-averaged distribution is a mixture of those distributions, weighted by \(w_m\).

Averaging threshold means can summarize the mixture, but it does not generally give the threshold corresponding to an averaged dose-response curve. Thresholds are nonlinear functions of model predictions, and some models may not reach the target at any dose in the clinically relevant range. In such cases, report the probability of no crossing or of exceeding the dose range rather than silently dropping those cases or assigning them an arbitrary finite threshold.

Diagnostics, Comparisons, and Limitations

A useful report gives the model weights, each model’s prediction, the within-model uncertainty, and the between-model contribution for the clinical quantity of interest. For a response probability, show the estimates on the probability scale; for a threshold, show component distributions or clinically relevant mixture probabilities. A variance decomposition alone can conceal skewness, multimodality, or a substantial tail, so inspect the full mixture distribution when decisions depend on quantiles or threshold exceedance.

Check that the predictions use the same dose, endpoint definition, follow-up time, and target population. Confirm that each within-model variance includes parameter uncertainty, and include future-outcome variability only when the target is a future outcome. Also inspect whether results are driven by a low-weight model with a very different prediction, whether a model assigns appreciable probability to an undefined threshold, and whether small changes in model weights materially alter the clinical summary.

A single selected model, such as the model with the largest posterior weight, gives a conditional prediction and omits between-model variation. Reporting predictions from every model separately makes disagreement visible but does not provide a weighted summary. Model averaging provides such a summary, while retaining the model-specific predictions helps reveal whether the summary is hiding important differences. A range across models is another descriptive option, but it is not a probability distribution and does not account for the model weights or within-model uncertainty.

All these approaches are limited by the candidate model set, the evidence used to assign weights, and the assumptions used to generate predictions. Model weights are not probabilities that a mechanism is biologically true in an absolute sense; they quantify relative support within the specified analysis. Neither a narrow within-model interval nor a stable model-averaged mean guarantees reliable extrapolation to a new dose range or population. Predictions should therefore be accompanied by their conditions and by evidence about how well those conditions match the intended clinical use.