Attributing Predictive Uncertainty to Its Sources

Why Attribute Predictive Uncertainty?

A human prediction can be uncertain for several different reasons. The preclinical observations may be sparse, estimated parameters may be imprecise, plausible structural models may give different predictions, and the cross-species discrepancy may be poorly characterized. If the target is a future observation rather than a typical human response, residual variability may contribute as well. A single predictive interval summarizes overall uncertainty, but does not show which source is driving it.

Attribution is useful when it changes what should be learned or tested next. If parameter uncertainty dominates, additional informative data may help. If structural uncertainty dominates, comparing model forms or gathering evidence that distinguishes them may be more valuable. If discrepancy dominates, improving the translational evidence or explicitly stress-testing the discrepancy assumptions may be more relevant. The introductory material on model discrepancy distinguishes discrepancy from ordinary observation error; that distinction matters here because the two terms represent different sources of predictive uncertainty.

A source contribution is not a universal property of a prediction. It depends on the target, the probability distributions assigned to uncertain quantities, and the decomposition rule. Before calculating contributions, state what is predicted and how the uncertainty sources are represented.

Define the Target and the Sources

These labels need operational definitions. Data uncertainty refers here to variation in the prediction that would result from variation in the data-generating process or the available preclinical data. Parameter uncertainty is variation in predictions over the parameter distribution conditional on the data and model. Structural uncertainty is variation associated with the candidate model \(M\). Discrepancy uncertainty is variation due to uncertainty in the systematic difference between the translational model and the human target. If predicting a future individual observation, a separate residual term may be needed. If predicting a typical response or a population-level quantity, that residual term may not belong in \(Y\).

In a fitted analysis, these sources can be dependent. For example, the data influence the parameter distribution, and model probabilities may also depend on the data. Therefore, one should not automatically calculate four marginal variances and add them. A sequential conditional-variance decomposition gives a precise accounting, provided the joint predictive distribution is specified.

A Sequential Variance Decomposition

Consider revealing uncertainty sources in a stated order. Begin with no information, then reveal \(D\), then \(M\), then \(\theta\), then \(\delta\), and finally the realized future quantity \(Y\), if residual variation is part of the target. Write the corresponding information sets as \(\mathcal{F}_0,\ldots,\mathcal{F}_5\), where \(\mathcal{F}_0\) contains no information, \(\mathcal{F}_1\) contains \(D\), \(\mathcal{F}_2\) contains \((D,M)\), \(\mathcal{F}_3\) contains \((D,M,\theta)\), \(\mathcal{F}_4\) contains \((D,M,\theta,\delta)\), and \(\mathcal{F}_5\) also contains \(Y\). Define \(\mu_j=E(Y\mid\mathcal{F}_j)\), with \(\mu_0=E(Y)\).

This result follows from a telescoping identity and the fact that successive conditional-mean increments are uncorrelated. Since \(\mu_5=Y\), the increments sum to \(Y-E(Y)\). For \(i<j\), the increment \(\mu_i-\mu_{i-1}\) is known given \(\mathcal{F}_{j-1}\), while \(E(\mu_j-\mu_{j-1}\mid\mathcal{F}_{j-1})=0\). Thus:

Squaring the sum of increments and taking expectations therefore leaves only the squared increments:

The decomposition is exact under the specified joint distribution, but its labels depend on the reveal order. Interactions are not discarded; the order determines where their shared contribution is assigned. If the purpose requires an order-neutral allocation, a different rule—such as averaging allocations over source orderings—must be specified. That is a different estimand, not a correction to the sequential identity.

Model Averaging and Conditional Contributions

When structural models are assigned probabilities, structural uncertainty can be calculated from variation in their conditional predictive means, while parameter and discrepancy uncertainty are calculated within models. This is a conditional variance decomposition: total variance equals the variance of conditional means plus the expected conditional variance. It accommodates models with different predictions and within-model uncertainty without treating the model choice as an ordinary numeric parameter.

Estimating Contributions by Controlled Simulation

In a complex translational model, analytic variance formulas may be unavailable. A controlled simulation can approximate the conditional expectations in the decomposition. Generate data replicates from an explicit sampling model or resampling scheme; for each replicate, draw a model from its specified distribution, draw parameters conditional on that replicate and model, and draw discrepancy according to its stated distribution. Evaluate the human prediction at each stage. Use the same downstream draws when comparing successive stages where possible; this pairing can reduce Monte Carlo noise in differences of conditional means.

Diagnostics, Alternatives, and Limitations

Several checks help detect misleading attribution. First, confirm that the simulated total variance and the sum of the components agree under the chosen decomposition; a material gap can indicate inconsistent weighting, missing terms, or Monte Carlo error. Second, increase simulation sizes and check whether the contributions stabilize. Third, inspect the distributions of simulated predictions: a variance share can conceal skewness, heavy tails, or distinct modes. Fourth, verify that the data-resampling scheme represents the intended sampling uncertainty and that discrepancy is not counted both as a random model term and as residual error.

The main alternative approaches answer related but different questions. Local sensitivity measures how a prediction changes near a chosen parameter value; it does not by itself allocate predictive variance across uncertain inputs. Global sensitivity analysis varies uncertain inputs over specified distributions and can assess contributions and interactions, but the result depends on those distributions and the chosen sensitivity index. Scenario analysis compares selected assumptions without assigning them probabilities, so it is useful for stress testing but does not automatically produce variance shares. Model averaging accounts for specified between-model uncertainty, but cannot quantify uncertainty from plausible models that were omitted. These approaches can complement a variance decomposition rather than being treated as interchangeable.

Attributions are limited by the evidence and the uncertainty model. A small estimated contribution does not prove that a source is unimportant if its distribution is narrowly or poorly specified. A large contribution does not prove that reducing it is feasible or that doing so will improve a decision. Variance shares also summarize spread, not the consequences of prediction error: a decision may be highly sensitive to a low-variance source near a critical threshold. For decision-focused questions, source attribution should be interpreted alongside the prediction target and the consequences of being wrong.

• Wrong reasoning: “I estimated separate variances for data, parameters, model structure, and discrepancy, so I can add them.”
• Correction: Variances add directly only under suitable independence assumptions or within a correctly specified conditional decomposition. Dependence and interactions require an explicit joint model and allocation rule.
• Wrong reasoning: “The data contribution is the variance of parameter estimates, so I will report both as separate components.”
• Correction: Data variation and conditional parameter variation must be defined separately in a nested scheme. Otherwise, the data’s effect through parameter estimation may be counted twice.