Averaging Predictive Distributions Instead of Parameters

Why Average Predictions Rather Than Parameters?

When several fitted NLME models are plausible, a prediction for an external patient can differ substantially across models. One way to represent that uncertainty is to combine the models’ predictive distributions. This retains each model’s own structure and uncertainty. Averaging parameter estimates first is different: it creates a new parameter vector, which may describe a model that was never fitted and whose predictions need not resemble the combined predictions from the original models.

The distinction matters especially for nonlinear compartment models. Their concentration predictions depend nonlinearly on parameters such as clearance, volume, and rate constants. In general, applying a nonlinear prediction function to an average parameter vector does not produce the average of the predictions.

Definition: Suppose $M$ candidate models give predictive distributions for the same future outcome $Y_*$, conditional on the same target information $x_*$ and training data $D_{\mathrm{train}}$. A weighted predictive distribution is the mixture $$ p_{\mathrm{mix}}(y_* \mid x_*,D_{\mathrm{train}}) =\sum_{m=1}^{M} w_m\,p_m(y_* \mid x_*,D_{\mathrm{train}}), \qquad w_m\geq 0,\quad \sum_{m=1}^{M}w_m=1. $$ The weights are fixed for the calculation here. They express how much each model-specific predictive distribution contributes to the combined prediction; they are not probabilities that the individual models are true.

The models must predict the same quantity for the same target. For example, predictions for concentration at the same time after the same dose, in patients with the same specified covariates, can be combined. A population prediction and an individual prediction conditional on that individual’s concentration history are not automatically interchangeable. The target, conditioning information, units, and included sources of predictive uncertainty must be aligned first.

Each component distribution should represent the predictive target consistently. Depending on that target, it may include residual variability, variation in individual random effects, and parameter uncertainty. A plug-in prediction that holds some quantities fixed can also be combined, but it represents a narrower uncertainty statement. The earlier material on defining external prediction targets and generating population and individual external predictions provides the relevant distinctions.

Nonlinearity Makes Parameter Averaging Different

For a one-compartment intravenous bolus model with linear elimination, a concentration prediction can be written as $C(t)=\frac{\mathrm{Dose}}{V}\exp(-kt)$, where $k=CL/V$. Even when candidate models share this same form and differ only in $k$, the prediction is nonlinear in $k$. With model weights $w_m$, the average predicted concentration is $\sum_m w_m \frac{\mathrm{Dose}}{V}\exp(-k_m t)$. The prediction using the weighted-average rate constant is $\frac{\mathrm{Dose}}{V}\exp[-(\sum_m w_m k_m)t]$, which is generally not equal to the first expression.

Worked Example: Averaging Predictions From Two One-Compartment Models

Suppose two fitted one-compartment models use the same hypothetical dose and volume, with $\mathrm{Dose}/V=10$ mg/L, but have elimination rate constants $k_1=0.10\ \mathrm{h}^{-1}$ and $k_2=0.05\ \mathrm{h}^{-1}$. Consider the concentration prediction 10 h after the dose and assign each model weight $0.5$.

The model-specific predictions are: $$ C_1(10)=10\exp[-(0.10)(10)]=10e^{-1}=3.679\ \mathrm{mg/L}, $$ $$ C_2(10)=10\exp[-(0.05)(10)]=10e^{-0.5}=6.065\ \mathrm{mg/L}. $$

The weighted average of these predicted concentrations is $$ 0.5(3.679)+0.5(6.065)=4.872\ \mathrm{mg/L}. $$

If instead the rate constants are averaged first, $\bar{k}=0.5(0.10)+0.5(0.05)=0.075\ \mathrm{h}^{-1}$. The resulting prediction is $10\exp[-(0.075)(10)]=10e^{-0.75}=4.724\ \mathrm{mg/L}$, not $4.872\ \mathrm{mg/L}$. Both calculations are arithmetically valid, but they answer different questions: the first averages the two predictions, while the second predicts from a new rate constant.

This example uses point predictions to isolate the nonlinear difference. A point prediction can be viewed as a degenerate predictive distribution concentrated at that value. In an external prediction task with uncertainty, the more complete object to combine is each model’s predictive distribution, not only its mean.

Properties of a Predictive Mixture

A mixture is formed by first selecting a component model according to the weights and then drawing an outcome from that model’s predictive distribution. Equivalently, its cumulative distribution function is the weighted sum of the component cumulative distribution functions:

F_{\mathrm{mix}}(y)=\sum_{m=1}^{M} w_m F_m(y).

This construction preserves features that can be lost by collapsing each model to a single number. For example, the mixture can be skewed or multimodal even if each component distribution is unimodal. Its predictive mean is the weighted mean of component means when those means exist. Its variance includes both within-model predictive variability and differences between model-specific means.

Result: If model $m$ has predictive mean $\mu_m$ and variance $\sigma_m^2$, then the mixture has $$ \mu_{\mathrm{mix}}=\sum_m w_m\mu_m, \qquad \sigma_{\mathrm{mix}}^2 =\sum_m w_m\left[\sigma_m^2+(\mu_m-\mu_{\mathrm{mix}})^2\right]. $$ The first variance term averages uncertainty within the component models; the second represents spread between their predictive means.

To justify the variance formula, write the mixture outcome as $Y_*$, and condition on which component model generated it. The law of total variance gives $\operatorname{Var}(Y_*)=E[\operatorname{Var}(Y_*\mid M)]+\operatorname{Var}(E[Y_*\mid M])$. The first expectation is $\sum_m w_m\sigma_m^2$. The second variance is $\sum_m w_m(\mu_m-\mu_{\mathrm{mix}})^2$, since $E[Y_*\mid M=m]=\mu_m$ and the overall mean is $\mu_{\mathrm{mix}}$. Adding these two terms yields the stated result.

Worked Example: Combining Two Predictive Distributions

At a specified time after dosing, suppose two hypothetical models give normal predictive distributions for concentration: model 1 has mean $3.679$ mg/L and standard deviation $0.800$ mg/L, while model 2 has mean $6.065$ mg/L and standard deviation $1.200$ mg/L. These standard deviations include the predictive variability intended for this target. Give both distributions weight $0.5$.

The mixture mean is $\mu_{\mathrm{mix}}=0.5(3.679)+0.5(6.065)=4.872$ mg/L. The within-model variance contribution is $0.5(0.800^2)+0.5(1.200^2)=0.5(0.640)+0.5(1.440)=1.040\ (\mathrm{mg/L})^2$.

The between-model contribution is $0.5(3.679-4.872)^2+0.5(6.065-4.872)^2=1.424\ (\mathrm{mg/L})^2$, with rounding. Thus $\sigma_{\mathrm{mix}}^2=1.040+1.424=2.464\ (\mathrm{mg/L})^2$, and the mixture standard deviation is $\sqrt{2.464}=1.570$ mg/L.

The component distributions remain normal, but their equal-weight mixture is not generally normal. In particular, a predictive interval for the mixture should be obtained from its mixture distribution, not by assuming that the mixture is normal with the calculated mean and standard deviation.

Compartment Structures and Incompatible Parameter Vectors

Parameter averaging is particularly problematic when candidate models have different structures or different parameter dimensions. A one-compartment model and a two-compartment model do not have matching structural parameter vectors that can be averaged component by component. Even where some parameters share names, such as clearance, their roles and associated predictions depend on the full model structure.

For a two-compartment intravenous bolus model under a standard biexponential representation, concentration can be written $C(t)=A e^{-\alpha t}+B e^{-\beta t}$, with positive coefficients and rate constants satisfying $\alpha>\beta>0$. This form has two exponential phases. It should not be confused with the one-compartment expression, which has one exponential term.

Worked Example: Combining One- and Two-Compartment Predictions

Consider two hypothetical models predicting concentration 10 h after a bolus dose. The one-compartment model has $\mathrm{Dose}/V=10$ mg/L and $k=0.10\ \mathrm{h}^{-1}$, so $C_{\mathrm{1C}}(10)=10e^{-1}=3.679$ mg/L. The two-compartment model has $A=6$ mg/L, $\alpha=0.50\ \mathrm{h}^{-1}$, $B=4$ mg/L, and $\beta=0.05\ \mathrm{h}^{-1}$. These are hypothetical, positive biexponential parameters with $\alpha>\beta>0$.

Its prediction is $$ C_{\mathrm{2C}}(10)=6e^{-(0.50)(10)}+4e^{-(0.05)(10)} =6e^{-5}+4e^{-0.5} =0.0404+2.4261 =2.467\ \mathrm{mg/L}. $$

With weights $0.5$ for each model, the combined point prediction is $0.5(3.679)+0.5(2.467)=3.073$ mg/L. This is a weighted prediction across two distinct structures. Averaging their structural parameters is not a meaningful alternative here: the one-compartment rate constant and the two-compartment coefficients and rate constants do not form a common parameter vector. For predictive uncertainty, each point prediction would be replaced by its corresponding model-specific predictive distribution before mixing.

Intervals, Diagnostics, and Alternatives

Because the mixture CDF is $F_{\mathrm{mix}}(y)=\sum_m w_mF_m(y)$, a mixture quantile is found by solving $\sum_m w_mF_m(y)=q$ for the desired probability $q$. It is generally not equal to $\sum_m w_m Q_m(q)$, the weighted average of component $q$-quantiles. Similarly, a mixture predictive interval should be based on mixture quantiles or another explicitly defined mixture interval, rather than averaging interval endpoints.

Before combining predictions, check that:

Every component predicts the same outcome, time point, population or individual target, and conditioning information.
All concentrations use the same units and prediction scale. In particular, predictions on a log scale should not be treated as arithmetic-scale concentrations without an explicit transformation and target definition.
The weights are nonnegative and sum to one, and are specified independently of the outcome being evaluated if the goal is an honest external evaluation.
Each component predictive distribution includes the intended uncertainty sources consistently. A mixture cannot repair uncertainty omitted from every component.
Component predictions are numerically valid over the relevant time and covariate range. Extreme predictions or implausible tails can dominate a mixture’s clinical implications even when their weights are modest.

A useful comparison is with choosing one model. Selection gives the chosen model’s predictive distribution and is simpler to communicate, but it does not represent disagreement among candidate models. A predictive mixture retains that disagreement, but its result depends on the chosen weights and candidate set. Parameter averaging is different from both: it produces predictions from an averaged parameter vector, which may not correspond to a fitted candidate or to a meaningful structural model. The introductory material on model selection criteria concerns comparisons for selection; a predictive mixture is instead a way to combine model-specific predictions.

This tutorial treats the weights as already specified. Estimating weights from predictive scores is a separate problem. Likewise, weighting predictions does not by itself establish that the models are equally credible, eliminate model misspecification, or guarantee calibration in a new population. External calibration and transportability still need to be assessed for the combined predictions.

Common pitfall: “The models have parameter estimates, so I can average those estimates and predict from the average.” This reasoning assumes that averaging parameters preserves predictions. It generally does not for nonlinear models, and it can fail outright when structures or parameter dimensions differ. To average predictions, calculate each model’s predictive distribution for the same target and combine those distributions with valid weights.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Two models predict means of 2 mg/L and 8 mg/L for the same target. With weights $0.25$ and $0.75$, what is the mixture mean?
Why is a weighted average of component standard deviations not generally the mixture standard deviation?
For the one-compartment example above, what concentration is predicted using the weighted-average rate constant, and why does it differ from the average model prediction?
Can the 10th percentile of a mixture generally be calculated by averaging the component 10th percentiles? Explain.
What must be checked before combining a population prediction from one model with an individual prediction from another?

Solutions. (1) The mixture mean is $0.25(2)+0.75(8)=6.5$ mg/L. (2) Mixture variance includes both within-model variance and between-model differences in means; averaging standard deviations omits this decomposition and does not generally produce the square root of the mixture variance. (3) The weighted-average rate constant is $0.075\ \mathrm{h}^{-1}$, giving $10e^{-(0.075)(10)}=4.724$ mg/L. It differs from the $4.872$ mg/L weighted prediction because the exponential prediction function is nonlinear in the rate constant. (4) No. The mixture 10th percentile solves $\sum_m w_mF_m(y)=0.10$; it is not generally the weighted average of component 10th percentiles. (5) Confirm that both predictions address the same target and condition on compatible information. A population prediction and a prediction conditioned on one person’s data usually represent different targets and should not be combined without a clearly defined common target.