Bayesian Model Averaging for Individual Forecasts

Why Average Individual Forecasts Across Models?

An individual PK forecast depends not only on that person's parameters and observations, but also on the structural model used to describe concentration over time. For example, plausible candidates might differ in their absorption structure or in whether a particular disposition component is represented. Sparse individual data may not clearly distinguish these structures, even when they imply meaningfully different future concentrations. Choosing one model and treating it as certain can hide that uncertainty.

Bayesian model averaging carries uncertainty about the structure into the forecast. It assigns each candidate model a posterior probability given the individual's data, forms a forecast under each model, and combines those forecasts using the posterior probabilities as weights. The resulting distribution reflects both uncertainty within each model and uncertainty about which model is appropriate.

Definition: Let $D_i$ denote the PK and, when relevant, PD observations available for individual $i$. Let $M_1,\ldots,M_K$ be a set of candidate structural models, with prior probabilities $P(M_k)$. The posterior model probability is $P(M_k\mid D_i)$. For a future quantity $Y_i^*$, Bayesian model averaging defines its individual predictive distribution as the mixture of the model-specific predictive distributions, weighted by $P(M_k\mid D_i)$.

Posterior Model Probabilities

Within candidate model $M_k$, let $\theta_k$ denote its parameters, including individual parameters that are uncertain after observing $D_i$. The evidence for $M_k$ is the probability of the observed data integrated over the model's parameter distribution:

p(D_i\mid M_k)=\int p(D_i\mid\theta_k,M_k)\,p(\theta_k\mid M_k)\,d\theta_k.

Bayes' rule updates the prior model probabilities using these evidences:

P(M_k\mid D_i)= \frac{p(D_i\mid M_k)P(M_k)} {\sum_{j=1}^{K}p(D_i\mid M_j)P(M_j)}.

The evidence averages the likelihood over the parameter prior; it is not the likelihood evaluated only at a best-fitting parameter value. Thus, posterior model probabilities depend on both how well a model can explain the observed data and how much prior parameter space provides that explanation. Candidate models must use comparable data and likelihood definitions, and their parameter priors must be proper for ordinary evidence-based comparisons. The prior model probabilities should also be stated: equal probabilities are a choice, not a requirement.

The Individual Predictive Mixture

For each model, first obtain the forecast conditional on that model and the individual's data. This forecast integrates over the posterior parameter uncertainty within the model. The overall forecast is then:

p(Y_i^*\mid D_i)= \sum_{k=1}^{K}P(M_k\mid D_i)\,p(Y_i^*\mid D_i,M_k),

where $p(Y_i^*\mid D_i,M_k)$ itself averages over $p(\theta_k\mid D_i,M_k)$ and includes relevant future residual or process variability. If the target is a latent concentration rather than a future measured concentration, measurement error is not automatically part of the target; the forecast must match the quantity the dosing question actually concerns.

The mixture is a distribution, not merely a weighted average of model-specific point predictions. Its mean is the weighted mean of the model-specific means. Its variance includes two components: the average predictive variance within models and the variance of the model-specific means. The latter captures how much the forecast changes across candidate structures.

Theorem: If $\mu_k=E(Y_i^*\mid D_i,M_k)$ and $v_k=\operatorname{Var}(Y_i^*\mid D_i,M_k)$, then, writing $w_k=P(M_k\mid D_i)$ and $\mu=\sum_k w_k\mu_k$, $$ E(Y_i^*\mid D_i)=\mu,\qquad \operatorname{Var}(Y_i^*\mid D_i)= \sum_{k=1}^{K}w_kv_k+ \sum_{k=1}^{K}w_k(\mu_k-\mu)^2. $$

To justify the variance formula, condition first on the model. The identity $\operatorname{Var}(Y_i^*\mid D_i)=E[\operatorname{Var}(Y_i^*\mid D_i,M)\mid D_i]+\operatorname{Var}(E[Y_i^*\mid D_i,M]\mid D_i)$ separates variation within models from variation in their conditional means. Since $M$ takes value $M_k$ with probability $w_k$, the first term is $\sum_k w_kv_k$, and the second is $\sum_k w_k(\mu_k-\mu)^2$. Omitting the second term understates forecast uncertainty whenever the candidate models imply different means.

Worked Example: Combining Two Model-Specific Concentration Forecasts

Suppose two hypothetical structural models have prior probabilities 0.60 and 0.40. For one individual's observed data, their evidences are 0.12 and 0.06, respectively. Both evidences are calculated for the same observations and with proper parameter priors. Under $M_1$, the future concentration forecast has mean 8.0 mg/L and variance 4.0 $(\mathrm{mg/L})^2$; under $M_2$, it has mean 12.0 mg/L and variance 9.0 $(\mathrm{mg/L})^2$.

The unnormalized posterior model weights are $0.60(0.12)=0.072$ and $0.40(0.06)=0.024$. Their sum is 0.096, so the normalized weights are:

w_1=\frac{0.072}{0.096}=0.75,\qquad w_2=\frac{0.024}{0.096}=0.25.

The model-averaged forecast mean is $0.75(8.0)+0.25(12.0)=9.0$ mg/L. Its within-model variance contribution is $0.75(4.0)+0.25(9.0)=5.25\;(\mathrm{mg/L})^2$. Its between-model contribution is $0.75(8.0-9.0)^2+0.25(12.0-9.0)^2=3.00\;(\mathrm{mg/L})^2$. Therefore, the total predictive variance is $5.25+3.00=8.25\;(\mathrm{mg/L})^2$, and the predictive standard deviation is $\sqrt{8.25}=2.872$ mg/L. Reporting only the variance within models would miss the extra 3.00 $(\mathrm{mg/L})^2$ associated with model choice.

Probabilities and Decisions From the Mixture

Probabilities of clinically relevant forecast events must also be averaged across models. For an event $A$, such as exceeding a concentration threshold, the law of total probability gives $P(A\mid D_i)=\sum_k w_kP(A\mid D_i,M_k)$. A probability computed from the mixture is generally not the same as evaluating the event at the averaged forecast mean. In particular, a mean concentration alone does not determine the probability of exceeding a limit.

Worked Example: Probability of Exceeding a Concentration Threshold

Use the posterior weights 0.75 and 0.25 from the preceding hypothetical example. Suppose the model-specific predictive distributions are $N(8,4)$ mg/L under $M_1$ and $N(12,9)$ mg/L under $M_2$, where the second argument is the variance. Calculate the probability that the future concentration exceeds 10 mg/L.

Under $M_1$, the standard deviation is 2 mg/L, so the standardized threshold is $(10-8)/2=1$. Thus $P(Y_i^*>10\mid D_i,M_1)=1-\Phi(1)=0.1587$. Under $M_2$, the standard deviation is 3 mg/L, and the standardized threshold is $(10-12)/3=-0.6667$. Thus $P(Y_i^*>10\mid D_i,M_2)=1-\Phi(-0.6667)$, approximately 0.7475.

P(Y_i^*>10\mid D_i) =0.75(0.1587)+0.25(0.7475) =0.3059.

The model-averaged probability of exceeding 10 mg/L is about 0.306. This is not the probability obtained by treating the 9.0 mg/L mixture mean as a known concentration. It is a probability over the full predictive mixture, including its model-specific spread.

For a predictive interval or other distributional summary, use the mixture distribution itself. In general, an interval formed by averaging the endpoints of model-specific intervals will not have the desired coverage under the mixture. Likewise, an average of model-specific quantiles need not be a quantile of the mixture. Numerical sampling is often convenient: draw a model according to the posterior weights, draw parameters from that model's individual posterior, then draw the forecast conditional on those parameters. Summaries of the resulting draws approximate the mixture's summaries.

Worked Example: Updating Model Weights With Another Individual Observation

Suppose the current posterior model probabilities for an individual are 0.75 for $M_1$ and 0.25 for $M_2$. A further hypothetical concentration observation is available. Its predictive density under $M_1$, integrating over the current parameter posterior, is 0.20 per mg/L; under $M_2$, it is 0.80 per mg/L. These densities refer to the same observed concentration and measurement units, so their relative values can update the model weights.

The updated unnormalized weights are $0.75(0.20)=0.15$ and $0.25(0.80)=0.20$. Normalizing by their sum, 0.35, gives:

P(M_1\mid D_i,\text{new observation})=\frac{0.15}{0.35}=0.4286,\qquad P(M_2\mid D_i,\text{new observation})=\frac{0.20}{0.35}=0.5714.

Suppose that, after updating within each model as well, the forecast means for a future concentration are 7.0 mg/L under $M_1$ and 11.0 mg/L under $M_2$. The updated model-averaged mean is $0.4286(7.0)+0.5714(11.0)=9.286$ mg/L. The updated forecast therefore reflects both the new evidence favoring $M_2$ and the revised individual forecasts within each model. Updating only the weights while leaving model-specific parameter forecasts stale would not, in general, be a complete Bayesian update.

Assumptions, Diagnostics, and Alternatives

The calculation is meaningful only relative to its candidate set and assumptions. The models must address the same forecast target and observed data, and the likelihoods and priors must be specified coherently. A model probability is conditional on the candidate models, their parameter priors, and the prior model probabilities; it is not an absolute probability that a model is true. Before relying on a forecast, check that the candidate structures are scientifically plausible, that each model's parameter posterior and predictive computation are stable, and that the resulting model weights and forecasts are not driven by an unnoticed coding or scaling difference.

Useful warning signs include one model receiving nearly all posterior weight despite small changes in reasonable prior choices, materially different forecasts under models with similar weights, and substantial disagreement between the candidate models over the target quantity. These do not automatically invalidate averaging, but they call for transparent reporting and scrutiny of the model assumptions and evidence calculations. If every candidate structure inadequately represents the available PK information, averaging them does not repair that inadequacy. Model averaging expresses uncertainty among the models considered; it cannot assign weight to an omitted structure.

The main alternative is to select a single model, for example the model with the largest posterior probability, and forecast conditional on it. Selection is simpler to communicate and can be reasonable when one structure is strongly supported and the decision is not sensitive to the remaining uncertainty. But a selected-model interval omits uncertainty about the selection itself. A model-averaged forecast retains that uncertainty, at the cost of depending on the candidate set and prior specifications. Neither approach removes uncertainty in individual parameters, which must still be propagated within each model.

Common pitfall: Averaging the model-specific means and then reporting only the selected model's predictive variance treats structural uncertainty as if it were zero. The correction is to use the full mixture: add the between-model variance to the weighted within-model variance, and calculate event probabilities and quantiles from the mixture distribution rather than from its mean alone.

For individualized dosing, model averaging can matter when plausible structures imply different exposure forecasts or different probabilities of meeting a target. Reporting posterior model weights alongside the mixture forecast makes the source of uncertainty interpretable: readers can see whether uncertainty is mainly within models, between their predictions, or both. The forecast remains conditional on the observations, model set, priors, and dosing scenario used to construct it.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

Two models have prior probabilities 0.50 each and evidences 0.08 and 0.04. What are their posterior model probabilities?
Why can a model-averaged predictive interval be wider than the intervals suggested by the within-model variances alone?
For an event $A$, model weights are 0.60 and 0.40, and the model-specific event probabilities are 0.10 and 0.50. Find the model-averaged event probability.
What does a posterior model probability of 0.90 mean, and what does it not mean?

Solutions. 1. The unnormalized weights are $0.50(0.08)=0.04$ and $0.50(0.04)=0.02$; normalizing by 0.06 gives probabilities 0.6667 and 0.3333. 2. Models with different forecast means add between-model variance to the weighted within-model variance, which can increase overall spread. 3. The mixture probability is $0.60(0.10)+0.40(0.50)=0.26$. 4. Conditional on the specified candidate set, priors, and data, the posterior probability assigned to that model is 0.90. It is not an unconditional claim that the model is true with probability 0.90, nor does it account for omitted candidate structures.