Model Averaging Across Integrated Evidence Chains

Why Average Integrated Evidence Chains?

An integrated prediction can depend on choices made throughout a chain of models: how exposure is generated, how exposure relates to outcome, and how those components are connected. A single selected chain can conceal uncertainty about those structural choices. Model averaging represents that uncertainty by combining predictions from plausible complete chains, rather than treating one chain as certainly correct.

The object being averaged is the prediction for a shared target: for example, exposure under a specified regimen in a defined population, or a clinical endpoint under that same scenario. The alternative chains may have different structures or parameters, but their predictions must be expressed for the same estimand, population, time point, and units. Parameters with different meanings across chains are not averaged directly.

Definition: Let $M$ denote which candidate integrated evidence chain generated a prediction, with candidate chains $M_1,\ldots,M_K$. If $w_m$ is the weight assigned to chain $M_m$, where $w_m\geq 0$ and $\sum_{m=1}^{K}w_m=1$, the model-averaged predictive distribution for a target $Y$ is the mixture $$ p(Y\mid D)=\sum_{m=1}^{K}w_m\,p(Y\mid D,M_m). $$ Here $D$ is the evidence used for inference, and each chain-specific distribution includes the uncertainty represented within that chain.

This is a mixture of predictive distributions, not an average of model structures. It allows more than one chain to contribute while preserving the distinct predictions each chain implies. The introductory material on weighting evidence from unequally reliable models motivates the role of weights; here the focus is on applying weights to complete integrated chains and their exposure and outcome predictions.

Choosing and Interpreting Weights

In Bayesian model averaging, weights are posterior probabilities for the candidate chains, conditional on the specified model set, prior probabilities, and observed evidence. For chain $M_m$, let $\pi_m=P(M_m)$ be its prior probability and let $p(D\mid M_m)$ be its marginal likelihood, integrating over that chain’s parameters. Bayes’ rule gives:

w_m=P(M_m\mid D) =\frac{p(D\mid M_m)\pi_m} {\sum_{j=1}^{K}p(D\mid M_j)\pi_j}.

This calculation requires the candidate chains to use a coherent probability model for the same observed evidence. Marginal likelihoods depend on the parameter priors, including their scale, so those priors must be defensible and proper for this calculation. The weights are conditional on the candidates considered: they do not establish that the list contains the true chain.

Predictive approaches choose weights to perform well at predicting held-out observations. One common approach, stacking, chooses nonnegative weights summing to one to maximize the sum of log mixture predictive densities for held-out data. The resulting weights are predictive combination weights; they are not posterior probabilities that the chains are true. Their interpretation depends on the validation scheme and prediction target.

Worked Example: Bayesian Weights From Marginal Likelihoods

Suppose two hypothetical complete chains, $M_1$ and $M_2$, have prior probabilities 0.5 each. For the same evidence $D$, assume their marginal likelihoods are 0.12 and 0.08. Calculate the posterior model weights.

The unnormalized weights are marginal likelihood times prior probability:

\widetilde{w}_1=0.12(0.5)=0.060,\qquad \widetilde{w}_2=0.08(0.5)=0.040.

Their sum is 0.100, so normalization gives $w_1=0.060/0.100=0.600$ and $w_2=0.040/0.100=0.400$. Conditional on this two-chain model set, the evidence, and the stated priors, the posterior probabilities are 0.600 for $M_1$ and 0.400 for $M_2$. These values are not probabilities that either chain is universally correct.

Combining Predictions and Their Uncertainty

For a numerical target, let $\mu_m=E(Y\mid D,M_m)$ and $v_m=\operatorname{Var}(Y\mid D,M_m)$. The averaged mean is the weighted mean of the chain-specific means. The predictive variance includes both the variance within each chain and the spread of the chain-specific means:

E(Y\mid D)=\sum_m w_m\mu_m, \qquad \operatorname{Var}(Y\mid D) =\sum_m w_m\left[v_m+(\mu_m-\bar{\mu})^2\right], \quad \bar{\mu}=\sum_m w_m\mu_m.

The second expression follows from conditioning on the chain. Write $E(Y\mid D,M)=\mu_M$ and $\operatorname{Var}(Y\mid D,M)=v_M$. The identity $\operatorname{Var}(Y\mid D)=E[\operatorname{Var}(Y\mid D,M)\mid D]+\operatorname{Var}(E[Y\mid D,M]\mid D)$ then gives the sum of the weighted within-chain variances and the weighted squared deviations of the chain means from $\bar{\mu}$. Thus, predictions can be uncertain even if each chain is individually precise, when their predicted means differ.

Worked Example: Averaging Exposure and Outcome Predictions

Use the hypothetical Bayesian weights $w_1=0.600$ and $w_2=0.400$. For a specified regimen and population, chain $M_1$ predicts exposure with mean 10 mg/L and predictive standard deviation 2 mg/L; chain $M_2$ predicts mean 14 mg/L and standard deviation 3 mg/L. For a clinical endpoint measured in points, their predictive means and standard deviations are 5 and 1 point for $M_1$, and 8 and 1.5 points for $M_2$. Calculate the mixture means and standard deviations.

For exposure, the averaged mean is $0.600(10)+0.400(14)=11.6$ mg/L. The chain-specific variances are $2^2=4$ and $3^2=9$ $(\mathrm{mg/L})^2$. Therefore:

\begin{aligned} \operatorname{Var}(E\mid D) &=0.600[4+(10-11.6)^2]+0.400[9+(14-11.6)^2]\\ &=0.600(6.56)+0.400(14.76)\\ &=9.84\;(\mathrm{mg/L})^2. \end{aligned}

The model-averaged exposure standard deviation is $\sqrt{9.84}=3.14$ mg/L. For the clinical endpoint, the averaged mean is $0.600(5)+0.400(8)=6.20$ points. Its variance is:

\begin{aligned} \operatorname{Var}(Y\mid D) &=0.600[1^2+(5-6.20)^2]+0.400[1.5^2+(8-6.20)^2]\\ &=0.600(2.44)+0.400(5.49)\\ &=3.66\;\mathrm{points}^2. \end{aligned}

The averaged endpoint standard deviation is $\sqrt{3.66}=1.91$ points. Both standard deviations include within-chain predictive uncertainty and disagreement between chain means. They summarize mixture variability; they do not imply that the mixture distribution is normal. If the exposure and endpoint predictions are to be used jointly, combine paired joint draws from each chain and select the chain for each draw using the weights. Separately mixing their marginal predictions does not preserve their dependence.

Predictive Weights From Held-Out Data

For stacking, let $y_i$ be held-out observation $i$, and let $p_m(y_i\mid D_{-i})$ be chain $m$'s predictive density for it, fitted without that observation. Stacking chooses weights to maximize:

\ell(w)=\sum_i \log\left(\sum_m w_m p_m(y_i\mid D_{-i})\right), \qquad w_m\geq 0,\quad \sum_m w_m=1.

The held-out predictions must correspond to the same observations and prediction task. In practice, out-of-sample predictions may be generated using cross-validation or a separate validation set. Reusing fitted predictions for observations that were used to fit the chains can give an optimistic assessment of predictive performance.

Worked Example: Two-Chain Stacking Weights

Consider two hypothetical held-out observations. Chain A assigns predictive densities 0.8 and 0.3; chain B assigns 0.2 and 0.7. Let $w$ be the weight on A, so B receives $1-w$. The stacked predictive densities are $0.2+0.6w$ and $0.7-0.4w$. Find the weight that maximizes their summed log density.

The objective and its derivative are:

\ell(w)=\log(0.2+0.6w)+\log(0.7-0.4w), \qquad \ell'(w)=\frac{0.6}{0.2+0.6w}-\frac{0.4}{0.7-0.4w}.

Setting the derivative to zero gives $0.6(0.7-0.4w)=0.4(0.2+0.6w)$, hence $0.42-0.24w=0.08+0.24w$ and $w=0.7083$. This is feasible, and the second derivative is negative throughout the interval $0\leq w\leq1$, so it is the maximum. At this weight the mixture densities are $0.625$ and $0.4167$, and the objective is $\log(0.625)+\log(0.4167)\approx-1.345$. At the endpoints the objectives are $\log(0.8)+\log(0.3)\approx-1.427$ for A alone and $\log(0.2)+\log(0.7)\approx-1.966$ for B alone. For these held-out predictions, the mixture achieves the largest summed log score. The weight 0.7083 is a predictive weight, not a posterior probability for chain A.

Conditions, Diagnostics, and Limitations

Before averaging, check that each candidate chain predicts the same target under the same scenario and uses the evidence coherently. In particular, shared observations should not inadvertently contribute more than once to the integrated evidence. The earlier material on preventing double counting of shared data addresses that dependency concern. Chain-specific predictive distributions should also carry forward the uncertainty represented by their linked components, rather than treating estimated inputs as fixed without justification.

Check the target and scale. If one chain predicts concentration at a different time point, population, or dose than another, their predictions are not directly mixable until they are defined for a common target.
Inspect chain-specific predictions. Large differences in exposure or outcome predictions can make the mixture broad or multimodal. A single mean and standard deviation may then conceal important features of the predictive distribution.
Check weight stability and predictive fit. Bayesian weights can be sensitive to prior choices and the candidate model set. Predictive weights can be sensitive to the validation observations and the validation design. Report the method and basis for the weights.
Preserve joint draws when needed. Exposure and outcome may be dependent within a chain. Mixing marginal distributions separately does not provide a joint distribution suitable for downstream simulation.

A common mistake is to average chain parameters, or to treat stacking weights as the probability that each chain is true. The correction is to average predictions on a common target scale, using weights whose interpretation matches how they were obtained. Another mistake is to report only the weighted mean: if the chains disagree, that omits a potentially substantial part of the predictive uncertainty.

Model averaging is not a remedy for a set of inadequate candidate chains. It also does not remove uncertainty about unconsidered structures or guarantee calibrated predictions. A single selected chain is simpler to communicate and may be reasonable when alternatives are unsupported or operationally irrelevant, but selection hides between-chain disagreement. Averaging can better represent structural uncertainty, at the cost of dependence on the candidate set, weighting method, and prediction target.

Check Your Understanding

Answer each question, then compare with the solutions below.

Two chains have equal prior probability and marginal likelihoods 0.15 and 0.05. What are their Bayesian posterior weights?
Two chains predict a target with means 4 and 10 units, each with within-chain variance 1 unit squared. Their weights are 0.75 and 0.25. Calculate the mixture mean and variance.
Why should a parameter with different meanings in two chains generally not be averaged directly?
Are stacking weights posterior probabilities that the corresponding chains are true? State what they represent.

Solutions. (1) The unnormalized weights are $0.15(0.5)=0.075$ and $0.05(0.5)=0.025$, which normalize to 0.75 and 0.25. (2) The mean is $0.75(4)+0.25(10)=5.5$ units. The variance is $0.75[1+(4-5.5)^2]+0.25[1+(10-5.5)^2]=0.75(3.25)+0.25(21.25)=7.75$ units squared. (3) Different meanings make a numerical average lack a common interpretation; predictions for a shared estimand can instead be averaged. (4) No. Stacking weights are chosen to optimize predictive performance under the specified held-out scoring procedure; they are not probabilities that chains are true.