Model Averaging for Uncertainty-Aware Decisions

Why Average Across Models?

A decision-relevant prediction can depend on choices about structure, covariates, or other modeling assumptions. If several candidate models remain plausible, reporting only the prediction from one selected model can conceal how much conclusions depend on that selection. Model averaging offers a direct way to combine predictions while retaining differences among the models. It does not require the models to agree, and it does not make their predictions equivalent; it summarizes them using specified weights.

The weights must be chosen or supplied for the decision analysis. They might reflect a prespecified weighting scheme or results from a model-comparison procedure. The calculation below does not establish that a particular weighting scheme is correct. It shows what follows once the candidate models and their weights have been specified.

Definition: Suppose there are $M$ candidate models, and model $m$ gives a predictive distribution $F_m(y)$ for the same outcome $Y$, on the same scale and for the same population and prediction horizon. A weighted model-averaged predictive distribution is $$ F_{\mathrm{mix}}(y)=\sum_{m=1}^{M} w_m F_m(y), $$ where $w_m\geq 0$ and $\sum_{m=1}^{M} w_m=1$. Equivalently, its density, when densities exist, is $f_{\mathrm{mix}}(y)=\sum_{m=1}^{M}w_m f_m(y)$. The weights describe the contribution of each model to this specified mixture; they are not, by themselves, proof that one model is the true data-generating process.

A useful interpretation is a two-stage predictive draw: first draw a model label $M^*$ with $P(M^*=m)=w_m$, then draw an outcome from that model’s predictive distribution. This constructs a mixture distribution. It is not the same as averaging model parameters and then predicting from a single model with those averaged parameters. That operation may produce a different distribution, or may not correspond to a valid model at all.

Assumptions and Basic Calculations

Before combining predictions, check that each component represents the same decision-relevant quantity. For example, predicted response at week 12 in the target population should not be mixed with response at week 24 in a different population. Each component should include the uncertainty intended for the decision, such as predictive variability and relevant parameter uncertainty. If an input is shared across candidate models, its dependence with other uncertain inputs should be handled consistently within the component predictions; otherwise, the mixture can combine incomparable or internally inconsistent uncertainty summaries.

For a discrete event $A$, such as meeting a development target, the mixture probability follows directly from the distribution definition:

P_{\mathrm{mix}}(A)=\sum_{m=1}^{M}w_m P_m(A).

This is an average of model-specific event probabilities, not the probability obtained by averaging point predictions and checking whether that average crosses the target. Those procedures answer different questions.

If model $m$ has predictive mean $\mu_m$ and predictive variance $\sigma_m^2$, the mixture mean and variance are:

\mu_{\mathrm{mix}}=\sum_{m=1}^{M}w_m\mu_m, \qquad \sigma_{\mathrm{mix}}^2 =\sum_{m=1}^{M}w_m\left[\sigma_m^2+(\mu_m-\mu_{\mathrm{mix}})^2\right].

The variance contains both average within-model variance and a between-model term. This decomposition is useful because it makes explicit when disagreement in model means adds predictive uncertainty.

Justification: Let $M^*$ denote the random model label in the mixture construction. Conditional on $M^*=m$, the outcome has mean $\mu_m$ and variance $\sigma_m^2$. The law of total variance gives $$ \operatorname{Var}(Y)=E[\operatorname{Var}(Y\mid M^*)]+\operatorname{Var}(E[Y\mid M^*]). $$ The first term is $\sum_m w_m\sigma_m^2$. The second is $\sum_m w_m(\mu_m-\mu_{\mathrm{mix}})^2$, since $E[Y\mid M^*=m]=\mu_m$ and $E[Y]=\mu_{\mathrm{mix}}$. Adding the terms gives the stated variance. The between-model term is positive when at least two positive-weight models have different means; it is zero if all positive-weight models have the same mean, or if only one model has positive weight.

Worked Example: Combining Predictive Means and Variances

Two hypothetical models predict the same outcome, measured in response units, at the same time point. Model 1 has weight $0.70$, mean $10$ units, and variance $4$ units squared. Model 2 has weight $0.30$, mean $14$ units, and variance $9$ units squared. The weights are nonnegative and sum to one, and the predictions concern the same outcome and target population, so they can be combined.

The mixture mean is $$ \mu_{\mathrm{mix}}=0.70(10)+0.30(14)=7+4.2=11.2\text{ units}. $$

The within-model contribution to variance is $$ 0.70(4)+0.30(9)=2.8+2.7=5.5\text{ units}^2. $$

The between-model contribution is $$ 0.70(10-11.2)^2+0.30(14-11.2)^2 =0.70(1.44)+0.30(7.84) =1.008+2.352 =3.36\text{ units}^2. $$

Therefore, $$ \sigma_{\mathrm{mix}}^2=5.5+3.36=8.86\text{ units}^2, \qquad \sigma_{\mathrm{mix}}=\sqrt{8.86}=2.977\text{ units}. $$

A calculation using only the weighted component variances would give $5.5\text{ units}^2$, missing $3.36\text{ units}^2$ of between-model variation. That omission understates variance here because both models have positive weight and different means. It would not necessarily understate variance if the between-model term were zero.

Even when every component is normal, a mixture need not be normal. For instance, components with separated means can produce a distribution with multiple peaks. Therefore, the mixture mean and variance alone may not describe its shape adequately. Inspecting or reporting the full mixture distribution can matter when decisions depend on tails or thresholds.

Target Probabilities Under Model Averaging

For a development target, calculate the target probability within each model and then weight those probabilities. This preserves the model-specific predictive distributions, including their spread and shape. A target probability summarizes the chance of meeting a defined criterion under the specified predictive mixture; it is not a statement that the target will be achieved with certainty if the probability is high.

Worked Example: Probability of Meeting a Response Target

Consider two hypothetical empirical predictive distributions for response, in percentage points. Each distribution assigns equal probability to its four listed outcomes. Model A has weight $0.60$ and predicts $16, 20, 24,$ or $28$ percentage points. Model B has weight $0.40$ and predicts $12, 18, 22,$ or $30$ percentage points. The target is at least $20$ percentage points.

Under Model A, three of four equally likely outcomes meet the target, so $P_A(Y\geq20)=3/4=0.75$. Under Model B, two of four meet it, so $P_B(Y\geq20)=2/4=0.50$. Thus, $$ P_{\mathrm{mix}}(Y\geq20) =0.60(0.75)+0.40(0.50) =0.45+0.20 =0.65. $$

The model-averaged predictive probability of reaching at least 20 percentage points is $0.65$, or 65%. For comparison, the component means are $22$ percentage points for Model A and $20.5$ for Model B, giving a mixture mean of $0.60(22)+0.40(20.5)=21.4$ percentage points. Knowing that the average prediction is above the target does not replace the target-probability calculation: the mean alone does not determine how much predictive mass lies at or above 20.

The same calculation applies to other event definitions, provided the event is defined consistently across models. If models produce simulation draws rather than analytic probabilities, estimate each $P_m(A)$ by the fraction of model-$m$ draws satisfying $A$, then combine those estimates using the specified weights. Monte Carlo error should be considered when the estimated probabilities are based on a finite number of draws.

Action Selection Across Models and Outcomes

An action-selection probability can be defined by applying a decision rule to each model and outcome draw. For each draw, calculate the utility of every available action and select the action with the greatest utility. The weighted fraction of draws on which an action is selected summarizes how often it is optimal across the mixture. This is a measure of decision stability under the specified uncertainty representation.

It is important to distinguish that summary from the action chosen by expected utility. A decision-maker who must select one action before the uncertain outcome is known would ordinarily compare each action’s expected utility under the mixture and select the one with the larger expected utility, subject to the decision context. The action selected most often across draws need not have the largest expected utility. State which quantity is being reported.

Worked Example: Probability That Each Dose Is Optimal

Suppose two hypothetical models have weights $0.60$ and $0.40$. In each simulated state, the utility difference is defined as $\Delta U=U(\text{high dose})-U(\text{low dose})$, in utility units. Select the high dose when $\Delta U>0$, and the low dose when $\Delta U<0$. Model 1 has four equally likely states with differences $+2,+1,-1,-3$. Model 2 has five equally likely states with differences $+3,+2,+1,-1,-2$. There are no ties.

Under Model 1, the high dose is optimal in two of four states, so its model-specific selection probability is $2/4=0.50$. Under Model 2, it is optimal in three of five states, giving $3/5=0.60$. The model-averaged probability that the high dose is optimal across a draw from the mixture is $$ P_{\mathrm{mix}}(\text{high optimal}) =0.60(0.50)+0.40(0.60) =0.30+0.24 =0.54. $$

The low dose is selected in the remaining $1-0.54=0.46$ of mixture draws. To find the expected utility difference, first average within each model: Model 1 has mean difference $(2+1-1-3)/4=-0.25$ utility units, and Model 2 has mean difference $(3+2+1-1-2)/5=0.60$ utility units. The mixture expected difference is $$ 0.60(-0.25)+0.40(0.60)=-0.15+0.24=0.09\text{ utility units}. $$

Thus, the high dose has the greater expected utility in this simplified comparison, even though it is optimal in only 54% of the mixture draws. The 54% figure describes how often it wins across uncertain states and model contributions; it is not the probability that a decision-maker should choose it, nor does it by itself quantify the magnitude of gains and losses.

Diagnostics, Alternatives, and Limitations

Before communicating model-averaged results, check that the component predictions are genuinely comparable, that the weights sum to one, and that any reported event probability or action rule was applied consistently. Useful diagnostics include:

Display component predictions alongside the mixture so that a weighted average does not hide substantial disagreement.
Check whether tail probabilities, quantiles, or decision rankings change materially when weights change within plausible alternatives.
Inspect the mixture distribution rather than assuming its shape from its mean and variance, especially when components are separated or have different tail behavior.
For simulation-based estimates, verify adequate draws within each component and quantify Monte Carlo error for probabilities near important decision thresholds.
Check for identical or effectively zero-weight components, which contribute little or nothing to the mixture and can make a reported multi-model summary misleading.

Model averaging differs from selecting a single best-supported model: selection uses one component, while averaging retains weighted contributions from several. It also differs from reporting each model separately, which makes disagreement visible but does not provide one combined predictive distribution. Finally, averaging point predictions alone gives a weighted mean but omits the component variances and the between-model contribution. That may be sufficient for a narrowly defined mean-prediction summary, but it is not a complete account of predictive uncertainty.

The method has important limitations. The result depends on the candidate model set and the specified weights; omitted models cannot contribute uncertainty. A mixture does not repair misspecification in its components, and it does not make incompatible predictions comparable. If weights are derived from data, their uncertainty and the assumptions of the weighting procedure may also matter. In practice, report the weighting basis and show how decision-relevant results behave under reasonable weight changes rather than presenting the mixture as uniquely determined.

Common pitfall: “The models have different means, so omitting the between-model term must always understate variance.” This is not always correct. The between-model term is $\sum_m w_m(\mu_m-\mu_{\mathrm{mix}})^2$, and it is positive when at least two positive-weight models have different means. With weights $1$ and $0$, for example, component means can differ while the zero-weight model contributes nothing and the between-model term is zero. Check the weights as well as the means.

Key takeaway: A weighted model average is a mixture of predictive distributions, not merely an average of point predictions. It combines within-model predictive uncertainty with between-model variation and allows target probabilities and action-selection summaries to reflect the specified contributions of the candidate models.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

Two models have weights $0.25$ and $0.75$, and their probabilities of meeting a target are $0.80$ and $0.40$. What is the mixture probability?
When is the between-model variance term positive?
Why does a mixture mean above a target not establish that the probability of meeting the target is above 0.5?
Can the action selected most often across predictive draws differ from the action with greatest expected utility? Briefly explain.

Solutions. (1) $0.25(0.80)+0.75(0.40)=0.20+0.30=0.50$, so the mixture probability is 50%. (2) It is positive when at least two positive-weight models have different means; zero-weight models do not affect it. (3) A mean does not determine the distribution’s shape or the probability mass above a threshold. Different distributions can have the same mean but different target probabilities. (4) Yes. Selection frequency counts how often an action has the greatest utility across draws, whereas expected utility also reflects the sizes of gains and losses. An action can win less often but have larger gains when it does.