Model Averaging For GLM Predictions

Why Average Predictions?

A model-selection rule chooses one candidate GLM, often the one with the smallest AIC or the best validation score. But when several models predict similarly, treating the leading model as certain can make predictions sensitive to small changes in the data. Model averaging offers another approach: combine predictions from a prespecified set of fitted models, giving more weight to models with stronger support.

The aim here is prediction, not a claim that the candidate models are all true or that their coefficients can be pooled into one meaningful parameter vector. For a GLM, a candidate’s fitted linear predictor is transformed through its link function to give a predicted mean. Because this transformation is generally nonlinear, averaging the predicted means is not the same as averaging the linear predictors or coefficients.

Definition: Suppose there are $M$ candidate GLMs, each fitted to the same observed data, and let $\hat{\mu}_m(x)$ be candidate $m$'s predicted conditional mean at covariates $x$. A prediction average is $$ \hat{\mu}_{\mathrm{avg}}(x)=\sum_{m=1}^{M} w_m\hat{\mu}_m(x), $$ where $w_m\geq 0$ and $\sum_{m=1}^{M}w_m=1$. The weights define a convex combination of the candidate predictions.

A convex combination remains within the range of the component predictions at each $x$. It does not, however, ensure that the average is better than every candidate on new data. That depends on the candidate set, the weights, and how well the data used to choose those weights represent the prediction problem.

AIC Weights

For candidate $m$, let $\mathrm{AIC}_m=-2\ell_m+2k_m$, where $\ell_m$ is its maximized log-likelihood and $k_m$ is its number of estimated parameters. AIC weights convert relative AIC values into nonnegative weights that sum to one. Define $\Delta_m=\mathrm{AIC}_m-\min_j\mathrm{AIC}_j$. The standard weights are

w_m^{\mathrm{AIC}}= \frac{\exp(-\Delta_m/2)} {\sum_{j=1}^{M}\exp(-\Delta_j/2)}.

These weights are meaningful as relative support within the candidate set, under the AIC predictive-comparison rationale. They are not posterior probabilities that the models are true. The candidate models should be fitted to the same response data, with likelihoods defined for the same observations and outcome. If models use different response data or incomparable likelihood conventions, their AIC values should not be put into this formula without justification.

Worked Example: AIC-Weighted GLM Predictions

Three candidate GLMs have AIC values $100$, $101$, and $104$. At a particular covariate value, their predicted means are $2$, $3.5$, and $6$. Calculate the AIC weights and average prediction, then compare with choosing the minimum-AIC model.

The minimum AIC is $100$, so the differences are $0$, $1$, and $4$. The unnormalized weights are $1$, $\exp(-1/2)$, and $\exp(-2)$. Numerically,

1+\exp(-1/2)+\exp(-2) =1+0.60653066+0.13533528 =1.74186594\ldots

Thus the normalized weights are approximately

w_1=0.5741,\qquad w_2=0.3482,\qquad w_3=0.0777.

They sum to $1.0000$ to the displayed precision. The averaged mean is

\begin{aligned} \hat{\mu}_{\mathrm{avg}} &=\frac{1(2)+\exp(-1/2)(3.5)+\exp(-2)(6)} {1+\exp(-1/2)+\exp(-2)}\\ &=\frac{2+2.12285731+0.81201168}{1.74186594}\\ &\approx 2.8331. \end{aligned}

Selecting only the minimum-AIC model gives a prediction of $2$. Averaging moves the prediction toward the other candidates, especially the second model, while the third has relatively little influence. The result is a prediction at this covariate value; it does not establish that the average is closer to the unknown conditional mean.

The weights are unchanged if the same constant is added to every AIC: each $\Delta_m$ remains the same. This matters because the formula depends on relative, not absolute, AIC values.

Result: AIC weights are nonnegative, sum to one, and are invariant to a common additive shift in all candidate AIC values. Indeed, $\Delta_m\geq 0$ implies $\exp(-\Delta_m/2)>0$, and dividing each positive quantity by their positive sum gives nonnegative weights summing to one. If each AIC is replaced by $\mathrm{AIC}_m+c$, then $$ (\mathrm{AIC}_m+c)-\min_j(\mathrm{AIC}_j+c) =\mathrm{AIC}_m-\min_j\mathrm{AIC}_j =\Delta_m, $$ so every weight is unchanged.

Validation-Based Weights

A second approach chooses weights to perform well on data held out from the fitting process. Let $y_i$ be validation outcomes and let $\hat{\mu}_{im}$ be the prediction for validation case $i$ from candidate $m$. For a loss function $L(y,\hat{\mu})$, validation-based averaging can choose weights in the simplex $$ \mathcal{W}=\{(w_1,\ldots,w_M):w_m\geq 0,\ \sum_m w_m=1\} $$ to minimize $$ \sum_{i=1}^{n_{\mathrm{val}}}L\left(y_i,\sum_{m=1}^{M}w_m\hat{\mu}_{im}\right). $$ This is often called stacking when weights are chosen by optimizing predictive performance of the combined predictions. For squared-error loss, it targets the predicted mean. Other outcomes require an appropriate scoring rule and a compatible predictive quantity.

For fixed validation predictions and squared-error loss, the objective is a continuous function of the weights. The simplex is a closed and bounded subset of finite-dimensional space, hence compact, so the objective attains a minimum. The minimum need not be unique: for example, two candidates with identical validation predictions cannot be distinguished by this criterion.

Worked Example: Weights from Validation Squared Error

Two candidate models predict the outcomes of two validation cases. The observed outcomes are $0$ and $1$. Model A predicts $0$ and $1.4$, while model B predicts $0.8$ and $0.8$. Let $w$ be the weight on A and $1-w$ the weight on B, with $0\leq w\leq 1$. Find the validation-MSE-minimizing weight.

The two averaged predictions are $0.8(1-w)$ and $1.4w+0.8(1-w)=0.8+0.6w$. The mean squared error is therefore

\begin{aligned} Q(w) &=\frac{[0.8(1-w)-0]^2+[0.8+0.6w-1]^2}{2}\\ &=\frac{0.64(1-w)^2+(-0.2+0.6w)^2}{2}. \end{aligned}

Differentiating gives $Q'(w)=[-1.28(1-w)+1.2(-0.2+0.6w)]/2$, which simplifies to $Q'(w)=-0.76+w$. The second derivative is $Q''(w)=1>0$, so the unconstrained minimum is the unique minimum; $w=0.76$ lies in the permitted interval. The averaged predictions are $0.8(0.24)=0.192$ and $0.8+0.6(0.76)=1.256$. Hence

Q(0.76)=\frac{0.192^2+(-0.256)^2}{2} =\frac{0.036864+0.065536}{2} =0.0512.

For comparison, model A alone has MSE $[0^2+(1.4-1)^2]/2=0.08$, and model B alone has MSE $[0.8^2+(0.8-1)^2]/2=0.34$. The optimized average does better on these validation cases than either model alone. That is a result on this validation set, not a guarantee of lower error on future cases.

The validation data must play a clear role. If they are used to choose the weights, reporting their optimized loss as an unbiased assessment of future performance is generally optimistic: the weights were chosen to fit those outcomes. Use separate test data, or a properly designed resampling procedure, for an additional performance assessment when such an assessment is needed.

What Averaging Guarantees—and What It Does Not

For squared-error loss, convexity gives a useful pointwise inequality. Fix an outcome $y$, candidate predictions $a_1,\ldots,a_M$, and convex weights $w_m$. Then

\left(y-\sum_{m=1}^{M}w_ma_m\right)^2 \leq \sum_{m=1}^{M}w_m(y-a_m)^2.

To prove it, write $y-\sum_mw_ma_m=\sum_mw_m(y-a_m)$. The function $z\mapsto z^2$ is convex, so Jensen's inequality gives

\left(\sum_mw_m(y-a_m)\right)^2 \leq \sum_mw_m(y-a_m)^2.

This shows the average prediction's squared error is no greater than the weighted average of the component squared errors, for the same outcome and fixed weights. It does not imply that the average beats the best individual candidate: a weighted average of losses can exceed the smallest loss. Nor does it guarantee an improvement in the population prediction risk when the weights were estimated from limited data.

Worked Example: Averaging Predictive Distributions

At one covariate value, suppose two candidate models give Poisson predictive distributions with means $1$ and $5$, and assign them equal weights. The average predicted mean is

\hat{\mu}_{\mathrm{avg}}=0.5(1)+0.5(5)=3.

If the full predictive distribution is averaged as a mixture, its probability of zero is

P_{\mathrm{mix}}(Y=0) =0.5e^{-1}+0.5e^{-5} \approx 0.5(0.367879)+0.5(0.006738) =0.187309.

A single Poisson distribution with mean $3$ instead has $P(Y=0)=e^{-3}\approx0.049787$. Thus, matching the mixture's mean with a Poisson distribution does not reproduce its predictive probabilities. The mixture variance is $7$: the component variances are $1$ and $5$, their average is $3$, and the variance of their means around $3$ is $0.5(1-3)^2+0.5(5-3)^2=4$. Consequently, the total variance is $3+4=7$, rather than the variance $3$ of a Poisson distribution with mean $3$. When uncertainty about the predictive distribution matters, average the predictive distributions themselves, not just their means.

Practical Interpretation and Common Pitfalls

For GLMs, a mean prediction is typically formed by applying each model's inverse link to its own fitted linear predictor, then averaging on the response scale. For example, with a log link, a weighted average of candidate means $\exp(\hat{\eta}_m)$ is generally different from $\exp(\sum_mw_m\hat{\eta}_m)$. The latter is a geometric-type combination of means, not the response-scale arithmetic average defined here. Coefficient averaging is still less generally interpretable when candidate models use different predictors or parameterizations.

Mistake: treating AIC weights as probabilities that the models are true. They are normalized relative-support weights within the stated candidate set, not posterior model probabilities.
Mistake: averaging link-scale predictions when the target is a response-scale mean. Apply each model's inverse link first, then combine the resulting means.
Mistake: assuming an average must outperform the selected model. Convexity gives a bound relative to the weighted component losses, not necessarily the best component loss or future risk.
Mistake: evaluating optimized validation weights on the same cases as if the score were untouched. Weight optimization uses information in those outcomes; independent assessment needs data not used to choose the weights, or an appropriate resampling design.

Model averaging is most useful when the candidate set represents plausible alternatives and the prediction target is clearly specified. It can reduce dependence on a single selection decision, while retaining the need to assess calibration, predictive performance, and the consequences of the chosen loss or scoring rule.

Check Your Understanding

Use these questions to check both the calculations and the interpretation of model-averaged predictions.

Two GLMs have AIC values $50$ and $52$. Write their unnormalized AIC weights and explain why the second is smaller.
Why are AIC weights not posterior probabilities that the candidate models are true?
For two candidate means $2$ and $8$ with weights $0.75$ and $0.25$, calculate their response-scale average. Why might averaging their linear predictors give a different value?
State the squared-error inequality for a convex combination of predictions. Does it guarantee that the average beats the best candidate?
Why can a mixture of Poisson predictive distributions differ from a single Poisson distribution with the mixture's mean?