Model Averaging After Covariate Selection

Why Average Across Covariate Models?

Covariate selection often leaves several plausible models rather than one clear winner. Two models may have similar support but imply meaningfully different predictions for a patient, or different estimates of a covariate effect. Reporting only the top-ranked model hides this uncertainty: the reported prediction or effect is conditional on that particular model having been chosen.

Model averaging retains a set of plausible models and combines their results using weights. In population pharmacokinetics (PK), the results may be individual or population predictions, such as clearance or concentration at a specified time, or a covariate-effect parameter defined comparably across models. Averaging does not make an inadequate candidate set adequate, and it does not establish that any candidate model is true. Its purpose is to represent uncertainty among the models included in the analysis.

Before averaging, specify the candidate set, the quantity to average, and the weighting rule. Models should generally be fitted to the same observations with comparable likelihood definitions and the same response data; otherwise, their information criteria or likelihood evidence may not be on a common scale. The target also needs a common meaning. For example, predictions of concentration at the same dose, time, and covariate values can be averaged even when the models use different covariate terms. Raw coefficients cannot be averaged meaningfully if their scales, parameterizations, or contrasts differ.

Information-Criterion Weights

For a candidate set fitted to the same data, an information criterion balances model fit against a complexity penalty. A common choice is AIC; AICc applies a finite-sample correction and is often preferable when the sample is not large relative to the number of estimated parameters. Define \(\mathrm{IC}_k\) as the chosen criterion for model \(M_k\), with smaller values indicating better support by that criterion within the candidate set. The relative difference and normalized weight are

These weights are nonnegative and sum to one. They quantify relative support within the specified candidate set under the selected information criterion. They are not, in general, probabilities that the models are true. Adding or removing candidate models can change the normalized weights even when the fitted models themselves do not change.

Bayesian Model Averaging

Bayesian model averaging uses posterior model probabilities as weights. For data \(D\), a model prior probability \(P(M_k)\), and the model’s marginal likelihood \(P(D\mid M_k)\), the posterior weight is

The marginal likelihood integrates the likelihood over the model’s parameter prior. Consequently, Bayesian weights depend on both the observed data and the prior distributions for parameters and models. Unlike AIC weights, these are posterior probabilities for the candidate models under the specified Bayesian model set and priors. They are not prior-free probabilities, nor do they cover models that were not considered.

A BIC-based approximation is sometimes used to approximate relative Bayesian model support under particular regularity conditions and prior assumptions. It should not be treated as identical to calculating posterior model probabilities from specified priors and marginal likelihoods. Whatever weighting method is used, report it clearly.

Uncertainty Within and Between Models

A model-specific estimate has uncertainty even if the model were fixed; uncertainty also arises because plausible models can give different estimates. For Bayesian model averaging, the total variance of a quantity \(Y\), conditional on the data, separates into these two components:

Here \(w_k=P(M_k\mid D)\), and \(\operatorname{E}(Y\mid D)=\sum_k w_k\operatorname{E}(Y\mid D,M_k)\). The first sum is the weighted within-model variance. The second is the between-model variance: it is large when models with substantial weight give quite different results.

The decomposition follows by conditioning on the model indicator. Write \(\mu_k=\operatorname{E}(Y\mid D,M_k)\) and \(\mu=\operatorname{E}(Y\mid D)\). For each model, expand \(Y-\mu=(Y-\mu_k)+(\mu_k-\mu)\), square, and take the conditional expectation. The cross-term is zero because \(\operatorname{E}(Y-\mu_k\mid D,M_k)=0\). Thus

Averaging this equality over models with weights \(w_k\) gives the stated variance formula. The same decomposition can be applied to model-specific estimates using their estimated within-model variances. For information-criterion weights, this can be a useful measure of uncertainty, but it is not automatically a calibrated posterior variance: the weights are not posterior probabilities, and uncertainty from estimating weights and searching over models may not be included.

Choosing What to Average and Checking the Result

Predictions are often natural averaging targets because every candidate model can usually predict the same quantity for the same covariate profile. Parameter averaging requires more care. If a covariate is absent in one model, its effect can sometimes be represented as zero for a prespecified common effect, as in the example above. This makes the average an unconditional effect across the model set, not the effect conditional on inclusion. If coefficients use different reference categories, transformations, or units, first express them on a shared scale or average a common prediction instead.

Check the following before interpreting an average:

• Common data and likelihood: verify that the criterion or evidence used for weighting is comparable across models.
• Common target: fix the dose, time, patient characteristics, contrast, or other estimand before calculating model-specific values.
• Weight concentration: report whether one model dominates or whether several models contribute materially. Weights near one do not prove the dominant model is correct.
• Prediction plausibility: inspect the model-specific and averaged predictions across relevant covariate ranges. An average can conceal poor fit or implausible behavior in an individual model.
• Candidate-set sensitivity: examine whether conclusions change when defensible candidate models or prior model probabilities change.

Model averaging is not the only approach. Selecting one model is simpler to communicate, but treats that selection as fixed in subsequent summaries and can understate uncertainty. Fitting a full model avoids discrete selection but may include weak or poorly identified effects. Cross-validation can assess predictive performance, and stacking can choose weights to optimize predictive performance; those weights are not automatically probabilities that models are true. These approaches answer related but different questions, so the choice should match the purpose of the analysis.

Common Pitfalls and Limitations

A frequent mistake is to call AIC weights “the probabilities that each model is correct.” They are normalized relative-support weights for a specified candidate set and criterion, not posterior model probabilities. The correction is to state the weighting method and interpret the result accordingly; use Bayesian language only when weights are actually posterior probabilities derived from model priors and marginal likelihoods.

Another mistake is to average incompatible coefficients simply because they share a symbol. A coefficient per unit of a raw covariate is not directly comparable to an exponent in a power model, and a coefficient under one reference category may not match a coefficient under another. Average a common prediction or translate each model’s result to the same estimand before combining.

Finally, averaging cannot rescue a candidate set that excludes important structure, fix misspecified observation or random-effects models, or guarantee good extrapolation. The result is conditional on the models, data, and weighting choices considered. Report those choices and, where relevant, show how the average and its uncertainty respond to reasonable alternatives.