Why Model Averaging Can Still Fail
Model averaging combines predictions from several candidate models. It can reflect uncertainty about which candidate is most useful, but it does not guarantee that the candidates include an adequate description of the exposure-response relationship. Nor does it guarantee that the resulting weights are stable or that predictions are reliable at exposures far from those observed. These limitations matter when an averaged result informs a dose or exposure decision: a precise-looking average may conceal a shared modeling error or a fragile conclusion.
Let \(M_1,\ldots,M_K\) be the candidate models, let \(w_m\) be the weight assigned to model \(M_m\), and let \(\hat g_m(C^*)\) be its predicted response at exposure \(C^*\). A model-averaged prediction is
This formula is a combination rule, not a statement that at least one candidate is correct. Weights based on information criteria, predictive performance, or Bayesian model probabilities answer different questions and rely on different assumptions. In particular, a candidate’s relative support within a specified set does not establish that the set contains the truth or an adequate approximation.
Shared Misspecification Is Not Averaged Away
Suppose every candidate imposes a decreasing response as exposure increases. If the true relationship falls and then rises, every candidate can miss the rebound. A weighted average of decreasing functions is also decreasing, so no choice of nonnegative weights summing to one can reproduce that feature. More generally, model averaging can reduce sensitivity to choosing one candidate over another, but it cannot create structure absent from all candidates.
The underlying justification is a simple convex-combination argument. If all candidate predictions at \(C^*\) lie between \(a\) and \(b\), then
To see this, multiply each pair of inequalities by \(w_m\geq 0\) and sum over \(m\). The lower bounds sum to \(a\sum_m w_m=a\), and the upper bounds sum to \(b\sum_m w_m=b\). Thus, the average remains within the candidates’ range. This property is useful for diagnosing impossible expectations: an average cannot recover a response shape, direction, or value that all candidates rule out at the prediction point.
Worked Example: A Candidate Set Misses a Rebound
Consider hypothetical mean response scores measured at exposures \(C=0,1,2,\) and \(4\) mg/L. Suppose the observed scores are 100, 80, 60, and 80 points. Two fitted candidate models are both constrained to predict a nonincreasing response:
The residual sum of squares for \(M_1\) is \(0^2+(-2)^2+(-4)^2+34^2=1{,}176\) points squared. For \(M_2\), it is \(0^2+(-4)^2+(-8)^2+28^2=864\) points squared. Both candidates miss the observed increase from 60 to 80 points between 2 and 4 mg/L. For illustration, suppose the model-averaging procedure assigns weights \(w_1=0.25\) and \(w_2=0.75\). These weights are stipulated here to demonstrate the averaging consequence; they are not calculated from these residual sums of squares.
At 4 mg/L, the averaged prediction is
The average is between the two candidate predictions, 46 and 52 points, and remains far below the observed score of 80 points. Since both candidate predictions decrease with exposure, their weighted average also decreases. The example does not establish that the rebound is the true biological relationship; it shows that these candidates cannot represent the pattern in these hypothetical data. Increasing the number of candidates or changing their weights would not fix the problem unless the expanded set can represent the relevant shape.
Unstable Weights Can Hide Fragile Conclusions
Weights may change substantially when candidate models have similar support. For example, normalized AIC weights are proportional to \(\exp(-\Delta_m/2)\), where \(\Delta_m\) is the difference between candidate \(m\)'s AIC and the minimum AIC. When two AIC values are close, a small change in their relative values can shift their weights. This is not automatically a defect: the data may genuinely offer little basis for distinguishing the candidates. The risk arises when the predictions differ meaningfully and the scientific conclusion depends on the resulting weighted value.
Worked Example: A Small Score Change Reverses a Threshold Decision
Suppose two hypothetical exposure-response models predict a reduction in a response score at a target exposure: \(M_1\) predicts 10 percentage points and \(M_2\) predicts 90 percentage points. Their AIC values are 100.0 and 100.2, respectively. With \(\Delta_1=0\) and \(\Delta_2=0.2\), the normalized weight for \(M_1\) is
The model-averaged predicted reduction is \(0.5250(10)+0.4750(90)=5.250+42.75=48.00\) percentage points. If a small change in the data reverses the AIC ordering, with \(M_1\) now 0.2 units above \(M_2\), the weights reverse to approximately 0.4750 and 0.5250. The average becomes \(0.4750(10)+0.5250(90)=4.750+47.25=52.00\) percentage points.
The change in the averaged prediction is 4 percentage points. Yet if a hypothetical decision rule uses 50 percentage points as a threshold, the conclusion changes from below the threshold to above it. This does not mean that AIC weights are inherently invalid; it means the threshold conclusion is sensitive to small changes in relative support. The predictions are in percentage points, and the example treats the threshold only as an illustration, not as a clinical recommendation.
A useful diagnostic is therefore not just to report the final weights, but to assess whether they and the decision-relevant predictions change under reasonable perturbations. Perturbations might include resampling subjects, modest changes to the analysis data, or defensible changes to the candidate set. If the conclusion changes, report that sensitivity rather than presenting one set of weights as decisive. This check complements calibration and weight diagnostics: acceptable average calibration in the observed data does not ensure stable weights or reliable predictions under a different target exposure.
Extrapolation Is a Separate Risk
A model can fit observed exposures reasonably well and still give unreliable predictions beyond them. Functional forms that are difficult to distinguish over a narrow range may imply very different behavior outside that range. The weighted prediction then inherits the extrapolation assumptions of its candidates. A narrow interval around the average, if calculated, would not demonstrate that those assumptions are correct.
Worked Example: Agreement in Range Does Not Validate an Extrapolation
Suppose two hypothetical models describe a response score \(R\) as a function of concentration \(C\) in mg/L. The linear model is \(R=100-4C\), and the Emax-shaped model is \(R=100-60C/(5+C)\). Assume observed concentrations extend from 0 to 10 mg/L. At 0 mg/L, both predict 100 points. At 10 mg/L, the linear model predicts \(100-4(10)=60\) points; the Emax model predicts \(100-60(10)/(5+10)=100-40=60\) points.
At an unobserved concentration of 20 mg/L, however, the predictions differ:
With illustrative equal weights, the averaged prediction is \((20+52)/2=36\) points. The value 36 is a valid arithmetic average of the two predictions, but neither their agreement at 0 and 10 mg/L nor the averaging operation establishes that 36 is a reliable prediction at 20 mg/L. The models encode different assumptions about behavior beyond the observed concentration range. Without data or defensible external scientific knowledge to evaluate those assumptions, the extrapolated average is weakly supported.
Support is not always summarized by the minimum and maximum exposure alone. If the response depends on covariates, an exposure may be represented in the data only for particular covariate values, or vice versa. A prediction at a combination rarely or never observed can be extrapolative even when each individual variable lies within its marginal range. For repeated observations, the interpretation of exposure also depends on how it is defined for the analysis; do not assume that a single concentration value represents every dosing history.
Diagnosing Risk and Choosing a Response
No single diagnostic proves that a candidate set is adequate. Instead, combine evidence about fit, weights, prediction targets, and scientific plausibility. The earlier material on predictive diagnostics and model comparison provides tools for evaluating candidates relative to one another; the checks below address what that relative evaluation cannot establish on its own.
State the exposure range and relevant covariate patterns represented by the data. Flag prediction targets outside, or sparsely represented within, that support.
Check residuals and observed-versus-predicted patterns for systematic curvature, nonmonotonicity, plateaus, or other features that the candidates cannot express. A common pattern across candidates may indicate shared structural inadequacy.
Assess whether weights, target predictions, or threshold conclusions change under reasonable data perturbations and defensible candidate-set changes. Report changes that matter to interpretation.
If candidates omit relevant response structure, develop scientifically defensible alternatives and reassess fit. If support is inadequate for the target, seek relevant data, restrict the target, or present the extrapolation as assumption-dependent rather than well established.
The principal alternatives address different problems. Expanding the candidate set can represent omitted shapes or mechanisms, but a larger set is not automatically better: added models need scientific justification, and relative weights still do not certify adequacy. Restricting the prediction target to a better-supported exposure range avoids making a claim the data cannot support, but may not answer the original question. Additional informative data can improve support where it is needed. Explicit model-discrepancy or robust approaches can represent uncertainty beyond the listed candidates, but they require further assumptions and do not remove the need for diagnostics.
A common mistake is to reason, “The weights sum to one, and several models were averaged, so the result accounts for model uncertainty and must be safer than any single prediction.” The weights summing to one only makes the result a convex combination. It does not account for models that were not considered, guarantee that the candidates are adequate, or validate predictions outside the observed support. The correction is to interpret the average conditionally: it summarizes predictions from this candidate set under the chosen weighting procedure. Assess adequacy, sensitivity, and support separately before relying on the result.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- At a target exposure, three candidates predict 20, 40, and 60 response units. Can a weighted average with nonnegative weights summing to one equal 75 units?
- Two candidates both predict a decreasing response over the full exposure range. What does a decreasing model-averaged prediction imply if the observed response increases at the highest exposure?
- Two candidate models have nearly equal support but substantially different predictions at the target exposure. Name one diagnostic and explain what it assesses.
- Two models agree at the observed exposure endpoints but diverge sharply at a target exposure beyond the observed range. What can agreement at the endpoints establish, and what can it not establish?
Solutions. (1) No. The average must lie between the smallest and largest candidate predictions, so it cannot exceed 60 units. (2) The average is also decreasing, because a nonnegative weighted sum of decreasing predictions is decreasing. The increase in the observed data suggests that the candidate set may omit relevant structure; changing weights alone cannot reproduce it. (3) Perturb the data reasonably, for example by subject-level resampling where appropriate, and examine whether weights and the target prediction change. This assesses sensitivity, not whether either candidate is true. (4) Endpoint agreement shows only that the predictions coincide at those exposure values. It does not establish which model, if either, describes behavior beyond the observed range, so the extrapolated prediction remains dependent on model assumptions.