Diagnosing Model-Averaged Predictions with Calibration and Weight Checks

Why Diagnose the Averaged Prediction?

A model-averaged prediction combines predictions from several candidate PK/PD models. Combining models can reduce dependence on a single fitted structure, but the average is not automatically accurate, calibrated, or adequately uncertain. A weighted prediction may be systematically too high, work well overall but poorly for a clinically important subgroup, or appear to use many models even though one model supplies nearly all the weight. Diagnostics should therefore examine both the predictions and the weights that produced them.

The diagnostics in this tutorial apply to predictions evaluated against outcomes not used to fit the candidate models or determine the averaging weights. For repeated PK/PD data, the validation split should respect the unit of prediction: when the intended use is prediction for new individuals, keep all records from an individual together. Subject-level cross-validation is one way to obtain such predictions. Reusing the fitting data can make calibration and coverage appear better than they will be for new observations.

The outcome and prediction must refer to the same quantity, time, and scale. For example, a prediction of a change from baseline cannot be directly averaged with a prediction of the absolute response unless both are first expressed as predictions of the same estimand. The weights may come from a model-averaging procedure such as Bayesian model averaging or predictive stacking, but the diagnostics below assess the resulting predictions rather than establish that one weighting method is universally preferable.

Calibration: Are Predicted Values on Target?

Calibration compares predictions with outcomes. For a continuous response, a basic summary is the mean prediction error, also called calibration-in-the-large with this sign convention. A positive value means observed outcomes exceed predictions on average; a negative value means predictions exceed outcomes on average. This summary can miss patterns in which predictions are too low in one range and too high in another.

A calibration plot can show observed outcomes against predicted values, often using groups of predictions to make the pattern visible. The ideal reference is equality: observed values track predicted values. A complementary linear calibration summary regresses observed outcomes on predictions, \(y_i=a+b\hat{y}_i+\epsilon_i\). The ideal values are \(a=0\) and \(b=1\). An intercept away from zero indicates a systematic offset under this linear summary; a slope below one often indicates predictions that vary too much, while a slope above one often indicates predictions that vary too little. These interpretations are diagnostic, not proof of a particular model defect. They also depend on the validation sample covering the prediction range of interest.

Predictive-Interval Coverage

A prediction interval expresses uncertainty about a future outcome, not just uncertainty about a model's mean prediction. For a nominal \(100(1-\alpha)\%\) predictive interval, empirical coverage is the fraction of validation outcomes that fall within their corresponding intervals. If the interval procedure is calibrated for the target population and prediction setting, its long-run coverage should be near the nominal level. A finite validation sample will not generally achieve the nominal percentage exactly.

Here \(L_i\) and \(U_i\) are the lower and upper limits for observation \(i\), and \(I(\cdot)\) is 1 when its condition is true and 0 otherwise. For model averaging, intervals should represent the predictive distribution of the averaged procedure, including relevant outcome variability and uncertainty. Simply averaging the candidate models' lower limits and averaging their upper limits does not in general produce a valid interval for the mixture predictive distribution. The interval construction must be appropriate to the averaging method.

Coverage alone is not enough. Extremely wide intervals can attain high coverage while offering little useful information. Assess interval width alongside coverage, and consider whether a clinically acceptable range of outcomes is adequately represented. Calibration plots, point-prediction error, and predictive intervals answer related but distinct questions.

Subgroup Checks and Weight Concentration

Overall summaries can conceal local failures. Where sample size and scientific purpose support it, repeat calibration and coverage checks for clinically meaningful subgroups, such as dose level, baseline severity, or a relevant covariate category. Define subgroup membership before examining results when possible. Report subgroup counts because small groups produce unstable summaries; do not interpret a noisy subgroup percentage as a definitive difference. For repeated observations, account for dependence within individuals or summarize at the independent validation unit rather than treating correlated records as independent evidence.

Weights provide a separate diagnostic. With nonnegative weights summing to one, define the effective number of contributing models as

This quantity equals \(M\) when all \(M\) weights are equal and approaches 1 when one model dominates. It summarizes concentration, not predictive accuracy, model truth, or the number of scientifically distinct structures represented. Its meaning also depends on which models are in the candidate set and how their weights were obtained.

Why These Checks Work—and Where They Can Mislead

The connection between average-model errors and candidate-model errors follows directly from the weights. For each validation observation, suppose all candidate predictions target the same outcome, and the weights are fixed for that prediction. Then

where the third expression uses \(\sum_m w_m=1\). Thus the averaged prediction error is the weighted average of candidate-model errors for that observation. Opposing errors can cancel: small overall bias does not guarantee small errors for individual patients, good calibration across the prediction range, or adequate interval coverage. The derivation also explains why weights that concentrate on one candidate make the average's point prediction particularly dependent on that candidate's predictions.

For a fair evaluation, the same validation outcomes must not have been used to estimate candidate parameters, tune the weights, or select the reported diagnostic pattern. Otherwise the evaluation is optimistic. In cross-validation, each prediction and its weight construction must exclude the held-out subject when the target is a new subject. Repeatedly inspecting validation results and changing the method in response also turns that data into development data; a fresh validation set or appropriately nested resampling is then needed for an independent assessment.

A calibration plot, an empirical coverage fraction, subgroup summaries, and weight concentration complement one another. A good overall mean error cannot certify a slope near one. Nominal coverage cannot certify useful interval width. An apparently balanced weight vector cannot certify good predictions. Conversely, a warning in one small subgroup or a low effective model count should prompt investigation rather than an automatic rejection.

Common Pitfall

A common mistake is to conclude, “The average prediction error is near zero, so the model-averaged predictions are calibrated.” The conclusion does not follow. Positive errors in one subgroup can cancel negative errors in another, and errors at low predictions can cancel errors at high predictions. Check the calibration pattern across the prediction range and relevant subgroups, then inspect interval coverage and uncertainty. Likewise, do not interpret a 90% interval as containing a particular future patient's outcome with 90% probability after observing the interval; its nominal coverage is a repeated-prediction property under the assumed prediction setting.

Model-specific diagnostics are useful for locating which candidates contribute to a pattern, while diagnostics of the averaged prediction directly evaluate the object intended for use. Neither replaces the other. Predictive cross-validation scores summarize comparative predictive performance under their validation design, but they do not by themselves show calibration at each range, subgroup, or interval level. Use these approaches as complementary evidence, with the validation design matched to the intended use.