Worked External Validation and Model Averaging Analysis

From Frozen Models to an External Comparison

External validation asks how well predictions from an already specified model perform for records that were not used to fit or revise it. A useful worked analysis must preserve that separation all the way through the calculation: the model, its parameters, the prediction procedure, and any averaging weights are fixed before the evaluation outcomes are examined. Otherwise, apparent validation performance can be optimistic because information from the evaluation set has influenced the predictions being assessed.

Here, the model predictions are hypothetical, and the four records below are hypothetical external concentration-time records. The purpose is to demonstrate the analysis, not to represent a clinical study. Both candidate models provide a predictive distribution for each record. For transparent arithmetic, these distributions are specified as normal on the concentration scale. Model A has a predictive standard deviation of \(0.4\) mg/L, and Model B has a predictive standard deviation of \(0.7\) mg/L. All records are assumed to have been correctly matched to the same external subject, nominal time, dose history, and covariates used to produce each prediction.

The normal distributions are a simplifying assumption for this worked calculation, not a universal choice for concentration data. A normal distribution has support below zero, which can be implausible for concentrations. In particular, Model B's distribution for record 4, \(N(0.5,0.7^2)\), assigns appreciable probability to negative values. A practitioner should use the predictive distribution actually justified by the model and measurement process, and examine whether its shape and support are credible.

Check the Records Before Scoring

Before computing a score, verify that each observed concentration is paired with the prediction for the same record and that units, time definitions, dosing history, and relevant covariates agree. For example, a prediction at 8 hours cannot be compared with an observed concentration at 12 hours merely because both belong to the same subject. The model and its prediction procedure must also remain frozen: no evaluation outcome should be used to refit parameters, revise the model, choose covariates, or tune the prediction method.

Calibration and Error Summaries

A simple calibration summary is the mean prediction error, here defined as observed minus predicted concentration. A positive value indicates underprediction on average for the records being summarized; a negative value indicates overprediction. It is not a complete calibration assessment, because positive and negative errors can cancel. Root mean squared error (RMSE) summarizes the typical magnitude of errors while giving larger errors extra weight.

Predictive interval coverage provides a different check: it records the fraction of observations falling inside intervals from the predictive distributions. For a normal predictive distribution, a central 90% interval is \(\mu_i\pm 1.645\sigma_i\). For Model A this gives a half-width of \(1.645(0.4)=0.658\) mg/L. Its four intervals are centered at \(0.7, 1.0, 0.8,\) and \(0.9\) mg/L, and all four observations fall inside their respective intervals. Model B's half-width is \(1.645(0.7)=1.1515\) mg/L, and its four observations also fall inside. The observed coverage is therefore \(4/4=100\%\) for each model in this tiny example. That result is highly imprecise and does not demonstrate that either model has 90% coverage in the population. Coverage should be interpreted alongside interval width: wider intervals can cover more observations while providing less precise predictions.

Scoring Predictive Distributions

A point prediction score evaluates a chosen summary such as the predictive mean. A distributional score evaluates the predictive distribution itself, including both its location and spread. For a continuous outcome, the logarithmic predictive score can be expressed as the log density assigned to the observed value. Larger log predictive density is better; equivalently, the negative log predictive density is a loss, for which smaller is better. Densities are not probabilities at an exact point, and their numerical values depend on the measurement units, so comparisons require the same outcome and units.

Applying the density formula to all four observations gives Model A densities of approximately \(0.9667, 0.8802, 0.8802,\) and \(0.9667\) \((\mathrm{mg/L})^{-1}\). Model B's densities are approximately \(0.5641, 0.5641, 0.5641,\) and \(0.4416\) \((\mathrm{mg/L})^{-1}\). Their mean log predictive densities are approximately \(-0.08077\) and \(-0.6337\), respectively. Thus Model A has the better mean log score in this illustrative set, while Model B's larger predictive SD does not compensate for its lower density at these observations. This is a comparison on four records, not a general ranking of the models.

Scoring a Stacked Predictive Mixture

A predictive mixture combines distributions, not parameter vectors. Let \(w_A\) and \(w_B\) be nonnegative weights that sum to one and are fixed before the evaluation outcomes are scored. At record \(i\), the mixture density is \(w_A f_{Ai}(y_i)+w_B f_{Bi}(y_i)\). The mixture mean is the same weighted average of the component means, but the mixture distribution generally is not a normal distribution with that mean and an averaged parameter vector. We use weights \(w_A=0.6\) and \(w_B=0.4\), stipulated here as having been estimated on a separate stacking dataset and then frozen.

This distinction follows directly from the definition of a mixture. The predictive density is obtained by summing the component densities with their weights, and the log score is then applied to that density. In general, \(\log(w_A f_A+w_B f_B)\ne w_A\log(f_A)+w_B\log(f_B)\). Because the logarithm is concave, Jensen's inequality gives \(\log(w_A f_A+w_B f_B)\ge w_A\log(f_A)+w_B\log(f_B)\) for positive densities and weights that sum to one. But this inequality compares mixture log density with the weighted component log densities for that record; it does not guarantee that the mixture beats the best single model on a validation set.

Stacking weights are chosen to optimize predictive performance on data reserved for that purpose, not by looking at the outcomes in the final evaluation set and then reporting the same set as untouched validation. Bayesian model averaging (BMA) instead uses model probabilities under a specified Bayesian model comparison framework to weight the predictive distributions. Both approaches average predictive distributions, but the interpretation and method for obtaining the weights differ. Neither approach averages the models' parameter estimates and treats that average as an automatically valid predictive model.

Reading the Comparison and Its Limitations

The numerical summaries do not all answer the same question. In this example, Model A has smaller RMSE than Model B, better mean log predictive density, and the same observed 90% interval coverage. The mixture has the smallest RMSE but does not have the best log predictive score. Reporting several prespecified summaries makes these distinctions visible. A strong analysis also examines error patterns by time, concentration range, site, and covariate support, rather than allowing an overall average to conceal systematic failures.

For a small evaluation set, apparent differences are unstable. Four records cannot establish transportability to a broad patient population, and the observed coverage proportion has considerable sampling uncertainty. Records from the same subject may also be dependent; treating them as independent replicates for uncertainty calculations can overstate the effective amount of information. This worked example therefore describes calculated performance, not a formal inferential claim about a population-level difference.

A further diagnostic is to inspect the predictive distributions themselves. The normal approximation used here assigns probability to negative concentrations, especially for Model B where the mean is \(0.5\) mg/L and the SD is \(0.7\) mg/L. If this conflicts with the scientific measurement process, density scores from that approximation may penalize or reward the model in misleading ways. A suitable positive-valued or otherwise scientifically justified predictive distribution should be used in a real analysis. Scores should also be calculated on a consistent outcome scale; density values change under unit transformations, even though correctly adjusted relative comparisons on the same data remain coherent.

Common Mistakes

• Updating the model with evaluation outcomes. Calling predictions external after using the same outcomes to refit or tune them breaks the intended separation. Freeze models and prediction procedures before evaluation.
• Scoring mismatched records. A wrong time, dose history, covariate record, or unit can produce an apparent model failure that is actually a data-alignment failure. Confirm the match before calculating metrics.
• Averaging log scores instead of densities. The mixture log score is \(\log(\sum_m w_m f_{mi}(y_i))\), not \(\sum_m w_m\log f_{mi}(y_i)\). Form the mixture density first.
• Equating good mean predictions with good distributions. RMSE uses predictive means; log score also responds to predictive spread and shape. Report which aspect each metric assesses.
• Overinterpreting a small external sample. Excellent observed coverage or a modest difference in average scores on a few records is not conclusive evidence of population-level performance.