Scoring External Concentration Predictions

Why score external predictions?

External validation asks how well a model predicts concentrations in data not used to develop or fit it. A numerical score can summarize prediction errors, but no single score answers every question. Root mean squared error (RMSE) emphasizes large errors on the concentration scale; mean absolute error (MAE) summarizes absolute errors; and log predictive density evaluates whether the model assigned high probability density to the observed concentrations. These scores answer different questions and must be calculated for a clearly defined prediction target.

For example, a population prediction for a new subject and a prediction conditional on that subject’s permitted earlier observations are not interchangeable targets. The prediction rule, including any allowed subject-specific information, must be fixed before scoring. The same external observations and target must be used when comparing models. A score is a summary of predictive performance on those observations; it does not establish that the model is calibrated or transportable to every population.

Choosing the scale before calculating

Concentration-scale RMSE and MAE answer questions in the original units, such as mg/L. They can be useful when an absolute difference of a given size has a consistent practical meaning. Squaring in RMSE gives relatively more influence to large absolute errors. MAE weights each absolute error linearly and is less dominated by a single large discrepancy. Neither score, by itself, says whether an error is important for a particular clinical or pharmacological decision.

Concentrations often span orders of magnitude. In that setting, an error of 2 mg/L is a larger relative discrepancy at 1 mg/L than at 100 mg/L. For strictly positive observations and predictions, one can instead score log-concentration errors. With natural logarithms, define \(r_k=\ln(\widehat C_k/C_k)\). Then log-scale RMSE is \(\sqrt{m^{-1}\sum r_k^2}\), and log-scale MAE is \(m^{-1}\sum |r_k|\). These scores are dimensionless and describe multiplicative discrepancies: \(|r_k|=\ln(2)\), for example, means that the prediction and observation differ by a factor of 2.

The logarithm must be applied to a dimensionless quantity. Writing \(\ln(C_k/C_0)\), where \(C_0\) is a fixed reference concentration in the same units, makes this explicit. For the log error, the reference cancels: \(\ln(\widehat C_k/C_0)-\ln(C_k/C_0)=\ln(\widehat C_k/C_k)\). Log-scale errors are not defined for zero or negative values. Do not add an arbitrary small constant to make them defined without stating and justifying how that changes the scoring target.

The point prediction also matters. Under squared-error loss, the predictive mean minimizes expected squared error; under absolute-error loss, a predictive median minimizes expected absolute error. A typical-subject prediction or a median prediction is not automatically the predictive mean. When comparing models, use the point-prediction summary appropriate to the stated target and score, and apply the same rule to each model.

Log predictive density uses the whole predictive distribution

RMSE and MAE require one point prediction per observation. Log predictive density instead evaluates the predictive distribution assigned to the observed concentration. Let \(f_k(c)\) be the predictive density for \(C_k\), conditional on the frozen model, external dosing and covariates, and whatever subject-specific history is allowed by the prediction target. The individual log predictive density is \(\ln f_k(C_k)\). Larger values are better: the observation lies in a region to which the model assigned greater density.

If the target is the collection of measurements jointly, the relevant score is the log of their joint predictive density. A sum of pointwise log densities is appropriate when those observations are being treated as independent under the predictive model, or when the joint density factors into the corresponding conditional densities. Repeated concentrations within a subject are generally related through shared subject-level effects, so simply multiplying marginal densities can fail to represent the joint predictive distribution. The scoring unit and density factorization should therefore be stated.

For positive concentrations, a lognormal predictive model is one common illustration. If \(Z_k=\ln(C_k/C_0)\) is normally distributed with mean \(\mu_k\) and variance \(\sigma_k^2\), then the density on the concentration scale includes a change-of-variable factor:

The factor \(1/c\) is required because the density is being expressed for concentration rather than log concentration. Omitting it gives a density for the transformed variable, not for the observed concentration. Predictive uncertainty must also be represented: evaluating only a fitted curve as though it were a complete probability distribution does not produce a valid log predictive density.

Why the log score rewards a credible distribution

The log predictive density rewards distributions that place substantial density near what is observed, while penalizing distributions that assign very little density there. A short justification explains why the log score evaluates more than closeness to a point prediction. Suppose the true density is \(g(c)\) and a model proposes density \(f(c)\), both on the same response scale. The expected log score under the true distribution is \(\int g(c)\ln f(c)\,dc\). Its difference from the score for the true density is

Here \(D_{\mathrm{KL}}(g\Vert f)\), the Kullback–Leibler divergence, is nonnegative for probability densities, and equals zero when the densities agree except possibly on sets of probability zero. Thus, in expectation, the true predictive distribution achieves the largest log score. This result assumes the densities are defined on the same observation scale and that the expected log score exists; it does not mean that a finite external sample must rank models in the same order as their expected scores.

Comparisons, checks, and limitations

Scores are meaningfully comparable only when the scored observations, prediction target, and response representation are aligned. Concentration-scale RMSE for one analyte cannot be directly compared with RMSE for another analyte in different units. Likewise, log predictive densities on different scales or unit conventions are not directly interchangeable. A change of concentration units changes a density by a Jacobian factor and shifts its log score; comparisons within a common unit convention remain interpretable.

Useful basic checks include confirming that every prediction is matched to the correct external observation and time, verifying concentration units and transformations, and checking that all models are scored on the same set of eligible observations. Inspect the individual calculations if an aggregate score is surprising: one large error can strongly affect RMSE, while a very small predictive density can contribute a strongly negative log score. These checks identify calculation problems; the score alone does not explain the scientific cause of an error.

Below-quantification observations require particular care. Replacing a censored concentration by zero or by the quantification limit and then treating it as exact can distort point-error scores and log predictive density. A censored observation should be scored using the predictive probability of the interval or event represented by the censoring process, when that information is available. Scores also depend on which observations are included: if missing or censored records are handled differently across models, a numerical comparison can be misleading.

RMSE and MAE are alternatives when only point predictions are to be assessed; they do not reward uncertainty estimates. Log predictive density evaluates a distribution, but depends on its assumed form and can be difficult to interpret as a concentration-scale error. Reporting a point score and a distributional score can therefore be informative, provided both match the same target. No one of these scores replaces evaluation of other predictive properties.