Prediction Error Metrics for External PK Validation

What a Prediction Error Measures

In external validation, a prediction error compares a model prediction with the corresponding observed concentration in data not used to estimate or update the locked model. The arithmetic is simple, but the result depends on two choices that must be stated: which prediction is being evaluated and what counts as one unit of evaluation. A prediction for each concentration record and a summary that gives each participant equal weight answer different questions when participants contribute different numbers of records.

Let \(C_{ij}\) be the observed concentration for participant \(i\) at record \(j\), and let \(\widehat{C}_{ij}\) be its corresponding prediction. Use the same concentration units for both quantities. This tutorial defines error as prediction minus observation, so positive error means overprediction. Some reports use the opposite sign; stating the convention is essential when comparing results.

If there are \(N\) evaluated records, with one prediction paired to each observation, the record-level metrics are

ME, MAE, and RMSE have the same concentration units as the observations; MSE has squared concentration units. ME summarizes signed average error, so positive and negative errors can cancel. MAE summarizes the typical absolute discrepancy in concentration units. MSE and RMSE give relatively greater influence to large errors because squaring grows faster than taking an absolute value.

The Prediction and Evaluation Units

A prediction must be identified before calculating its error. Population predictions use the locked population model and the available design information without estimating a participant-specific random effect from that participant's observed concentrations. Conditional predictions incorporate participant-specific information, commonly through an estimated individual effect. The two prediction types answer different predictive questions; their errors should be labelled and not treated as interchangeable evidence about the same task.

The earlier material on Population and Conditional Prediction Performance distinguishes those targets. Here, the key requirement is to keep the chosen prediction vector fixed while calculating its metrics. For example, an error calculated from a population prediction cannot be described as the error of a conditional prediction simply because both refer to the same observed concentration.

The evaluation unit also matters. A pooled record-level metric gives every concentration record equal weight. If one participant contributes many more records than another, that participant contributes more terms to the sum. An equal-participant summary instead first calculates a summary within each participant and then averages those participant summaries. Neither is automatically correct in all settings: the appropriate choice depends on whether the target is a typical evaluated record or a typical participant.

Why Signed and Unsigned Metrics Differ

Signed errors retain direction. For example, equal errors of \(+3\) and \(-3\) mg/L have an ME of zero, even though neither prediction is exact. Absolute and squared errors do not cancel in this way: in that example MAE is 3 mg/L and MSE is 9 (mg/L)\(^2\). For this reason, ME alone is not a measure of overall error magnitude.

The relationship between MSE, signed bias, and dispersion follows from a simple decomposition. Treat the \(N\) observed errors as a fixed set, define their mean as \(\bar{e}\), and define their variance with divisor \(N\) as \(v_e=N^{-1}\sum_{k=1}^{N}(e_k-\bar{e})^2\). Since \(e_k=(e_k-\bar{e})+\bar{e}\), expand the square and sum:

The middle term is zero because the deviations from their arithmetic mean sum to zero. This identity shows why squared error reflects both the spread of errors around their mean and the magnitude of the mean error. It uses the error variance with divisor \(N\); substituting a sample variance with divisor \(N-1\) would require adjusting the identity.

Metrics for Different Prediction Targets

A performance metric is conditional on the predictions used to create its errors. If population predictions and conditional predictions are both of interest, calculate and report their metrics separately, using matched observations and an explicitly stated evaluation unit. Conditional predictions can use information from a participant's concentration data, so lower conditional error does not by itself show that population predictions will perform better in a new participant or external setting.

Assumptions, Checks, and Limitations

These descriptive calculations require no normality assumption. They do require correctly matched observations and predictions, a consistent concentration scale and unit, and a defined evaluation set. Every included pair must refer to the same participant, sampling occasion, analyte, and intended prediction target. The model and its predictions should also correspond to the external validation procedure being assessed, rather than predictions generated after using the validation observations to update the model.

Before interpreting a metric, check the following:

• Direction: confirm that error is prediction minus observation, so positive values mean overprediction.
• Pairing: verify that each prediction is matched to its actual observed concentration and record.
• Prediction type: label population or conditional predictions and state what participant-specific information was used.
• Weighting: report whether records are pooled or participant-level summaries receive equal weight.
• Units and scale: report concentration units for ME, MAE, and RMSE, and squared units for MSE.

A metric compresses many errors to one number and can conceal important features of the set. ME can be near zero when large positive and negative errors cancel. MAE and RMSE do not show whether errors tend to overpredict or underpredict. RMSE is particularly sensitive to a small number of large errors; that sensitivity can be informative when large misses matter, but it can also make the summary unstable in a small external sample. MSE is even less immediately interpretable because its units are squared.

All four metrics depend on the concentration scale. Errors of 1 mg/L have a different practical meaning at low and high concentrations, and an absolute-error metric does not adjust for that difference. Relative or logarithmic errors define different summaries and require their own choices and assumptions; they should not be substituted silently for absolute concentration errors, especially when concentrations are zero or close to zero. A performance summary also does not describe prediction-interval coverage or establish how observed concentrations relate to predictions across their range.

There is no universally best single metric. A clear external validation report usually gives a signed summary alongside at least one magnitude summary, identifies the prediction type and evaluation unit, and explains why that target is relevant. Comparing models using one metric is meaningful only when the observations, prediction target, inclusion rules, and weighting are comparable.

Common Mistake: Treating a Small Mean Error as Accurate Prediction

A common but incorrect argument is: “The ME is close to zero, so predictions are accurate.” A near-zero ME establishes only that average signed errors are small in the chosen aggregation. It does not rule out large errors in both directions. For example, errors of \(+5\) and \(-5\) mg/L have ME \(=(5-5)/2=0\) mg/L but MAE \(=(5+5)/2=5\) mg/L. The correction is to report signed direction and error magnitude separately, and to state the unit and weighting for each.