Analytical Validation of Model Predictions

What Analytical Validation Evaluates

Analytical validation asks whether a specified model-and-prediction procedure produces predictions that agree sufficiently with observed clinical data for its intended use. It is not enough to report that predictions are “close.” The prediction target, prediction time, available inputs, evaluation population, and clinically important errors must be stated before results are examined. A model intended to predict a concentration before a dose decision, for example, should be evaluated against observations relevant to that prediction task—not against a different outcome selected after seeing the results.

The model should generate predictions for a validation data set without using the corresponding observed outcomes to fit or adjust those predictions. The observations and predictions must refer to the same target, time point, and units. The observed values are treated as the comparison reference, although measurement error in clinical observations can also contribute to discrepancies. Analytical validation therefore evaluates the complete prediction procedure under its specified conditions; it does not establish that the model will perform equally well at another site or in a later period.

A defensible evaluation prespecifies the intended prediction (such as an individual concentration or dose-related quantity), the eligible cases, the relevant prediction horizon, the primary performance metric, any clinically justified tolerance, and the plots to be reviewed. The validation data should be collected and processed in a way consistent with the intended use. A favorable average metric cannot compensate for a mismatch between the validation task and the clinical task.

Metrics: Bias, Magnitude, and Spread

No single metric captures every aspect of predictive performance. Signed errors summarize direction; absolute and squared errors summarize magnitude; and tolerance-based summaries answer whether errors are small enough for a stated purpose. Metrics should be reported in context and with units.

MPE near zero does not mean that individual predictions are accurate: positive and negative errors may cancel. MAE is directly interpretable in the target’s units and is less dominated by a single large error than RMSE. RMSE is also in the target’s units, but a large discrepancy contributes disproportionately. The sample standard deviation \(s_e\) describes error spread around the observed MPE; it is not the same as RMSE.

A useful check follows from the fact that the average squared error equals the squared mean error plus the average squared deviation from that mean. Taking square roots shows that RMSE cannot be smaller than the absolute MPE. Also, by the inequality between the quadratic and arithmetic means of nonnegative values, RMSE is at least as large as MAE. These relations are useful for checking calculations, not for deciding whether either metric is clinically acceptable.

When the clinical question concerns a meaningful error limit, also report the proportion of cases within a prespecified tolerance. The tolerance must be justified by the intended decision; it is not a universal property of the model. Relative errors can help compare performance across scales, but require care when observed values are near zero, and their interpretation depends on the denominator convention. Report the definition if they are used.

Plots and Clinically Relevant Summaries

Plots complement numerical summaries by showing features that an average can hide. An observed-versus-predicted plot places \(y_i\) on the horizontal axis and \(\hat{y}_i\) on the vertical axis, with the identity line \(\hat{y}=y\) as the reference for exact agreement. Points above that line are overpredictions under the error convention here; points below are underpredictions. The axes should have the same scale when visual distance from the identity line is being judged.

An error-versus-prediction plot places \(e_i\) against \(\hat{y}_i\), often with a horizontal reference at zero. A systematic slope, widening spread, clusters, or isolated extreme errors can signal that a single overall metric is not describing all cases adequately. These are prompts to investigate—not automatic proof of a particular cause. A histogram or other distribution plot of errors can reveal skewness or tails, while hiding which patients or prediction ranges produced those errors. Pair plots with summaries rather than substituting one for the other.

A tolerance-based summary makes a different statement from MAE or RMSE. If the intended use has a clinically justified absolute-error tolerance \(\delta\), report the number and proportion of validation cases for which \(|e_i|\leq\delta\). If overprediction and underprediction have different consequences, symmetric absolute error may be insufficient: prespecified directional summaries or decision-specific consequences may be more informative. Do not choose a tolerance after inspecting errors simply to produce a favorable percentage.

Prespecification, Dependence, and Failure Modes

The interpretation of a metric depends on how cases enter its calculation. With one prediction per patient, each patient contributes equally to an unweighted average. With repeated predictions for some patients, a pooled metric gives more weight to patients with more observations. That may be appropriate if the target is performance per prediction event, but not if the target is average performance per patient. State which estimand is intended. Repeated measurements within a patient are also correlated, so treating them as independent can make uncertainty estimates too optimistic. If uncertainty intervals are reported, their calculation should respect the sampling structure, for example by resampling patients rather than individual records when patients contribute multiple records.

Before interpreting results, check that each prediction is paired to the correct observation, that units and time points match, and that missing or excluded cases are described. Confirm that predictions were generated with only information available at the intended decision time; using later information can make performance appear better than it would be in practice. Examine the range and composition of evaluated cases, since a narrow or atypical set may not represent the intended application. A clean plot and a low average error cannot repair a mismatch in the prediction task or biased case inclusion.

Metric choice involves trade-offs. MPE is transparent about direction but permits cancellation. MAE is robust relative to RMSE but does not emphasize large errors as strongly. RMSE penalizes large errors more, which can be useful when large discrepancies are especially concerning, but one extreme observation may dominate it. A tolerance-based proportion is easy to relate to an acceptance rule but discards information about how far errors fall outside the tolerance. The observed-versus-predicted plot gives a direct view of agreement; error plots and distributions help reveal structures that an aggregate number may conceal.

• Do not equate zero bias with good prediction. Opposing errors can average to zero while individual errors remain large.
• Do not treat an acceptable average as universal acceptability. A metric can conceal large errors in a clinically important range or subgroup.
• Do not use a post hoc threshold as an acceptance criterion. Define clinically meaningful tolerances before examining validation performance.
• Do not interpret descriptive metrics as proof of causation or transportability. They describe agreement in the evaluated data and conditions.