Model Qualification Through External and Temporal Evaluation

Why Evaluate Transportability?

A population PK model can describe its development data well and still predict poorly in a new setting. The new study may enroll a different patient mix, use a different dosing regimen, or collect concentrations at different times. Model qualification therefore asks a focused question: does the fitted model make sufficiently reliable predictions for the intended development decision beyond the data used to build it?

External evaluation uses data from a separate study or setting. Temporal evaluation holds out data collected later, such as a later cohort or a subsequent period. In either case, the essential safeguard is to make predictions without using the evaluation concentrations to fit the model or estimate individual effects. A model that is updated using those data may be useful, but its performance on them is no longer an independent evaluation.

Transportability is always relative to a prediction target. Predicting concentrations at scheduled sampling times is not the same task as predicting individual AUC, a population exposure distribution, or a subgroup contrast. The target, prediction method, and acceptable performance should be specified before examining evaluation results.

Set Up a Fair Evaluation

Before calculating any metric, specify which data are held out, what the model predicts, and how predictions will be compared with observations. The model structure and parameter estimates are fixed for the evaluation. Predictions may use information available at the time the prediction would be made, such as dose history and baseline covariates. They should not use held-out concentrations to estimate an individual's empirical Bayes random effects when the goal is to test prospective population predictions. Doing so conditions on the answer and can make predictions look artificially accurate.

The observation and prediction must refer to the same quantity, unit, time, dosing history, and population definition. Concentration predictions should be evaluated at the actual observation times, not at convenient times that were not sampled. If one participant contributes many samples and another contributes only a few, a pooled observation-level metric gives the densely sampled participant greater weight. That may be appropriate for a sample-level objective, but for a participant-level objective, calculate participant summaries or give each participant equal weight.

A fair comparison also checks whether differences in assay, dose records, covariate definitions, or study conduct could explain a discrepancy. Use the model and observed data on comparable terms; for example, account for the evaluation study's dosing and sampling design when generating predictive simulations. Do not attribute every mismatch to a structural model defect before checking these inputs.

Prediction Errors and What They Measure

For an observed concentration \(C_i\) and its corresponding prediction \(\hat{C}_i\), define the signed prediction error as \(e_i=C_i-\hat{C}_i\). Positive errors mean the prediction was below the observation; negative errors mean it was above. For \(n\) paired observations, the mean prediction error summarizes signed bias, the mean absolute error (MAE) summarizes typical absolute discrepancy, and the root mean squared error (RMSE) penalizes larger errors more strongly.

For concentration predictions, these measures have the same concentration units as the data. Mean error near zero alone is not evidence of precise prediction: overprediction and underprediction can cancel. MAE and RMSE help show the size of discrepancies, but their values depend on the scale and sampling design. Neither provides a universal pass/fail rule.

A short identity explains why bias and overall error should be considered together. Let \(\bar{e}\) be the mean of the \(n\) errors, and define their empirical variance using divisor \(n\). Since \(e_i=(e_i-\bar{e})+\bar{e}\), squaring and averaging gives:

The cross-product term is zero because \(\sum_{i=1}^{n}(e_i-\bar{e})=0\). Thus, squared error reflects both the spread of the errors and squared bias. A small mean error can coexist with substantial error spread.

Compare Exposure Distributions and Patient Mix

Concentration-level metrics do not fully assess predictions of exposure. If the intended use concerns AUC, peak concentration, or another exposure summary, compare predicted and observed values of that same summary. Useful comparisons include central tendency, spread, and relevant quantiles, such as the median and upper tail. A close mean does not guarantee a close distribution: the model could predict the center correctly but misrepresent variability or high exposures.

Observed individual exposure summaries should be derived in a way that is appropriate to the evaluation data and does not use the model under evaluation to create the “observed” values in a way that guarantees agreement. When exposure must be estimated from concentration samples, state how that estimate was obtained and consider its uncertainty. Predictions should likewise represent relevant parameter and population variability rather than only a typical individual's profile.

Population composition matters. For example, if clearance-related covariates differ between studies, their overall exposure distributions may differ even when the model correctly predicts exposure within each covariate group. Compare like with like, or standardize both sets of summaries to a common covariate distribution. This separates a change in the mix of patients from a failure to capture relationships represented in the model. It does not remove the need to check whether the model's covariate relationships remain plausible in the new setting.

Predictive Intervals, Uncertainty, and Temporal Evaluation

A point prediction omits uncertainty. A predictive interval represents a range of outcomes expected for a future observation or individual under the model, including the variability relevant to that prediction target. Its coverage can be checked by calculating the fraction of held-out observations falling within the interval. Coverage should be assessed alongside interval width: extremely wide intervals may achieve high coverage but provide little decision value. Coverage can also be examined across prespecified clinically meaningful time points or groups, provided the evaluation data support those comparisons.

Evaluation results themselves are uncertain, particularly when there are few participants. Because repeated samples from one participant are correlated, uncertainty summaries should respect participant-level clustering; for example, resampling participants rather than individual records can preserve within-participant dependence. Report the amount of evaluation data and avoid treating many samples from a few people as equivalent to broad independent evidence.

Temporal evaluation follows the same logic as external evaluation but uses later data. Freeze the model and prediction procedure before examining the later observations. A change in enrollment criteria, dosing, assay, or clinical practice can affect performance; documenting such changes helps distinguish a genuine transportability concern from a difference in how the data were generated. If the model is refitted using the later data, that is model updating, not an independent test of the original model.

Diagnostics, Alternatives, and Limitations

A single aggregate score can hide where a model fails. Examine prediction errors against time after dose, predicted concentration, dose, and important covariates. Look for systematic underprediction or overprediction, patterns at particular phases of the concentration-time profile, and clusters of large errors within participants. Compare observed and predicted exposure distributions, including spread and upper-tail behavior when those features matter to the decision. These checks complement, rather than replace, the established goodness-of-fit and simulation-based diagnostic tools.

External evaluation is stronger evidence of transportability than assessment on the same data used to develop a model, because it tests prediction in a distinct data set. Temporal holdout can provide a useful test of later-data performance, but may be less informative about a substantially different setting if the later cohort is otherwise very similar. Internal goodness-of-fit assesses consistency with development data; simulation-based diagnostics assess whether data features resemble model-generated data under specified conditions. Neither alone substitutes for independent prediction.

There is no universal prediction-error threshold that qualifies every population PK model. Acceptable error depends on the decision, measurement scale, and consequences of over- or underprediction. A model used to support a dose decision near a safety boundary may require more scrutiny of the relevant upper tail than one used for a broad descriptive summary. Small evaluation samples, different study designs, uncertain observed exposure estimates, and unrepresented patient characteristics all limit conclusions. Qualification is evidence for a specified use, not proof that a model is true or will work in every future population.