What Is Being Validated?
A population PK model can be evaluated for more than one purpose. One analysis may quantify uncertainty in a typical clearance estimate; another may assess how well the model predicts concentrations for new patients; a third may ask whether a model developed in one study performs in a different clinical setting. These are related questions, but they are not interchangeable. A narrow parameter interval does not establish accurate predictions, and a good fit to the development data does not by itself establish performance in a new population.
A validation claim is interpretable only after stating which population is of interest, what prediction task the model must perform, and which performance quantity is being estimated. For example, predicting a first-dose concentration before any individual concentration data are available is a different task from predicting a later concentration after one or more measurements from that patient have informed an individual estimate.
For a concentration outcome \(Y\), let \(Z\) denote the information available when a prediction is made, such as dose history, covariates, sampling time, and any allowed prior concentration data. Let \(\hat f(Z)\) be the prediction rule. A general performance estimand is
where \(P^*\) is the target distribution of patients, observations, and study conditions, and \(L\) is a specified loss or performance function. The expectation is over the target population and relevant outcome variation, with the prediction rule and information set held to the stated task. The same fitted model can have different performance estimands under different \(P^*\), prediction tasks, or loss functions.
Three Distinct Targets
| Assessment | Target | What it answers | What it does not establish |
|---|---|---|---|
| Parameter-uncertainty assessment | Uncertainty in a specified fitted parameter or derived model quantity under a stated sampling and model framework | How variable or imprecisely estimated is the parameter? | How accurately the model predicts new outcomes |
| Internal predictive validation | Prediction performance for the source population and setting, using data not used to fit the corresponding prediction rule | How well might the rule predict new cases from a similar source setting? | Performance in a materially different population or setting |
| External predictive validation | Prediction performance in a separately obtained target population or setting | How well does the rule transport to the specified external setting? | Performance in every population, design, or clinical use not represented by that target |
A parameter-uncertainty assessment concerns an estimated model quantity, for example typical clearance or an interindividual variance. Its interpretation depends on the estimator, the assumed model, the data structure, and the uncertainty method. Predictive validation instead concerns outcomes generated for a specified target and evaluated against predictions made under a defined information set. Parameter uncertainty can be incorporated into predictions, but that does not turn an uncertainty interval for a parameter into a predictive validation result.
Internal predictive validation aims at new observations from the same underlying source setting. A held-out subset or a resampling-based prediction assessment can serve this purpose if predictions for each evaluated case are generated without using that case's outcome to fit or tune the corresponding rule. Evaluating predictions on the same observations used to fit the model is an apparent-performance assessment; it can be optimistic because the fitted model has already adapted to those data. External validation uses a separately obtained data source intended to represent a specified target beyond the development sample. “External” describes the relationship of the evaluation data to model development; it does not guarantee that the external patients are representative of every intended-use population.
Specify the Population, Task, and Estimand
The target population should be described in clinical and design terms. It may include patients with a specified indication, age range, renal-function distribution, dose regimen, formulation, and care setting. It should also specify the observation unit and sampling design: predictions at scheduled times, predictions for each measured concentration, or one summary per patient. These choices matter because patients with dense sampling can contribute many more observations than sparsely sampled patients.
The prediction task states what information is available at prediction time and whether the model may be updated using patient-specific data. A population prediction uses population-level model information and the allowed covariates and dosing history, without estimating an individual effect from that patient's concentrations. An individualized prediction can use specified patient measurements to update that patient's prediction. A validation must reproduce the intended task: using post-dose concentrations to inform an individual prediction is not a fair evaluation of a task that claims to predict before those concentrations are observed.
The performance estimand states both the metric and the averaging scheme. For a point prediction \(\hat Y_{ij}\) of concentration \(Y_{ij}\) for subject \(i\) at occasion \(j\), define the prediction error \(e_{ij}=Y_{ij}-\hat Y_{ij}\). A mean signed error describes directional bias, while a mean absolute error and root mean squared error describe different aspects of error magnitude. A coverage estimand for a predictive interval asks for the probability that an outcome from the target setting falls inside the interval generated for the stated task. Coverage is meaningful only with the interval level, prediction task, and target distribution specified.
These quantities answer different questions. Signed errors can cancel, so a small bias can coexist with large individual errors. MAE weights absolute deviations linearly; RMSE gives relatively greater influence to large errors. If performance is intended to represent a typical patient, first summarize within patient and then average across patients, or otherwise define a patient-weighted estimand. Averaging every sample equally instead gives greater weight to patients with more samples. Neither weighting is universally correct; the estimand must match the use.
Worked Example: Parameter Uncertainty Is Not Predictive Validation
Suppose a hypothetical population PK analysis estimates typical clearance as \(5.20\ \mathrm{L/h}\), with an estimated standard error of \(0.60\ \mathrm{L/h}\). For illustration, assume a normal approximation is reasonable and use a 95% interval based on \(1.96\) standard errors.
Calculation. The interval is
This calculation summarizes uncertainty in the estimated typical clearance under the stated approximation and model framework. It does not estimate the error of a concentration prediction for a new patient, the proportion of future concentrations covered by a predictive interval, or performance in a new study. To answer any of those questions, one must specify a prediction task and evaluate the resulting predictions against outcomes in an appropriate target population.
The normal approximation is an assumption, not a consequence of reporting a standard error. It may be questionable for a parameter constrained to be positive, for a highly skewed or weakly identified estimate, or near a boundary. The interval's units are those of clearance because it concerns clearance; prediction errors for concentrations would instead have concentration units.
Internal Predictive Validation and Its Estimand
An internal validation should mimic the intended use while preventing evaluated outcomes from influencing the corresponding prediction. For example, if the aim is prediction for new patients from the development setting, each evaluated patient's outcome information must be withheld from model fitting and any tuning that would otherwise use it. Subject-level separation is important when multiple records from one patient are correlated: placing some records from a patient in training and others in validation can make the task easier than prediction for a genuinely new patient.
The validation estimate is usually an empirical approximation to the target expectation. For a subject-level loss \(L_i\) in \(n\) evaluated subjects, an equally weighted estimate is \(\frac{1}{n}\sum_{i=1}^{n}L_i\). This estimates a subject-average target only when the evaluated subjects and their weighting reasonably represent that target. It does not automatically account for uncertainty from selecting or fitting the model, and an internal result can be imprecise when the validation sample is small.
Worked Example: Internal Point-Prediction Performance
In a hypothetical internal holdout, five separate patients are evaluated for one concentration prediction each. Their errors, defined as observed minus predicted concentration, are \(0.8\), \(-1.2\), \(0.4\), \(-0.6\), and \(1.1\ \mathrm{ng/mL}\). Assume each patient's prediction was produced without using that patient's held-out concentration, and suppose these five patients represent the intended source setting for this simple illustration.
Mean signed error.
The positive value indicates that, on average in these five evaluations, observed concentrations exceeded predictions by \(0.100\ \mathrm{ng/mL}\). The small average does not imply that the individual errors are small.
Mean absolute error.
Root mean squared error.
All three estimates describe this specific internal evaluation and its one-prediction-per-patient task. They are not guarantees for future patients, and five patients provide limited evidence about the target average. If instead each patient's many concentration records were pooled with equal record weights, the result could differ and would answer a different, observation-weighted question.
External Predictive Validation
External validation evaluates a prediction rule in data separate from those used for its development, with the target population and setting explicitly described. Differences from development data can be clinically important: covariate ranges, dose regimens, assay procedures, sampling times, adherence, co-medication, and clinical practice may all affect observed performance. A poor external result may reflect transport failure, a changed prediction task, data-quality differences, or a mismatch between the stated target and the actual external sample. The validation result alone does not identify which explanation is responsible.
An external result applies directly to the evaluated setting and metric. Generalizing beyond it requires further argument. In particular, one external study cannot certify a model for all hospitals or patient groups. Conversely, a change in average performance does not automatically mean the model is unusable; the relevant question is whether the performance meets a prespecified need for the intended decision and whether any deficiencies are consequential.
Worked Example: External Predictive-Interval Coverage
A hypothetical model supplies a nominal 90% predictive interval for one concentration at a prespecified time for each of 24 patients in a separate external study. The task is prediction without using that patient's concentration to update the prediction. The observed concentration falls inside the interval for 21 of the 24 patients.
Estimated coverage.
In this external sample, 87.5% of the evaluated concentrations were covered. The nominal level is 90%, but a difference between the observed fraction and 90% is not, by itself, proof of miscalibration; the estimate is based on only 24 patients. Under an illustrative independent Bernoulli approximation, its standard error is
This approximation assumes independent patient-level coverage indicators and is only a rough summary of sampling variability. The interpretation also depends on the chosen time and prediction task: it does not establish coverage at other times, for multiple concentrations jointly, or after individual updating. The external conclusion is therefore limited to the stated target and evaluation.
Justification: Why the Target Distribution Changes the Answer
The performance estimand is an expectation over a distribution, so changing that distribution can change the target even when the model and metric remain fixed. For a simple two-group illustration, suppose the expected absolute prediction error is \(1\ \mathrm{ng/mL}\) in group A and \(3\ \mathrm{ng/mL}\) in group B. If the target population has proportions \(0.8\) and \(0.2\), respectively, its mean absolute error is
If another setting has equal proportions of the two groups, the target mean absolute error is
The difference, \(0.6\ \mathrm{ng/mL}\), follows from the changed group composition, not from a changed within-group model performance. This is why “the model's MAE” is incomplete without the target population and weighting. The same logic applies to sampling times, dose regimens, and other design features that alter the mix of evaluated predictions.
Diagnostics, Alternatives, and Common Pitfalls
Before interpreting a validation summary, inspect whether the data and predictions actually match the stated target. Useful checks include comparing covariate and dose distributions between development and validation data; verifying that predictions use only information available at the intended decision time; examining performance across time, dose, and clinically relevant subgroups; and checking for patient-level overlap or other information leakage. A single average can conceal systematic underprediction in one subgroup and overprediction in another.
- Apparent fit mistaken for validation. Good agreement on data used to fit the model is not an independent estimate of performance on new cases.
- Parameter precision mistaken for prediction accuracy. A precise typical-parameter estimate does not rule out structural misspecification, residual error problems, or poor predictions for individuals.
- Observation weighting mistaken for patient weighting. Pooling all records can allow densely sampled patients to dominate. State whether the target is a typical patient, a typical concentration observation, or another unit.
- External label mistaken for transportability. A separate dataset is external, but its results only support claims about its represented target and task.
- Metric chosen after seeing results. Selecting a favorable metric or subgroup after evaluation weakens the interpretation. Define the primary estimand and relevant summaries in advance where feasible.
The main alternatives are complementary, not substitutes. Parameter-uncertainty assessment addresses uncertainty in fitted quantities. Internal predictive validation estimates performance for new cases from the development setting while limiting reuse of evaluated outcomes. External validation tests performance in a separately sampled target setting. Predictive checks using the development data can reveal patterns inconsistent with model predictions, but checking model adequacy on those data is not the same claim as independent out-of-sample validation. Likewise, a single global error metric may be supplemented by calibration, coverage, or subgroup summaries; each answers a distinct question and should be tied to a clear target.
Check Your Understanding
For each question, identify the relevant target or interpret the calculation. Brief solutions follow.
- A model has a narrow confidence interval for typical clearance. Does this establish that its predictions for new patients are accurate?
- In a new-patient prediction task, why can putting some records from each patient in both a training set and a validation set give an overly favorable assessment?
- A validation sample contains 10 patients with one record each and one patient with 20 records. What issue arises if all 30 records receive equal weight when the goal is performance for a typical patient?
- State the coverage estimate if 45 of 50 external observations fall inside their specified predictive intervals, and give its interpretation.
- Suppose absolute prediction error is \(2\ \mathrm{ng/mL}\) in each of two groups. Can a changed mixture of those groups change their weighted mean absolute error? Explain.
Solutions. (1) No. It summarizes uncertainty in a parameter, not predictive performance; evaluate predictions for the stated target task. (2) Records from the same patient share information, so training records can indirectly inform predictions for that patient's held-out records; the task is no longer prediction for a wholly new patient. (3) The densely sampled patient contributes 20 of 30 records and dominates an observation-weighted average; for a typical-patient target, define a patient-level summary or another appropriate patient weighting. (4) \(45/50=0.90\), or 90%; in this sample, 90% of the evaluated observations were covered, which does not alone establish exact 90% coverage in the target population. (5) No, not in this example: both groups have the same error, so any mixture has weighted mean absolute error \(2\ \mathrm{ng/mL}\). In general, unequal group-specific errors make the mixture relevant.