Why the Distinction Matters
External validation asks how well a specified prediction procedure performs in data that did not determine that procedure. In population pharmacokinetics (PK), the procedure may use a population model, covariates, dosing history, and a defined prediction method to predict concentrations or another PK quantity. The external outcomes are compared with predictions, but they do not determine the model or the analysis choices being evaluated.
A tempting next step is to use those outcomes to adjust the predictions: for example, shift predicted concentrations upward, refit a slope, change a covariate effect, or select a different structural model. These may be reasonable steps toward a better model, but they change the question. The data now help create the procedure, so their performance is no longer an independent evaluation of that updated procedure.
“Fixed” does not mean that the model was never estimated. Its parameters may have been estimated in development data before external evaluation. It means that the procedure being assessed—including any choices made after examining data—is specified without using the validation outcomes. The earlier material on locking a model before external validation establishes why this specification should be documented in advance.
What Counts as Updating?
Updating includes any outcome-informed action that changes the predictions under evaluation. It is broader than re-estimating every parameter in a nonlinear mixed-effects model. Even a small adjustment can be an update if it is chosen from the evaluation outcomes.
- Parameter refitting: re-estimating population parameters, covariate effects, random-effect variances, or residual-error parameters from the external outcomes.
- Calibration: estimating an intercept, slope, or other mapping from predictions to observed outcomes using those outcomes.
- Model or method selection: choosing among structural models, covariate sets, prediction methods, or error models after comparing their external outcome performance.
- Outcome-guided analysis changes: selecting time windows, subgroups, transformations, or exclusions because they improve a reported external result.
Not every calculation involving external data is updating. Computing a prespecified error metric, plotting residuals, or estimating a confidence interval to summarize performance does not change the prediction procedure. The distinction is whether an outcome-derived result is fed back to alter the model or the procedure whose performance is being claimed.
A protocol may also use information available before the outcomes to define predictions: for example, dosing records, baseline covariates, or a prespecified rule for handling a sampling design. Such inputs are not automatically updating. But a method that uses a participant’s observed concentrations to estimate that participant’s individual random effects is a different prediction procedure from one that predicts using population information alone. It must be defined in advance and evaluated as that particular procedure; the observations used to personalize predictions cannot be described as unused outcomes for that participant.
Three Data Roles
A defensible workflow assigns each data source a role before examining the outcome results. The roles may correspond to separate studies, separate participants within a study, or both, depending on the intended claim and the data structure.
| Role | Permitted use | What it can support |
|---|---|---|
| Development data | Estimate and select the original model and prediction procedure | Development and internal model-building decisions |
| Updating data | Make prespecified outcome-informed changes to the locked model | Construction of an updated procedure |
| Final evaluation data | Apply the frozen procedure and compare predictions with outcomes; do not feed results back into it | Independent evaluation of that procedure in the stated target setting |
An external study can serve as the final evaluation set for the original model. If its outcomes are then used to recalibrate that model, the study has also become updating data. Its original validation results still describe the original locked procedure, but they do not independently establish performance of the updated one. A new, untouched evaluation set is needed for that latter claim.
Separation must respect the unit of dependence. If one participant contributes multiple concentration records, randomly assigning records from the same participant to both updating and final evaluation sets can leak participant-specific information. Assign the participant as a whole to one role when the intended evaluation is on new participants. If the intended use is instead prediction of later records in already observed participants, define that prospective prediction task and separate information by time accordingly. With several sites or studies, holding out whole sites or studies may be more relevant when the claim concerns transport to new settings.
Worked Example: Evaluating a Locked Prediction Procedure
Suppose a hypothetical external PK dataset contains four independent participants. A locked model predicts a concentration at a prespecified time, and the observed concentrations are \(9, 9, 13,\) and \(15\) mg/L. Its predictions are \(8, 10, 12,\) and \(14\) mg/L. The prediction method and time point were fixed before the outcomes were inspected.
Define each error as prediction minus observation. The errors are \(-1, 1, -1,\) and \(-1\) mg/L. For this small illustrative dataset, the mean absolute error (MAE) is the average absolute error, and the root mean squared error (RMSE) is the square root of the average squared error:
These calculations summarize the locked procedure’s predictions in this hypothetical external sample. Since neither the model nor the calculation choices were selected using these four outcomes, reporting the metrics is validation, not updating. The results do not, by themselves, establish performance in every population or sampling design.
Why Reusing Outcomes Creates Optimism
The mechanism can be seen with a simple calibration update. Let \(P_i\) denote a model prediction and \(Y_i\) the corresponding observed outcome. Suppose an analyst adds one constant offset \(a\) to every prediction and chooses \(a\) to minimize the sum of squared errors in the same dataset:
Differentiating and setting the derivative to zero gives the offset that minimizes this training-data criterion:
Because \(Q(\hat{a})\) is the minimum over all possible offsets, it cannot exceed the unadjusted squared-error total \(Q(0)\). This is a mathematical property of fitting on those same data, not evidence that the adjusted procedure will perform better on new participants. The offset has been chosen to reduce discrepancies that happened to occur in the updating sample; some of those discrepancies may be specific to that sample.
Worked Example: A Calibration Offset Is an Update
Use the four hypothetical participants above. Their observed-minus-predicted differences are \(1,-1,1,\) and \(1\) mg/L. The offset minimizing squared error in these data is:
Adding \(0.50\) mg/L to each prediction gives \(8.5, 10.5, 12.5,\) and \(14.5\) mg/L. The resulting errors (prediction minus observation) are \(-0.5, 1.5, -0.5,\) and \(-0.5\) mg/L. Thus:
The adjusted procedure has smaller in-sample MAE and RMSE here. But the same outcomes determined the offset and assessed the result. The metrics describe how the fitted offset behaves in its updating data; they are not independent validation of the updated procedure. The unadjusted results remain the validation results for the original locked procedure.
A Defensible Updating and Evaluation Workflow
Updating can be scientifically useful, especially when a model shows systematic miscalibration in a new setting. The goal is not to avoid updating; it is to keep the resulting claims aligned with how the data were used.
State the population, prediction time, available inputs, and prediction method. Distinguish population predictions from any procedure using participant-specific observations.
Record its parameters and analysis choices before examining external outcomes. Apply it as specified and report its performance as validation of the original procedure.
If changes are warranted, specify which outcomes may be used, what may be adjusted, and the rule for making those adjustments. Do not use final evaluation outcomes to choose the update.
Record the resulting model, parameter values, inputs, and prediction rules before applying it to final evaluation data.
Apply the frozen updated procedure to the final evaluation set and report performance there. If those results prompt further tuning, they become updating information for the next version.
The procedure is strongest when the final evaluation data are separate at the level relevant to the claim, and when no outcome-informed choices are made after those data are opened. If data are scarce and a truly untouched set is unavailable, resampling or cross-validation can estimate performance under defined procedures, but it does not turn a repeatedly reused external dataset into an independent external evaluation. The limitations and exact data reuse should be stated plainly.
Diagnostics, Alternatives, and Common Pitfalls
A practical audit is to trace every decision back to its information source. Ask whether the prediction, model choice, exclusions, or subgroup definition would have been the same if the external outcomes had remained hidden. If not, the decision is outcome-informed and the data have contributed to updating or selection.
- Check version control: compare the model specification and analysis plan dated before outcome access with the final analysis. Record any deviations and why they occurred.
- Check the data flow: document which participants, studies, or records informed fitting, updating, and final evaluation. Look for repeated participants or shared information across roles.
- Check for outcome-driven selection: repeated metric calculations are not inherently updating, but choosing the best-looking model or subgroup after seeing those metrics is.
- Report each task separately: label original-model validation, updated-model development, and final evaluation distinctly. Do not present updating-set performance as though it were an independent result.
The main alternatives are not interchangeable. Locked external validation estimates performance of the original procedure in a new dataset. Updating followed by held-out evaluation estimates performance of the updated procedure in separate data. Resampling-based evaluation can assess a specified model-building workflow within available data, subject to its assumptions and design; it does not provide the same evidence as an independent external study. The appropriate choice depends on whether the scientific question concerns transport of the original model or performance after adaptation.
Common mistake: “The external RMSE was high, so we recalibrated the model on those outcomes; the lower recalibrated RMSE proves the updated model is validated.” This confuses fitting with evaluation. The lower value is expected to some degree because the same outcomes selected the offset and measured its fit. The correction is to report the original model’s external validation separately, describe the recalibration as updating, and evaluate the frozen updated model in data that did not inform that update.
Check Your Understanding
Answer each question, then compare with the solutions below.
- An analyst estimates a prediction offset from an external study’s outcomes, then reports the offset model’s error on that same study. What role did the study play?
- A participant contributes six concentration samples. Why might assigning three records to updating and three to final evaluation be inappropriate for evaluating predictions in new participants?
- A locked model’s predictions are compared with outcomes using a prespecified metric. No model choice or analysis change follows. Is this updating?
- After selecting a covariate using external outcomes, the analyst evaluates the selected model in the same external cohort. What is the limitation of that reported performance?
Solutions. (1) The study served as updating data because its outcomes determined the offset. Its same-sample error is not an independent evaluation of the updated procedure. (2) Records from one participant can share participant-specific information; splitting them can leak information across roles and does not mimic evaluation in new participants. Assign the participant wholly to one role for that target. (3) No. This is validation of the locked procedure if the metric and analysis were specified independently of the outcomes and no outcome-informed change is made. (4) The cohort both selected and evaluated the covariate choice, so the reported performance can be optimistic and is not independent validation of the selected model. Evaluate that frozen choice in untouched data for an independent assessment.