When Validation Stops Being Independent
A PBPK model is validated by comparing its predictions with data that did not inform the model or the choices used to evaluate it. The comparison is useful only if those choices are protected from the validation outcomes. Leakage occurs when information from validation data, directly or indirectly, influences a modeling or reporting decision, and the same data are then presented as an independent assessment.
Leakage is not limited to fitting a parameter to every validation observation. It can arise when a modeler adjusts one physiological parameter after seeing a concentration profile, selects the endpoint that gives the most favorable result, removes observations because they disagree with predictions, or revises the model after inspecting the validation residuals. Each action makes some aspect of the reported result depend on the data being used to judge it.
The earlier material on calibration and validation distinguishes the roles of those activities; material on participant, study, and measurement independence addresses dependence among observations. Here the concern is a different kind of independence: whether validation results have influenced the process being evaluated. A dataset can be independent in its sampling units yet no longer be an untouched validation set after test-informed choices.
How Leakage Creates Optimism
Suppose two candidate models have the same true mean validation loss, but their observed losses differ because a finite validation set gives noisy estimates. If the lower observed loss is used to select the model, that same set is no longer an independent check of the selection. A short derivation shows the source of the optimism.
Write the observed loss for candidate \(j\) as \(L_j=\mu+\epsilon_j\), where \(\mu\) is the shared true mean loss and \(\epsilon_j\) is a mean-zero estimation error. For two candidates, the selected observed loss is
Taking expectations and using \(E(\epsilon_1)=E(\epsilon_2)=0\) gives
The inequality is strict when the two errors differ with positive probability. Thus, even when the candidate models are equally good on average, selecting the one with the smaller observed loss makes its loss on the selection data look favorable on average. Comparing that selected model to the same data does not remove this selection effect. A separate, untouched dataset is needed for an independent evaluation of the selected model.
Worked Example: Adjusting a Parameter After Seeing a Profile
Hypothetical example. A team freezes a PBPK model, then compares it with three validation concentrations in mg/L. The observed concentrations are 10, 5, and 2.5; the frozen model predicts 8, 4, and 2. For clarity, calculate the root mean squared error (RMSE), the square root of the mean of the squared observed-minus-predicted differences.
For the frozen predictions, the RMSE is
After viewing the profile, a modeler changes a clearance-related parameter until the predictions are 9, 4.5, and 2.25 mg/L. The resulting RMSE is
The lower RMSE describes agreement after the parameter was adjusted using these observations. It is not an independent validation result. The model may have improved, but this dataset has now helped calibrate it; an untouched dataset is needed to evaluate the adjusted model independently.
Leakage Through Evaluation and Data Decisions
A validation plan includes more than a parameter set. It also defines what predictions will be evaluated, how source data will be transformed, which observations meet quality criteria, and which model version will generate the predictions. If these decisions change in response to observed validation results, the interpretation of the resulting assessment must change too.
Worked Example: Choosing the More Favorable Endpoint
Hypothetical example. Before seeing validation outcomes, a team specifies AUC from 0 to 24 hours as the primary endpoint. The PBPK model predicts an AUC of 90 mg·h/L; the observed AUC is 120 mg·h/L. The prediction is 30 mg·h/L below the observation, or
The team also records a secondary maximum concentration, \(C_{\max}\). The predicted value is 8.4 mg/L and the observed value is 8.0 mg/L, so the relative difference is
If the team highlights \(C_{\max}\) only after seeing that it agrees more closely, it has selectively emphasized a favorable result. The computed differences are correct, but choosing the endpoint after reviewing the outcomes changes the evidential meaning of the report. The prespecified AUC result remains relevant; the secondary result can also be reported, with its status and the post-outcome selection made clear. Neither should be presented as though the endpoint choice were independent of the data.
Worked Example: Excluding a Discrepant Observation
Hypothetical example. Four observed concentrations are 1.1, 2.2, 4.4, and 16 mg/L, while the corresponding predictions are 1, 2, 4, and 8 mg/L. Suppose the last observation is removed solely because it strongly disagrees with the model. With all four observations, the RMSE is
After removing the last observation, the calculated RMSE is
The sharp decrease is real arithmetic, but it does not justify the exclusion. Removing the point because it disagrees with the prediction uses the outcome to define which outcomes count. An exclusion can be defensible when it follows a documented quality rule applied without regard to model agreement—for example, a prespecified, independently verified assay failure. The reason and rule should be recorded, and results should be reported transparently. A mismatch alone is not evidence that a measurement is invalid.
Worked Example: Revising the Model After Validation
Hypothetical example. A frozen PBPK model predicts concentrations of 4, 8, and 16 mg/L, compared with observed values of 5, 10, and 20 mg/L. Its RMSE is
After inspecting these results, the team revises the model and obtains predictions of 4.8, 9.6, and 19.2 mg/L. The RMSE on the same observations becomes
The revised predictions agree more closely with the data that prompted the revision. That is evidence of a potentially useful model-development step, not independent validation of the revised version. The original model’s comparison can be reported as the prespecified validation result, with the revision disclosed as subsequent development. The revised model requires a new untouched evaluation before it can be described as independently validated.
Safeguards Before, During, and After Evaluation
The practical objective is to prevent outcome information from reaching people or decisions that could alter the prediction or its evaluation. Safeguards should be established before validation outcomes are reviewed. They should cover not only parameter fitting, but also data handling, endpoint definitions, code, model revisions, and reporting.
Document the model version, parameter values, input assumptions, prediction times, endpoints, units, and planned summaries before outcome access. Preserve the corresponding model files and record their version or checksum.
Specify transformations, unit conversions, missing-data rules, and quality exclusions in advance. Keep the rationale and source for each exclusion. Apply the rules consistently, not selectively to improve agreement.
Where feasible, a custodian can provide model inputs while withholding outcomes. Generate and time-stamp predictions, lock the prediction file, and retain a record of who had access to each file and when.
Run the prespecified evaluation, retain the complete result set, and document deviations. Avoid changing endpoints, exclusions, or model settings in response to the observed agreement.
If validation outcomes guide a parameter choice, model revision, or evaluation decision, treat the affected data as development information for that decision. Use a genuinely untouched dataset to assess the resulting version independently.
Diagnostics, Alternatives, and Limits
A useful leakage audit asks what information each decision-maker could see when making each consequential choice. Compare dated prediction files, model versions, analysis plans, exclusion records, and communications. Look for parameters changed after results became available, endpoints appearing only in the final report, exclusions without outcome-independent justification, or code changes made after inspecting mismatches. A change after outcome access is a warning to investigate, not automatic proof of misconduct: a documented correction to a software error may be necessary, but the resulting reanalysis should not be portrayed as an untouched assessment if the validation outcomes informed that correction.
There are two legitimate roles for data, and they should not be confused. Calibration data can support parameter estimation and model development. An untouched validation set can assess a model whose relevant choices have already been fixed. Repeatedly examining one validation set while making changes turns it into development data for those changes. Likewise, internal resampling can help compare or tune models within a development dataset, but it does not make a dataset independent if results from it have already guided the final choices. A fresh, untouched assessment is the alternative when independent evaluation is the goal.
No safeguard guarantees that a model is biologically correct, that the validation data represent every intended use, or that recorded measurements are error-free. Prespecification and access controls address information flow; they do not replace sound scientific reasoning or transparent reporting. If leakage occurs, it cannot be undone by hiding the decision or by computing another summary on the same outcomes. State what was seen and changed, distinguish the development comparison from independent evaluation, and avoid describing the affected results as untouched validation.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A team changes a PBPK parameter after reviewing validation concentrations, then reports the improved fit on those same concentrations. What is the problem, and what should the team do next?
- A prespecified primary endpoint agrees poorly, while a secondary endpoint agrees well. When may the team report the secondary endpoint, and what must it disclose?
- An observation fails a laboratory quality criterion written before outcome review. Is excluding it necessarily leakage? What should be documented?
- Why can choosing the candidate with the smallest observed validation loss produce an optimistic estimate of its performance?
- A team discovers a software error after seeing validation results and corrects it. Can the corrected model’s results on the same data be called an untouched validation?
Solutions. (1) The validation data informed the parameter choice, so the improved fit is development evidence, not independent validation; evaluate the revised model on untouched data. (2) It may report the secondary endpoint, but should label it as secondary and disclose that its selection followed outcome review; it should not substitute for the prespecified primary result. (3) Not necessarily: a prespecified, outcome-independent quality rule can justify exclusion. Record the rule, evidence that it was met, and the exclusion. (4) Noisy observed losses vary; selecting the minimum tends to select a favorable noise component, as shown by the negative expected minimum error in the derivation. (5) No. Because the outcome data informed the correction, the corrected model’s comparison is not an untouched assessment; disclose the correction and use fresh data for independent validation.