What a Failed Validation Does—and Does Not—Show
A validation failure means that a prespecified prediction criterion was not met for the data and conditions evaluated. It does not, by itself, identify why. In a PBPK analysis, a discrepancy could arise from a data-processing mistake, an input or protocol mismatch, uncertainty in measured quantities, or a limitation in the model’s representation of physiology or drug behavior. These explanations call for different responses.
The first task is therefore not to adjust a parameter until the observed and predicted values agree. It is to establish whether the comparison was assembled and calculated correctly. If it was, the discrepancy is evidence about the model’s predictive performance under the tested conditions. That evidence must be reported, even if development work later produces an improved model.
These categories are about the explanation for the discrepancy, not its size. A very large difference may be caused by a simple unit error; a modest but systematic difference may reflect a real limitation. The distinction should be supported by traceable evidence, not chosen because one explanation makes the model look better.
A Controlled Response to Failure
A sound response preserves the original prediction, the validation result, and the ability to evaluate a revised model fairly. Before any model change, retain the prediction files, analysis code, input data as received, and the record of when predictions were finalized. Check data provenance, units, time alignment, analyte and matrix definitions, dose and route, protocol deviations, and the prespecified endpoint calculation. The earlier material on missing samples and protocol deviations is relevant when those issues affect which observations can be compared.
If a verifiable processing error is found, correct it by a rule justified from source documentation, not by selecting the correction that best matches the prediction. Record the original value, corrected value, reason, source, and impact. Recalculate the comparison transparently. If the correction required choices informed by the observed outcomes, or if it changes the analysis pipeline after those outcomes were examined, do not present that same dataset as an untouched independent test of a revised model.
If the data and analysis are sound, treat the result as a prediction failure. Diagnose patterns in context: whether the discrepancy is consistent across times, endpoints, doses, or subgroups; whether it is an overall bias or a time-dependent mismatch; and whether the inputs used for the prediction match the test conditions. These checks help formulate a scientific explanation. They do not license selecting a model revision on the basis of the test set and then reporting that same set as evidence that the revision works.
Preserve the original predictions, observed data as received, code version, and analysis specification. Keep the failed result visible.
Trace data and calculations to source records. Correct only errors supported by independent documentation, and report both the original and corrected analyses when relevant.
If the comparison is valid and the criterion is not met, document it as a prediction failure and investigate plausible causes without hiding the result.
Use data designated for model development, with a stated rationale for the change. Record which structures, inputs, or parameters changed and why.
Prespecify predictions and criteria for data not used to diagnose, tune, or select the revised model. Report the earlier failure as part of the model history.
Worked Examples
Worked Example: Finding a Unit-Conversion Error
Suppose a hypothetical validation compares predicted AUC of \(6.0\ \mathrm{mg\cdot h/L}\) with an imported observed AUC recorded as \(6000\ \mathrm{\mu g\cdot h/L}\). The prespecified acceptable range is a fold error no greater than 1.25. For a positive observed-to-predicted ratio \(R\), define fold error as \(FE=\max(R,1/R)\).
If the imported number is mistakenly treated as \(\mathrm{mg\cdot h/L}\), then \(R=6000/6.0=1000\), and \(FE=1000\). That comparison appears to fail dramatically. The source report, however, identifies the observed unit as micrograms, not milligrams. Since \(1000\ \mathrm{\mu g}=1\ \mathrm{mg}\), the correctly converted AUC is \(6000/1000=6.0\ \mathrm{mg\cdot h/L}\).
After correction, \(R=6.0/6.0=1.00\) and \(FE=1.00\), within the stated limit. This is evidence of a data-processing error, not evidence that the PBPK model needed a 1000-fold adjustment. The correction is defensible because the unit is verified from the source record. The audit trail should preserve the imported value, the conversion, and the reason. If investigators had chosen a conversion after trying alternatives to improve agreement, the comparison would need to be reported as outcome-informed rather than as an untouched independent evaluation.
Worked Example: Revising a Genuine Underprediction
In a hypothetical test, the model predicts an AUC of \(10.0\ \mathrm{mg\cdot h/L}\) and the observed AUC is \(15.0\ \mathrm{mg\cdot h/L}\). The ratio is \(R=15.0/10.0=1.50\), so \(FE=\max(1.50,1/1.50)=1.50\). With a prespecified maximum fold error of 1.25, this prediction fails. A source-data audit finds no unit, timing, or endpoint error.
The team then examines separate development data designated for model refinement. In three hypothetical comparable development datasets, observed-to-predicted AUC ratios are 1.30, 1.20, and 1.25. A simple shared multiplicative correction, if scientifically justified by the model context, can be estimated by their geometric mean:
The development evidence suggests a correction of about 1.25 for the specified prediction context. The team must still justify what model component could produce that correction; multiplying all predictions by 1.249 without a scientific rationale is not automatically a valid PBPK revision. Most importantly, applying this factor to the failed test prediction would give \(10.0\times1.249=12.49\ \mathrm{mg\cdot h/L}\), but that value is not a new validation result. The observed test value of \(15.0\ \mathrm{mg\cdot h/L}\) has already been examined and used to motivate revision. It remains a documented failure of the original model, not an independent test of the revised one.
Worked Example: Evaluating the Revised Model on New Data
Suppose the revised model is frozen before a new, hypothetical independent evaluation. Three new observed and predicted AUC pairs, all in \(\mathrm{mg\cdot h/L}\), are: observed 13 and predicted 12; observed 19 and predicted 20; observed 10 and predicted 8. The prespecified rule is that each pair must have \(FE\leq1.25\).
For the first pair, \(R=13/12=1.083\), so \(FE=1.083\). For the second, \(R=19/20=0.950\), so \(FE=\max(0.950,1/0.950)=1.053\). For the third, \(R=10/8=1.25\), so \(FE=1.25\). Each pair meets the stated per-pair criterion, including the third pair at the boundary.
The correct conclusion is limited: all three predictions meet this particular prespecified rule in this hypothetical evaluation. Three pairs alone do not establish performance across all populations, protocols, or drugs. The report should state the scope of the tested conditions and the sample size, rather than generalizing beyond the evidence.
Why Reusing the Test Set Creates Optimism
A test set is independent evidence only while its outcomes are not used to choose the model being evaluated. The reason is straightforward. Suppose a team considers several candidate revisions and calculates each candidate’s error on the same test set. Even if every candidate has the same true predictive quality, selecting the candidate with the smallest observed test error tends to select one that benefited from random agreement with that particular set. The reported minimum error is then not an unbiased estimate of future performance.
For a short justification, let \(E_j\) be the test error for candidate \(j\), and let \(j^\ast\) be the candidate chosen because its observed error is smallest. Then, for every candidate \(j\), \(\min_k E_k\leq E_j\). Consequently, the selected error is no greater than the error of any fixed candidate on that same test set. When errors vary partly because of sampling variability, selection favors a low realization. This inequality does not say that every selected model is worse; it explains why its selected test error cannot be treated as a fresh, independent estimate.
The appropriate alternative is to separate roles. Development data may be used to diagnose, estimate, and select revisions. New independent data are then used to evaluate the frozen revision. If no new data are available, the honest conclusion is that the revised model has not yet received a new independent validation. A development fit, an internal resampling exercise, or a corrected reanalysis may provide useful information, but it does not become an independent test merely by being labeled one.
Choosing Data and Documenting the Revision
Development data should be relevant to the failure being addressed. A dataset that informs a suspected absorption issue may be more useful for revising an oral prediction than one that only informs systemic elimination. Relevance alone is not enough: the data’s role must be explicit, and the model change should be constrained by scientific knowledge and the intended prediction context. Avoid adding multiple unsupported adjustments just because they improve agreement with the failed test.
A reproducible record identifies the original model version and failure; the data used for development; the proposed mechanism; every changed assumption, input, or parameter; and the resulting model version. It also states which data were used in any choice and which new data were reserved. A predictive criterion and analysis plan should be fixed before new outcomes are inspected. If the revised model fails again, that result is also evidence to report, not a reason to repeatedly relabel the same data as a new test.
| Situation | Appropriate interpretation | Next step |
|---|---|---|
| Traceable unit or processing mistake | The original comparison was assembled incorrectly; document its correction and impact. | Recalculate transparently; reserve new independent data if outcomes informed choices or the model is revised. |
| Valid comparison that misses the criterion | The model failed for the tested conditions under the prespecified rule. | Diagnose and revise using suitable development data, then test on new independent data. |
| Revision selected after inspecting test outcomes | The inspected set is no longer an independent test of the selected revision. | Report the selection history and obtain a new independent evaluation. |
Common Pitfall and Limitations
A common mistake is: “The model missed the test observations, so I will adjust the parameter until those observations fall inside the acceptance range, then report that the model passed.” The correction is that this procedure uses test outcomes for model selection. Agreement after adjustment describes a fit to the inspected data, not independent predictive performance. The original failure must remain in the record, and the revised model needs new independent data for evaluation.
Not every discrepancy can be resolved by a simple recalibration. Different mechanisms can produce similar AUC errors, and an adjustment that improves one endpoint can worsen a concentration-time profile or another population’s predictions. A development dataset may also be too small or too narrow to support the proposed change. The response should therefore distinguish what the available evidence establishes from what remains uncertain. A data audit can identify processing mistakes; it cannot prove that the model is generally valid. Likewise, passing a prespecified criterion on new data supports performance only for the tested conditions and endpoints.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A test prediction misses its criterion. The source file shows that the observed concentration was entered in micromoles per litre but treated as millimoles per litre. What should be checked and reported before any model parameter is changed?
- After seeing test outcomes, a team chooses the parameter set with the lowest error on that test set. Can the same test set independently validate the selected parameter set? Why?
- A revised model is estimated using a development dataset and has not yet been evaluated on new data. What can be claimed about its validation status?
- For observed AUC \(8.0\ \mathrm{mg\cdot h/L}\) and predicted AUC \(10.0\ \mathrm{mg\cdot h/L}\), calculate \(R\) and \(FE\). Does it meet a maximum fold-error rule of 1.25?
Solutions. (1) Verify units against source documentation and apply the documented conversion consistently. Preserve the original entry, corrected value, reason, and recalculated result; do not change a model parameter to compensate for a unit mistake. (2) No. The outcomes were used to select the parameter set, so the set is development evidence for that selection, not an independent test of the selected model. New independent data are needed. (3) The model has been revised using development data, but it has not yet received a new independent validation. (4) \(R=8.0/10.0=0.800\), and \(FE=\max(0.800,1/0.800)=1.25\). It meets the rule exactly at the boundary, assuming the rule includes values equal to 1.25.