Why Reproducible Reporting Matters
An independent validation is informative only when a reader can tell what was predicted, which data were used to evaluate those predictions, and whether the model was altered after the outcomes became available. A concise statement that a PBPK model “performed well” is not enough: another analyst needs the evidence and analysis choices that make the conclusion checkable.
This tutorial focuses on reporting the validation analysis, not on tuning a model to improve its fit. The introductory material on independent validation distinguishes evaluation data from data used to develop or calibrate a model. Here, that distinction becomes an auditable record: the report should show how independence was assessed, what was frozen before evaluation, how endpoints and exclusions were handled, and how the results support the stated conclusion.
The Record That Makes a Validation Auditable
Begin with a clear account of the evaluation data. Identify the study or studies, protocol, population, sites, dates or data cut, assay and measurement units, and the source of the analysis dataset. State the number of participants, samples, and evaluable endpoint records as applicable. A database name alone is not sufficient: include the dataset version or extraction date and how the data were transformed into the analysis file. Where permitted, a file checksum or another stable identifier can help verify that two analysts used the same file.
Independence requires a substantive check, not simply a label such as “external.” Document whether participants, outcome records, or summary results overlap with model development, calibration, or prior model revisions. Describe the check—for example, linkage using study identifiers or a documented subject crosswalk—and its result. If some overlap cannot be ruled out, state what is known and what remains uncertain. Shared assay methods, clinical practices, or a site do not by themselves prove that outcomes were reused, but they can limit how broadly the validation result should be generalized.
Next identify the exact model used to generate predictions. Record its version or immutable identifier, parameter set, structural assumptions relevant to the prediction, input files, software and version, solver settings, and any scripts or processing steps. State when the version was frozen relative to access to the validation outcomes. If predictions were generated in a blinded process, describe how the prediction file was preserved and linked to the final analysis dataset. A reproducible record should make it possible to distinguish a prediction generated before evaluation from one regenerated after results were known.
Prespecification should be visible in the report. Name each primary and secondary endpoint, its definition, units, time point or interval, analysis population, and the rule for handling unavailable or unevaluable values. Give the date and location of the protocol or analysis plan, or reproduce the relevant rules. Criteria chosen after seeing the prediction errors are descriptive, not a prospective pass/fail standard.
Identify the source, version, and construction of the evaluation dataset; count the records relevant to each endpoint.
Describe potential overlap with development or calibration data, the check performed, and any unresolved uncertainty.
Record the model, parameters, software, and computational settings used to generate the predictions.
Define endpoints, diagnostics, exclusions, and missing-data handling before interpreting the validation outcomes.
Show the counts, diagnostics, deviations, and a conclusion limited to the population and conditions evaluated.
Define the Prediction-Error Quantities
For a paired observation \(i\), let \(y_i\) be the observed endpoint and \(\hat y_i\) the corresponding PBPK prediction, with both expressed in the same units. The report must say how predictions were matched to observations—for example, by participant and scheduled sampling time—because an incorrect match can create apparently plausible but invalid diagnostics.
These log-scale quantities have a useful derivation. Since \(e_i=\ln(\hat y_i/y_i)\), averaging log ratios gives \(\bar e=\frac{1}{n}\sum_i \ln(\hat y_i/y_i)\). Exponentiating converts that average back to the ratio scale: \(B=\exp(\bar e)\). Taking absolute log errors before averaging removes direction, and exponentiating gives \(G\). Thus bias and typical error answer different questions: overprediction and underprediction can partly cancel in \(B\), while neither cancels in \(G\).
These definitions require positive paired values. Do not apply logarithmic ratios directly to zero or below-quantification observations; report their handling under the prespecified endpoint rules and use suitable diagnostics for those data. Also report the number of paired records and their units of analysis. Multiple samples from the same participant are not automatically independent observations, so a simple count of samples should not be presented as a count of independent participants.
Worked Example: Calculating Prediction-Error Diagnostics
Suppose a hypothetical independent study has three paired concentrations, all in mg/L. The observed values are 10, 20, and 40 mg/L; the frozen model predicts 12, 18, and 60 mg/L. All values are positive and matched at the prespecified times, so the log-ratio diagnostics apply.
The prediction ratios are \(12/10=1.20\), \(18/20=0.9000\), and \(60/40=1.50\). The corresponding log errors are \(\ln(1.20)=0.1823\), \(\ln(0.9000)=-0.1054\), and \(\ln(1.50)=0.4055\).
The mean log error is \(\bar e=(0.1823-0.1054+0.4055)/3=0.1608\). Therefore, the multiplicative bias factor is \(B=\exp(0.1608)=1.174\). In this hypothetical set, the geometric mean prediction is about 1.174 times the observed concentration on the multiplicative scale, indicating modest average overprediction.
The geometric mean fold error is \(G=\exp((0.1823+0.1054+0.4055)/3)=\exp(0.2311)=1.260\). The individual fold errors are 1.20, 1.111, and 1.50, so all three are within two-fold. A report should show the number of pairs, the units, and the endpoint and time context; these three values alone do not establish performance in other populations or protocols.
Exclusions, Missing Values, and Diagnostics
A reproducible report gives a flow from scheduled records to analyzed pairs. For each endpoint, state how many observations were expected, available, excluded, and included. For every exclusion, give the prespecified reason and its frequency. Distinguish an unavailable observation, such as a missed sample, from an observed value that was excluded by a defined assay or protocol rule. Do not omit a record merely because its prediction error is large. If a deviation from the plan was necessary, identify it, explain when and why it occurred, and present its potential effect on interpretation.
Diagnostics should make the pattern of error inspectable, not just reduce it to a single summary. Report observed-versus-predicted displays and time-structured residual checks where relevant, alongside numerical summaries. Describe whether errors appear directional, concentrated at particular times, or associated with a prespecified subgroup or protocol condition. These are descriptive findings unless the analysis plan specified a formal inferential procedure. A favorable average can conceal a consistent miss at a clinically important time point.
Check that units, transformations, and endpoint definitions agree between observed and predicted data. Confirm that predictions are paired to the intended subject, dose, and sampling time, and that any rounding or aggregation is documented. If diagnostics look implausible, investigate data mapping, units, model inputs, and execution records before interpreting a biological explanation. Corrections must be traceable; silently replacing a prediction file or changing a matching rule breaks the audit trail.
Worked Example: Reporting a Missing Scheduled Sample
In a hypothetical validation, four samples were scheduled for one endpoint. The first three had observed concentrations of 10, 20, and 40 mg/L and frozen predictions of 12, 18, and 60 mg/L. The fourth sample was missed, although the model had generated a predicted concentration of 30 mg/L for its scheduled time. The analysis plan specified that this endpoint required a paired observed and predicted value and did not specify imputation.
The fourth record cannot contribute to a paired prediction-error calculation: there is no observed value \(y_4\), so \(r_4=\hat y_4/y_4\) is undefined. The correct accounting is four scheduled records, three paired records included, and one unavailable observed sample. The prediction of 30 mg/L can remain in the archived prediction file, but it is not evidence of prediction accuracy for that record.
For the three included pairs, the calculations from the preceding example give \(B=1.174\), \(G=1.260\), and three of three fold errors within two-fold. A reproducible report states both the results and the missingness: for example, “Three of four scheduled observations were evaluable; the fourth was unavailable because the sample was missed. No imputation was performed, as specified in the analysis plan.” It should not imply that the fourth prediction was validated.
From Results to a Bounded Conclusion
A conclusion should connect the result to criteria set before the validation outcomes were examined. It should state which endpoints met which criteria, which did not, how many records contributed, and the validation conditions to which the conclusion applies. Meeting a criterion does not establish that the model is correct in every setting. Conversely, failure to meet a criterion does not by itself identify whether the source is an implementation error, measurement limitation, model discrepancy, or a mismatch between the evaluated protocol and the intended use.
The reporting approach also differs from recalibration. In independent validation, the frozen predictions are evaluated as generated; changing parameters using the evaluation outcomes turns those outcomes into development information. A revised model may be worth investigating, but its apparent performance on the same data is no longer an independent validation of that revision. Report the original evaluation first and preserve the distinction between an independent assessment and later exploratory work.
Worked Example: Writing a Criteria-Based Conclusion
Suppose an analysis plan for a hypothetical concentration endpoint prespecified three criteria: geometric mean fold error \(G\leq1.50\), multiplicative bias factor between 0.8000 and 1.250 inclusive, and at least 80% of paired predictions within two-fold. In the three-pair example, \(G=1.260\), \(B=1.174\), and three of three pairs, or \(100\%\), were within two-fold.
Each criterion is met: \(1.260\leq1.50\), \(0.8000\leq1.174\leq1.250\), and \(100\%\geq80\%\). A suitably bounded conclusion is: “For the prespecified concentration endpoint in this hypothetical evaluation, all three prediction-error criteria were met among three evaluable paired samples. One scheduled sample was unavailable. These results describe performance under the evaluated protocol and do not establish performance in other populations or settings.”
The small number of paired samples is essential context. The criteria were met for the observed records, but the result provides limited information about the range of errors that might occur in a larger or different evaluation. The report should not turn a threshold-based conclusion into a claim of universal validity.
Common Reporting Pitfalls
| Incorrect approach | Why it is inadequate | Better reporting practice |
|---|---|---|
| Calling a dataset independent without describing an overlap check | The reader cannot assess whether evaluation outcomes informed development. | Identify potential sources of overlap, the check performed, and its result or limitations. |
| Giving a model name but no version or computational details | A different parameter file or software setup can produce different predictions. | Record a stable model identifier, inputs, software, and relevant settings. |
| Changing exclusions after inspecting errors | The analyzed set can be selected to make performance look better. | Use prespecified rules, report deviations, and account for all records. |
| Reporting only an overall error summary | Opposing biases or time-specific failures may be hidden. | Pair numerical diagnostics with plots, counts, and endpoint context. |
| Claiming the model is validated everywhere because criteria were met | The evidence applies only to the evaluated population, protocol, endpoints, and data. | State the scope and limitations of the evaluation explicitly. |
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- What evidence should a report provide to support the claim that evaluation outcomes were independent of model calibration?
- Why is a predicted value without its paired observation not included in a paired log-ratio diagnostic?
- A report gives \(B=0.90\) and \(G=1.40\). What does each quantity describe, and why are they not interchangeable?
- What should be reported if a sample is excluded because of a protocol deviation that was not specified in the analysis plan?
- If a model meets prespecified error criteria in one study, what is an appropriately scoped conclusion?
Solutions. (1) Identify development and evaluation data sources, describe plausible participant or outcome overlap, state the linkage or other check performed, and report its result and any unresolved uncertainty. (2) The ratio requires both values: \(r_i=\hat y_i/y_i\). Without \(y_i\), the ratio and its logarithm are undefined, so the prediction is not an observed validation pair. (3) \(B=0.90\) indicates average multiplicative underprediction, while \(G=1.40\) summarizes the average magnitude of multiplicative error without direction; opposite-signed errors can reduce bias without reducing error magnitude. (4) Report the record count, the specific deviation and reason, when it was identified, the rule applied, and any relevant sensitivity or descriptive analysis without hiding the deviation. (5) State that the criteria were met for the specified endpoints and evaluable records under the tested population and protocol; do not claim validity for untested settings.