Reporting Verification and Validation Evidence

Why Verification and Validation Evidence Must Be Traceable

A PBPK submission should allow a reviewer to follow a chain from the model that was run to the evidence used to judge it. That chain includes the implementation and simulation records, the checks performed on the model, the data used for evaluation, the comparisons made, and the criteria used to interpret the results. A statement such as “the model was verified and validated” is not enough on its own: it does not identify what was checked, which evidence supports the statement, or whether that evidence is relevant to the intended use.

In this tutorial, verification means checking that the mathematical model has been implemented and numerically solved as intended. Validation means evaluating whether model predictions are adequate for a stated purpose by comparing them with relevant observations and other evidence. These activities answer different questions. Correct software execution does not establish that the model represents biology adequately; agreement with observations does not, by itself, show that the implementation is free of coding or numerical errors.

The earlier material on decision-relevant acceptance criteria established that criteria should be connected to the decision the model is intended to support. Reporting should therefore make the link explicit: identify the criterion, show the evidence to which it applies, report the result, and state what conclusion that result supports. Avoid implying that a favorable check proves more than it actually tests.

What a Reviewable Evidence Record Contains

For each check or evaluation, report enough information for a reviewer to understand and, where practical, reproduce the result. The record should identify the model version and simulation configuration; the component, equation, or output being examined; the test conditions and inputs; the observed or reference result; the computed result; the acceptance criterion; and the outcome. Link records to the relevant software configuration and simulation files described in the earlier material on reproducible simulation records.

For validation evidence, also identify the evaluation dataset and its role. State the study or data source, population and conditions, measured endpoint, units, sampling times, and any exclusions or transformations. Explain whether the data were used to estimate or tune model inputs. Data used in calibration are not independent evidence of predictive performance: fitting a model to those data and then reporting the fit as an independent validation would overstate the evidence. If data have multiple roles, label them clearly and explain the consequences for interpretation.

Evaluation should match the intended context as closely as the available evidence permits. For example, a dataset collected under a different dose, route, population, or formulation may still provide useful evidence, but the difference and its relevance should be stated. Do not silently treat a related dataset as if it directly represented the target application.

Verification: Implementation Checks and Mass Balance

Implementation checks can include comparison with hand calculations or an independently computed solution in a simplified case, checks of limiting behavior, and confirmation that units, parameter transformations, and boundary conditions are applied as specified. The test should be designed so that an error would be detectable. Reporting only that “code was checked” does not tell the reviewer what was tested or what result was obtained.

Mass balance is a useful diagnostic when the modeled accounting boundary is defined. For a conserved quantity, the change in the total amount within the boundary equals the amount entering minus the amount leaving. Internal transfers between compartments must cancel when the compartment balances are summed. For drug-related quantities, transformations between parent drug and metabolites require care: amounts should be expressed on a consistent basis, such as moles of a tracked moiety or drug-equivalent mass, rather than adding unlike masses without justification.

Here, \(A_i(t)\) is the amount in compartment \(i\), \(I(t)\) is the rate of input across the accounting boundary, and \(O(t)\) is the rate of irreversible output across that boundary. To see why internal transfers do not appear in the total, let \(q_{ij}(t)\) be the transfer rate from compartment \(i\) to compartment \(j\). The sum of compartment equations includes \(-q_{ij}(t)\) for compartment \(i\) and \(+q_{ij}(t)\) for compartment \(j\); their sum is zero. Only boundary input and output remain.

For a finite simulation, compare the total input with the amount retained plus cumulative output. Define the balance residual as input minus accounted amount. Report the residual in the same units as the amount and, where useful, its magnitude relative to input. A tolerance should be stated in advance and justified for the numerical method and application. A residual outside tolerance calls for investigation, not an automatic conclusion that a particular mechanism is missing: possible causes include omitted accounting terms, unit errors, incorrect transfer bookkeeping, or numerical integration error.

Validation Evidence: Evaluation Data and Comparisons

A transparent observed-versus-predicted comparison requires aligned quantities. Specify whether the comparison is between concentration and concentration, exposure summary and exposure summary, or another matched endpoint. Check that units, dose, route, population, sampling times, and relevant study conditions correspond. If predictions are made at observed sampling times, say so. If observations are summarized across participants while predictions represent a different summary, explain the alignment rather than implying a direct like-for-like comparison.

Report the observed and predicted values together, with units and the relevant conditions. Show how the stated acceptance criterion is applied. A criterion might specify an allowed range for a prediction-to-observation ratio or a required pattern across endpoints, but its form should be justified by the context of use and set before judging the results. A particular fold range is not a general regulatory rule. Where observations are variable or sparse, describe the evidence and its limitations; do not let a concise table imply that uncertainty or representativeness has been resolved.

From Results to a Bounded Conclusion

Verification and validation evidence should lead to conclusions that match the scope of the checks. A successful mass-balance test supports the claim that the tracked quantity was adequately accounted for under the tested conditions and tolerance. It does not show that the model’s physiological assumptions are correct. A simplified analytical test supports the claim that the tested implementation behavior matches its reference within tolerance; it does not establish that all model components are correctly implemented. Agreement with an evaluation dataset supports a conclusion about those predictions and conditions, not every possible extrapolation.

Use an evidence table or an equivalent structured presentation to make this reasoning auditable. For each item, include the evidence identifier, purpose, test conditions or dataset, criterion, result, pass or concern, and implication for the intended use. Separate findings from interpretation: first report what the check returned, then state what the result does and does not support. When a criterion is unmet, report it plainly and explain whether the concern changes the decision, requires further evidence, or limits the conclusion.

This reporting is distinct from a full account of predictive performance, which considers broader questions about how predictions behave across clinical evaluations. Here the priority is to show the evidence trail for implementation checks and specified evaluations, without treating a small set of comparisons as a comprehensive characterization of performance.

Diagnostics, Alternatives, and Common Pitfalls

Warning signs include a mass-balance residual that changes with time-step size, unexplained discontinuities, unit inconsistencies, predictions compared at mismatched times, or a dataset whose calibration role is unclear. A large residual should prompt checks of all flows crossing the defined boundary and of the numerical solver settings. A systematic observed-predicted discrepancy should prompt review of data alignment and model assumptions as well as the comparison itself. These signals identify issues to investigate; they do not uniquely diagnose their cause.

Different evidence types are complementary rather than interchangeable. Analytical or limiting-case checks can isolate implementation behavior; mass balance checks accounting and numerical consistency; comparison with observed data evaluates predictions in empirical settings. None replaces the others. Mass balance alone cannot show predictive adequacy, and observed agreement alone cannot rule out compensating coding errors or parameter choices. Likewise, a single successful test cannot justify claims about untested components or contexts.

• Pitfall: “The simulation ran, so the model is verified.” Successful execution only shows that the software completed a run. Report tests against specified references or invariants, with results and tolerances.
• Pitfall: “The predicted values are close, so the model is validated.” “Close” is undefined unless the comparison, conditions, and acceptance criterion are stated. Apply a pre-specified, use-relevant rule and bound the conclusion to the evidence.
• Pitfall: “The fitted dataset confirms predictive ability.” Data used to tune the model do not provide independent confirmation of its predictions. Label calibration and evaluation roles and avoid presenting the same evidence as both without qualification.
• Pitfall: “A small mass discrepancy is harmless.” Size alone does not determine acceptability. Compare the residual with the stated tolerance, investigate its source, and explain the consequence for the tracked quantity.