Why a Simulation Record Needs More Than a Model File
A PBPK result is produced by more than the model equations. The result also depends on the software that interprets them, the model and input files supplied, numerical solver settings, scenario choices, and any code used to prepare or process the output. A model file without this surrounding information may be readable but still insufficient to reproduce the reported result.
A useful simulation record makes the path from a specified set of inputs to a reported output traceable. Its purpose here is reproducibility and reviewability: identifying what was run and how. A record that permits a run to be repeated does not, by itself, show that the model is verified, validated, or suitable for a particular decision.
A practical run record can be represented as a tuple:
Here \(E\) is the software and computational environment, \(M\) is the model-file set, \(D\) is the input-data set, \(C\) is the code and its changes, \(S\) is the simulation configuration, and \(O\) is the saved output and its processing history. The tuple is a checklist, not a mathematical model of drug disposition.
What to Capture in a Run Record
Record the software product, exact version, and—when available—the build or patch identifier. Include the operating system and relevant runtime or dependency versions if they can affect execution. If the software uses a proprietary engine or a managed computing environment, identify it sufficiently for a reviewer to distinguish the environment used from a superficially similar installation.
Identify each model file by its name, version or revision, and location in the submission package. Include linked submodels, scripts, and templates needed to execute the model; do not assume that a top-level project file contains every dependency. Record input datasets in the same way, including their units, population or scenario labels where relevant, and any transformations made before the run. A file’s name alone is weak identification: a file can be replaced while retaining its name.
For stronger identification, record a cryptographic digest, such as a SHA-256 checksum, for each important file. A digest is a compact identifier calculated from file contents; matching digests provide strong evidence that the files are identical. It does not explain what a file contains, so pair it with a clear file name and description. Store the files themselves in a controlled package or archive when permitted, rather than relying on a path to a mutable location.
For a simulation, document the settings that can change the calculated trajectory or reported summaries. These commonly include solver type, absolute and relative error tolerances, maximum step size, simulation start and end times, output sampling times, event handling, and initialization. Also record the simulated population and scenario, dose and administration schedule, and any run-specific switches. If a setting is software-defaulted, say so and identify the software version: defaults can change between releases.
For stochastic procedures, record the random-number generator and seed, if available, as well as the number of draws and relevant sampling settings. A fixed seed may not guarantee identical results across software versions or hardware, so record those too. If an algorithm is nondeterministic or uses parallel computation, identify that limitation and describe the expected comparison standard rather than implying bit-for-bit identity.
Finally, distinguish the model’s code from changes made to it. Record the original revision, each modification, its rationale, and the resulting revision or patch. Include scripts that import data, launch runs, extract outputs, or calculate reportable endpoints. The output record should identify the files produced, the endpoint definitions and units, and any post-processing code used to obtain reported values.
From Configuration to Reproduction
The reason to capture all parts of \(R\) is straightforward. A computational result can be written schematically as \(O = F(E,M,D,C,S)\), where \(F\) denotes execution and output processing. This notation does not assume that the computation is always deterministic; it emphasizes that each listed component can affect what is produced.
Suppose a record omits \(S\). Two analysts might use the same model and dataset but different solver tolerances or dosing schedules. They have not established that they ran the same simulation. If the record omits \(E\), the model may be interpreted by a different software build. If it omits \(C\), an undocumented code edit could account for a changed output. Thus, to attribute an output to a defined computation, the record must identify each potentially relevant component or explicitly state that it was not applicable.
If \(O_1\) and \(O_2\) differ, a comparison of the recorded inputs and settings helps locate the difference. If they agree, the record documents which computation was repeated; agreement alone does not establish scientific correctness.
Worked Example: Building a Run Manifest
Suppose a hypothetical submission reports a simulated steady-state trough concentration of 2.46 mg/L. The analyst needs to define the run well enough for a colleague to reproduce it. The following details are illustrative, not from a real study.
The analyst records PBPK software “ModelSuite,” version 6.2.1, build 1840; operating system “Linux distribution 9.4”; model file “compound_A_project,” revision 3.4; and input file “adult_reference_population.csv,” revision 2. The package gives each file a SHA-256 digest, labelled here with shortened illustrative identifiers: model “a91f…72c0” and dataset “b204…18aa.” These shortened strings are labels for the example, not usable file checksums.
The run-specific record states: adult reference population; oral dose 100 mg every 24 hours; 20 doses; simulation from 0 to 480 hours; solver “stiff solver”; relative tolerance \(10^{-6}\); absolute tolerance \(10^{-9}\) in the software’s documented concentration scale; maximum step 0.25 hours; and trough extraction immediately before each scheduled dose. It also identifies a post-processing script, revision 1.1, and the output file containing the time course.
The result is then traceable: the reported 2.46 mg/L is linked to the specified files, settings, and extraction rule. In this example, the reported time is 480 hours because \(20 \times 24 = 480\) hours. The record should clarify that the predose sample is taken at the dosing event at 480 hours, not at an unspecified nearby output grid point. Without that distinction, the same nominal “trough” could be extracted differently.
Run-to-Run Comparisons and Numerical Tolerance
A rerun may not produce bit-for-bit identical files, even when the scientifically relevant output is effectively unchanged. Differences can arise from floating-point arithmetic, software implementation, hardware, parallel execution, or output formatting. A record should therefore specify whether exact identity is expected or whether numerical agreement is judged using a stated tolerance.
One common comparison rule for a scalar result \(y\) is to declare agreement when the absolute difference is no greater than a combined absolute and relative tolerance:
Here \(\tau_{\text{abs}}\) and \(\tau_{\text{rel}}\) are prespecified comparison tolerances, expressed in compatible units or as a dimensionless relative tolerance. They are reproducibility criteria, not universal constants and not evidence of model adequacy. Choose them to suit the quantity, units, and numerical behavior being compared. For profiles, compare values on a defined common time grid or use another specified profile-comparison method; comparing different sampling times is not an unambiguous test.
Worked Example: Checking a Rerun Against a Saved Result
In a hypothetical rerun, a recorded \(C_{\max}\) is 12.40 mg/L and the rerun gives 12.42 mg/L. The prespecified comparison uses \(\tau_{\text{abs}}=0.01\) mg/L and \(\tau_{\text{rel}}=0.001\). Check whether the values meet that rule.
The absolute difference is
The permitted difference is
Since \(0.02 \leq 0.0224\), this result meets the stated numerical comparison rule. The conclusion is limited to this scalar endpoint and this tolerance. It does not establish that all profile points match, that the model is scientifically adequate, or that the tolerance is appropriate for every endpoint.
Diagnosing Missing or Inconsistent Records
A useful record supports diagnosis when a rerun does not match. Begin by checking file identities and software versions, then compare model and input revisions, code changes, solver settings, scenario choices, and output-processing steps. If the discrepancy remains, examine time grids, event handling, initialization, and any stochastic or parallel features. Keep the original run record and outputs: overwriting them with a rerun destroys the evidence needed to compare the two computations.
Record the endpoint name, units, scenario, and exact output location or extraction rule.
Compare file names, revisions, and digests for the model, input data, and scripts.
Check software build, solver configuration, dose schedule, initialization, time grid, and run-specific switches.
Compare matching outputs using the prespecified exact or numerical criterion, and document any discrepancy rather than silently replacing the result.
Common warning signs include a model file with no revision or digest, a stated “default solver” without a software version, output values without units, and a script that is mentioned but absent from the package. Another warning sign is a rerun compared at a different time point or with a different endpoint extraction rule. These gaps do not prove that a result is wrong; they prevent a confident account of how it was produced.
Worked Example: Tracing a Changed Result
Consider a second hypothetical run record. The saved result is a trough concentration of 1.80 mg/L. A rerun gives 1.87 mg/L, and the comparison tolerance is 0.02 mg/L absolute plus 0.001 relative. The permitted difference is \(0.02 + 0.001(1.80)=0.0218\) mg/L; the observed difference is \(|1.87-1.80|=0.07\) mg/L. Since \(0.07 > 0.0218\), the rerun does not meet the stated comparison rule.
The analyst checks the manifest and finds that the saved run used dataset revision 2, while the rerun used revision 3. This is a plausible source of the discrepancy, but the difference alone does not establish that it caused the full 0.07 mg/L change. To investigate, the analyst reruns with the saved dataset revision and retains the same software build, model file, code, and settings. If that controlled rerun returns 1.80 mg/L, the evidence supports the dataset change as the explanation in this case. If it does not, the remaining components must be checked. Each run and its output remains separately recorded.
Choosing a Record Format and Recognizing Its Limits
A structured, machine-readable manifest is useful for comparing runs, checking required fields, and linking records to exact files. A concise narrative is useful for explaining decisions such as why a solver was chosen or how an endpoint was extracted. A strong submission can use both: structured fields for stable identifiers and settings, plus brief prose for rationale and exceptions.
| Approach | Strength | Limitation |
|---|---|---|
| Structured manifest | Easy to compare, search, and check for missing fields | May not explain the rationale behind a setting |
| Narrative description | Can explain choices and context clearly | May omit details or describe settings ambiguously |
| Screenshots alone | Can show a particular software interface state | Often difficult to search, compare, or use to reconstruct all settings |
A saved project file can preserve settings, but it should not be treated as a complete record unless its dependencies, external data, software environment, and output-processing steps are also identified. Conversely, a detailed document cannot substitute for missing model files or code. Reproducibility also has practical limits: access restrictions, discontinued software, hardware dependence, and nondeterministic algorithms may prevent an exact rerun. State such constraints and provide the strongest available record rather than claiming exact reproduction without support.
Common Mistake: Treating Reproducibility as Validation
A common but incorrect argument is: “The rerun matched the submitted result, so the PBPK prediction is reliable.” Matching shows that the defined computation could be repeated to the stated degree. It does not show that the model equations, parameter values, assumptions, or prediction are correct for their intended use. Those are separate scientific questions. The reverse error is also possible: a small numerical mismatch does not automatically mean the model is scientifically unsound; first determine whether the run settings or computational environment differed and whether the mismatch matters for the reported result.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why is recording only the name of a model file insufficient to identify the exact model used?
- A simulation record gives the software name but not its version. Give one reason this can prevent a meaningful reproduction.
- A recorded endpoint is 4.00 mg/L and the rerun is 4.03 mg/L. With an absolute tolerance of 0.01 mg/L and relative tolerance of 0.002, does it meet the combined comparison rule?
- What should be recorded if a run uses a stochastic procedure and a random seed?
- Does reproducing a reported concentration establish that the PBPK model is suitable for the intended use? Explain.
Solutions. (1) A file can be replaced while keeping its name; record a revision and, where feasible, a digest or an archived copy. (2) Defaults, solver implementations, or file interpretation can differ by version, changing the computation. (3) The absolute difference is \(|4.03-4.00|=0.03\) mg/L. The permitted difference is \(0.01+0.002(4.00)=0.018\) mg/L. Because \(0.03>0.018\), it does not meet the rule. (4) Record the random-number generator, seed, draw count, software environment, and relevant sampling settings; note if the algorithm may remain nondeterministic. (5) No. Reproduction concerns repeating the computation. Suitability requires separate scientific evaluation for the intended use.