Why Computational Records Matter
A population PK result depends not only on the model and analysis data, but also on the software and computational choices used to obtain it. Two analysts may use the same data and model description yet produce different results if, for example, they use different software builds, dependency versions, optimizer settings, or random-number seeds. Conversely, recording a software name and version does not by itself make an analysis reproducible.
For regulatory review, the goal is to leave a clear, auditable record of the computational environment and the settings that materially affected execution. This supports review of how the submitted results were generated and helps another qualified analyst rerun the analysis. It does not guarantee identical output on every machine, nor does it establish that the model or analysis is scientifically appropriate.
The reproducibility target should be explicit. Exact reproducibility means that a rerun produces identical output at the specified level, such as identical text files or identical parameter values to the recorded precision. Numerical reproducibility means that results may differ in the last digits or in stochastic details but agree within a stated, technically justified tolerance. A rerun that merely completes without error is not, on its own, evidence that the same computation was reproduced.
What to Record
A useful computational record is specific enough to identify the actual run, rather than just the general software product. Record the items that apply to the analysis and identify items that do not apply. If a setting was left at a default, state that it was the default and, where possible, identify the software version that defines it. “Default settings” without that context is ambiguous because defaults can change between releases.
| Record component | Useful details | Why it matters |
|---|---|---|
| Primary software | Product name, version, build or release identifier, and relevant license or module information | Different builds or modules may implement different functionality |
| Dependencies | Versions of libraries, packages, plugins, and supporting tools that affect the computation | Indirect dependencies can change numerical behavior or execution |
| Analysis implementation | Script or model-file identity, source-control commit or file hash, and any locally applied modifications | A software version does not identify the code that was run |
| Estimation options | Estimation method, optimizer, initial values, bounds, tolerances, iteration or evaluation limits, and other relevant options | These choices can affect convergence and the computed result |
| Execution settings | Random seed when relevant, number of threads or workers, parallelization settings, and run-specific options | These may affect stochastic procedures, execution order, or numerical results |
| Computational environment | Operating system and version, architecture, runtime, compiler or numerical libraries when relevant, and container or virtual-environment identifier if used | Some computations depend on platform and linked components |
| Run evidence | Run identifier, date, command or invocation, completion status, warnings, and retained logs or output manifest | Connects the recorded configuration to the actual execution |
This information complements, rather than replaces, the data-lineage record: the computational record identifies how software was configured and executed, while the analysis must also identify the input data and their provenance. Avoid relying on local file paths alone; paths may not exist outside the original environment. Use stable identifiers for the relevant code, configuration, and input artifacts, while handling controlled data according to applicable access and privacy requirements.
A compact way to reason about completeness is to represent a run as a tuple:
Here, \(D\) identifies the input data, \(C\) the analysis code and model implementation, \(S\) the software and settings, \(E\) the computational environment, and \(I\) the invocation and run record. A rerun can be meaningfully compared with the submitted run only when these components are either matched or their differences are understood. This is a practical completeness check, not a claim that identical tuples guarantee identical output: nondeterministic execution and platform-specific numerical behavior can still matter.
Worked Records for Different Computational Setups
Worked Example: Preparing a Run Manifest
Suppose, hypothetically, that an analysis was run using a licensed population modeling application and a script that prepares input files and launches estimation. The analyst initially records only “Application X, version 8.” That entry identifies the product release but not the complete run.
To make the record reviewable, the analyst adds the exact build shown by the application, the operating-system release, the script’s source-control commit, and the versions of the scripting runtime and packages used by that script. The run record states the estimation method and optimizer, the recorded initial values, parameter bounds, convergence tolerances, and maximum evaluation limit. It also records whether the application’s defaults were used for options not explicitly set.
The analyst then adds the run identifier, execution date, exact command or launch procedure, number of parallel workers, completion status, and a reference to the retained log. If the application writes a build identifier or environment report, that information is retained with the run. The final manifest now links the code, settings, environment, and execution to one run, instead of relying on a product name alone.
This record does not assert that every listed detail changed the estimate. It makes the actual computational conditions inspectable and helps a reviewer identify what would need to be matched or assessed in a rerun.
Worked Example: Recording a Stochastic Run
Consider a hypothetical bootstrap procedure with 500 resamples, executed with four workers. The analyst records the software and package versions, the number of resamples, the seed entered in the run configuration (314159), and the number of workers. The log also indicates whether the software assigns a reproducible random-number stream to each worker.
A second analyst reruns the procedure with the same seed and 500 resamples but uses one worker. The seed alone does not establish that the bootstrap samples will match: random-number generation may depend on how streams are assigned to workers and on execution order. The two analysts should first check the software’s documented behavior and the retained run configuration. If worker count or stream assignment affects the generated samples, they should match those settings for an exact comparison, or compare results using a prespecified numerical or inferential criterion appropriate to the purpose.
The appropriate record therefore includes more than “seed = 314159.” It includes the software and relevant library versions, resample count, worker count, random-number generator or stream settings when available, and whether parallel execution was used. If the software does not provide a way to preserve worker-specific streams, that limitation should be stated rather than implying the seed guarantees identical resamples.
Worked Example: Assessing a Rerun in a Different Environment
In a hypothetical review, the submitted run used application build 8.2.1 on a 64-bit Linux system and returned an objective-function value of 128.4371. A rerun uses build 8.2.1 on a different operating system and returns 128.4373. The difference is \(128.4373 - 128.4371 = 0.0002\).
The arithmetic shows a small numerical difference, but it does not establish that the runs are equivalent. The analyst checks that the data, code, estimation options, initial values, and convergence criteria match. The analyst then checks the logs for warnings, convergence status, and differences in numerical libraries or execution settings. If the comparison criterion was specified in advance as agreement within 0.001 for this diagnostic value, then \(0.0002 < 0.001\), so this value meets that particular criterion. That check alone does not show that all parameters, diagnostics, or outputs agree.
The record should identify the operating-system difference and any other environment differences, and the comparison should cover the outputs that matter to the rerun’s purpose. If no justified comparison criterion was specified, the analyst should not invent one after seeing the result and call the rerun equivalent. Report the observed difference and explain what was checked.
How to Build and Check the Record
A disciplined process reduces the chance that important settings are reconstructed from memory after the analysis has finished. Capture the run configuration at execution time, retain the log, and link both to stable identifiers for the code and input artifacts. Where a package manager can produce a dependency lock file, retain it; where software is installed from a controlled image, identify that image. A lock file or image improves specificity but is not a substitute for the run settings, a successful execution record, or an explanation of relevant platform dependencies.
Record the software build, code or model-file identifier, and relevant dependencies. Distinguish the version installed from the version actually used for the run.
Record estimation options, explicit settings and relevant defaults, seeds, parallelization, and the command or procedure that launched the computation.
Link the run identifier to logs, completion status, warnings, and the outputs produced. Preserve enough information to distinguish this run from other attempts.
Compare the actual inputs, implementation, settings, and environment. State what matched, what differed, and the criterion used to judge numerical agreement.
A basic justification for recording these components follows from the structure of computation. Output is produced by applying an implementation and settings to inputs within an environment. If any of those inputs to the computation changes, the output may change; therefore, recording only the application name cannot identify the computation. This is not a proof that every environment detail affects every analysis. It is a reason to record relevant details and to make claimed equivalence depend on evidence rather than assumption.
Diagnostics, Alternatives, and Limits
Several warning signs indicate an incomplete record: a report says only “standard settings”; a seed is listed without the procedure or parallel settings; the software version is given without a build or module when those distinctions exist; or a rerun cannot be linked to the code and configuration that produced it. Logs that show failed convergence, warnings, or a different execution path must not be omitted merely because a later run completed. Preserve enough run history to make the selected run and its status clear.
Different approaches provide different levels of control. A manually maintained environment description is flexible but can omit dependencies or drift from the actual run. A package lock file captures specified package versions but may not capture operating-system libraries or licensed software components. A container or virtual machine can package more of the environment, but may still depend on host hardware, external services, license servers, or settings supplied at run time. Directly recording the settings and retaining logs remains necessary with any of these approaches.
Exact reproduction can be impractical when numerical libraries, hardware, software licensing, or stochastic parallel execution differ. A container cannot make every platform detail identical, and a matching seed does not necessarily make a stochastic computation identical. Records also cannot establish that the model is correct, that convergence was adequate, or that an analysis choice was justified. Those questions require scientific review and other evidence. Be precise about the claim: identify whether the aim is exact rerun, numerical agreement, or simply traceability of the submitted execution.
Common Mistake
Wrong reasoning: “The rerun used the same application version and random seed, so it is the same analysis.” This conclusion overlooks code changes, dependency versions, estimation settings, input differences, worker configuration, and platform effects.
Correction: Treat software version and seed as parts of a larger run record. Compare the code and inputs, relevant dependencies, estimation and execution settings, and computational environment. Then state what was matched and what output criterion was used. Matching labels is not equivalent to demonstrating matching computation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why is “software name and version” not a complete computational record?
- A rerun uses the same random seed but a different number of parallel workers. What should be checked before claiming that it reproduces the original stochastic run?
- A software package was installed at version 4.1, but the recorded run used version 4.0. Which version belongs in the run record?
- What is the difference between exact reproducibility and numerical reproducibility?
Solutions. (1) The entry does not identify the executed code, dependencies, estimation options, environment, invocation, or run evidence. (2) Check how the software assigns random-number streams and whether execution order or worker count changes the generated samples; record the actual settings and compare using an appropriate criterion. (3) Record version 4.0 as the version used for that run; version 4.1 may be noted separately as an installed or later version. (4) Exact reproducibility requires identical output at a specified level; numerical reproducibility permits differences within a stated, justified tolerance. Neither should be claimed without checking the relevant outputs and conditions.