Reproducibility of Analysis and Computational Artifacts

Why Reproducibility Is an Evidence Question

A reported model result is more useful when another qualified analyst can trace how it was generated and, where feasible, execute the analysis again. In model-informed drug development, that may mean regenerating a parameter estimate, a simulation summary, or a decision-relevant prediction from the specified data and computational workflow. A description that merely says “the model was run in standard software” does not establish that the reported result can be reproduced.

Reproducibility is an audit of the relationship between a result and the artifacts and instructions needed to regenerate it. It is not the same as checking whether the model is scientifically appropriate, whether its assumptions are justified, or whether its predictions are validated. Those questions remain important, but they require different evidence. The earlier material on data provenance and analysis-set reconciliation concerns which data entered the analysis; here the question is whether the relevant data and computational process are identified and usable for a rerun.

The phrase “specified result” matters. Reproducing one clearance estimate does not establish that every figure, simulation, or conclusion from the analysis can also be regenerated. Likewise, a package may allow a rerun but not support independent verification if the data are inaccessible or the code depends on an undocumented proprietary component. The reviewer should define the scope of the reproduction claim before judging whether its supporting materials are sufficient.

The Computational Chain and Its Assumptions

A useful conceptual model is to regard an output as the result of a computational pipeline:

Here \(D\) is the input dataset and its version, \(C\) is the code, \(E\) is the software and execution environment, \(S\) is the set of run settings, and \(R\) represents random-number information when the analysis is stochastic. The expression is a review framework, not a claim that every workflow uses all five components: a deterministic calculation may not need a random seed, for example. It emphasizes that an output can change if any relevant component changes.

A defensible assessment assumes that the reported output is identifiable, the relevant workflow can be distinguished from unrelated analyses, and there is a meaningful way to compare original and regenerated results. If these conditions do not hold, the reviewer may still assess documentation quality, but should not claim that an actual reproduction test has succeeded or failed.

A review should look for the following, at a level proportionate to the result and the workflow:

• Input data: a stable identifier or version for each analysis dataset, including relevant processed inputs; a data dictionary or schema where needed; and a checksum or other integrity record when available. If data cannot be distributed, the access route and conditions should be stated.
• Code: the relevant source code, scripts, or a uniquely identified version in a repository, with instructions identifying the entry point and required files. A link to a changing repository without a commit or release identifier may not identify the code that produced the reported output.
• Computational environment: software names and versions, required libraries or dependencies, and, when they could affect results, operating-system, compiler, hardware, or numerical-library details. A lockfile or container image can help specify dependencies but does not by itself explain how to run the analysis.
• Run settings: the command or procedure used, configuration files, model or simulation options, input paths, and relevant execution choices such as parallelization. Settings should be recorded rather than inferred from defaults that may change across versions.
• Randomness and execution records: the random-number generator and seed where relevant, parallel random-stream configuration if used, and logs or metadata that help establish what actually ran and whether it completed as expected.

These items are not a universal checklist of mandatory files. The governing question is whether a knowledgeable reviewer can unambiguously identify the inputs and computational conditions relevant to the output under review. Requirements may also be constrained by confidentiality, privacy, licensing, or regulated-system controls. In those cases, controlled access, executable review in a secure environment, or a documented alternative may be more appropriate than unrestricted distribution.

Why a Seed Is Not the Whole Randomness Record

For a stochastic workflow, a seed initializes a random-number generator. It does not necessarily determine the realized output by itself. A more complete representation of the random component is:

Here \(q\) is the seed, \(A\) is the generator algorithm, \(V\) is its implementation or version, \(K\) represents the sequence and organization of random-number requests (including parallel-stream behavior), and \(U\) is the resulting sequence of random values. This gives a short justification for a common audit finding: recording \(q\) alone does not guarantee the same \(U\). If the generator changes, or the code requests values in a different order, the same initial seed may lead to different random values being used for particular calculations. If every relevant part of the process is deterministic and unchanged, a seed may be sufficient to initialize the same sequence; that condition needs to be supported, not assumed.

Testing a Rerun and Interpreting Differences

When the relevant artifacts are available, a reproduction test should follow the documented workflow and record the execution conditions. Before comparing outputs, specify what is being compared and what agreement means. The criterion should fit the output: a parameter estimate, a simulated event probability, and a generated figure do not necessarily call for the same comparison rule. Exact equality can be appropriate for some deterministic outputs, but may be unnecessarily strict when numerical libraries, hardware, or stochastic variation can cause small differences.

A tolerance should not be chosen after seeing the rerun difference merely to make the result appear acceptable. It should be justified in advance by the precision of the reported result, the numerical behavior of the workflow, and the scientific relevance of the output. If no defensible comparison rule was specified, the reviewer can report the observed difference and its context, but should avoid classifying it as acceptable solely because it seems small.

Failure Modes, Alternatives, and Limits

An audit should distinguish between a missing record, an inaccessible artifact, a failed execution, and a numerical mismatch. They imply different remedies. For example, a run blocked by a missing dependency may indicate incomplete environment documentation; it does not show that the model calculation itself is unstable. A successful run that produces a different result calls for investigation of inputs, versions, settings, numerical behavior, and random-number handling before the discrepancy is interpreted.

Common warning signs include a mutable data or code link with no version identifier; reliance on unrecorded software defaults; an installation that requires undocumented manual edits; missing run settings; unavailable data with no stated access process; or an output that cannot be tied to a particular run. These signs weaken the evidence that the reported result can be regenerated. They do not, by themselves, demonstrate that the result is incorrect.

Reproduction is also distinct from an independent reanalysis, which may implement the analysis anew, and from independent replication, which may use new data or a separate investigation to examine whether a finding recurs. Those approaches can provide valuable evidence, but they do not answer exactly the same question as rerunning the original computational workflow. Conversely, rerunning the original code can reproduce a coding error or an inappropriate analysis; reproducibility alone is not scientific validation.

The practical limits should be recorded explicitly. Data privacy may prevent open release; licensed software may restrict distribution; and hardware or numerical libraries may prevent bit-for-bit equality even when results are substantively consistent. A secure review environment, an archived executable environment, or a controlled-access data process may address some barriers, but each has its own access and maintenance requirements. A container or archived environment can improve specification, yet it cannot guarantee future availability, correct execution, or scientifically sound results.

A useful report therefore states the reproduction target, the artifacts and versions examined, what was actually executed, any deviations or barriers, and how output agreement was judged. The conclusion should be no broader than the evidence: “the listed estimate was regenerated within the prespecified tolerance” is more informative and defensible than “the analysis is reproducible.”