Auditing Reproducibility From a Clean Checkout

Why Audit From a Clean Checkout?

A project may run successfully on its author’s computer because that computer contains more than the repository: cached data, manually installed packages, environment variables, edited configuration files, or outputs left by an earlier run. A successful run in that workspace therefore does not by itself show that the documented project can be reproduced by another analyst.

A clean-checkout audit tests a more specific claim: starting from a stated version of the repository and the documented, permitted inputs, can a colleague set up the required environment, execute the documented workflow, and obtain the expected outputs without relying on undeclared local state? This is a practical test of the handoff, not merely an inspection of files or a review of code.

The audit is strongest when the person performing it did not prepare the project and begins in a new directory or isolated environment. The revision must be identifiable, for example by a commit identifier, and the expected outputs and their comparison rules must be specified before inspecting the new run. Otherwise, an analyst can unconsciously adjust the criteria to fit whatever happened.

Specify the Audit Before Running It

A reproducibility claim is meaningful only when its scope is clear. Record the repository location and exact revision; the operating system or execution environment; the setup instructions to be followed; the required inputs and where they may be obtained; the documented entry point and execution order; and the outputs that will be checked. For a pharmacometric project, expected outputs might include a model-run summary, parameter estimates, diagnostic tables, or figures. A clean audit need not redistribute confidential patient-level data: it must state which authorized inputs are required and confirm that the auditor can access them under the project’s procedures.

Also decide what “same output” means. Some outputs should match exactly, such as a configuration file or a table of fixed labels. Numerical results may need a defined tolerance because software versions, hardware, parallel computation, or numerical libraries can cause small differences. A tolerance is not a license to overlook material changes: it should reflect the intended reproducibility claim and the scale and units of the quantity being compared.

The audit assumes that the chosen revision and reference outputs are authentic, the required inputs are appropriate and available, and the comparison rule is suitable for the output. It also assumes that the auditor follows the documented procedure rather than adding undocumented fixes. If any of these conditions is not met, report the limitation rather than treating a partial or altered run as a successful reproduction.

A Controlled Audit Sequence

Keep the stages distinct so that a failure can be localized. Record the initial state before installation or execution; in particular, note the revision and whether the checkout is clean. Follow the written setup instructions as a colleague would. Do not silently install extra packages, copy in local configuration, or edit scripts to get a run through. If a necessary step is missing, record it as an audit finding.

The logic behind a fresh starting point is straightforward. Let the run result depend on both repository contents \(R\) and local state \(L\): \(Y=f(R,L)\). A run on the author’s computer observes \(f(R,L_{\mathrm{author}})\), so it cannot establish that repository contents alone, together with the declared inputs and environment, are sufficient. A clean checkout removes many undeclared components of \(L_{\mathrm{author}}\). If the colleague can then follow the documented steps and meet the prespecified output criteria, that is evidence for the narrower and useful claim that the stated handoff is executable under the audited conditions. It does not prove that every machine or future software environment will behave identically.

Check Outputs Against a Prespecified Rule

A workflow that exits without an error has not necessarily recreated the intended analysis. It may have produced an empty table, used the wrong configuration, or written outputs to an unexpected location. Check that the expected files exist, that their contents are meaningful, and that the relevant values meet the specified criteria. Comparing only filenames or only a successful exit status is insufficient.

Tolerance rules must be chosen with care. Rounding both values before comparison can conceal meaningful discrepancies; compare at the prespecified precision or using the explicit tolerance formula. A single overall pass/fail label can also obscure an important failure in one output, so report checks individually. If an output is stochastic, specify which summaries or properties are expected to agree and how variability will be handled. A fixed random seed may support repeatability in a given environment, but it does not guarantee identical results across all software or hardware.

Diagnostics, Alternatives, and Limits

Useful audit evidence includes the revision identifier, a record of the clean starting state, setup commands or steps followed, software-environment information, input identifiers or checksums where appropriate, execution logs, exit status, and an inventory of expected outputs and comparison results. Protect confidential material: provenance records should identify authorized inputs without exposing private data. If the workflow modifies tracked files, uses network resources, or relies on licensed software, record those facts and the conditions under which the run was performed.

Several patterns weaken the evidence. A run that starts from an old working directory can reuse generated files. An undocumented manual edit changes the tested procedure. A missing output may be overlooked when only the program’s exit status is checked. A tolerance chosen after seeing discrepancies is not an independent test. Each situation should be recorded as a limitation or deviation, not hidden behind a general statement that “the project ran.”

A code review can assess whether scripts and dependencies appear understandable, but it does not show that the documented steps work in a fresh environment. Re-running the workflow in the developer’s current workspace is faster, but it does not adequately test dependence on that workspace. A clean run in an isolated environment, including a container or managed continuous-integration system, can strengthen control over software and starting state; it still needs declared inputs, meaningful output checks, and a recorded revision. These approaches complement rather than replace an independent clean-checkout audit.

A pass is bounded evidence, not a universal guarantee. It applies to the revision, inputs, environment, and criteria actually tested. It does not establish that the model is scientifically valid, that every parameter estimate is correct, or that future revisions will reproduce the same outputs. A reproducible computational result can still be based on an inappropriate analysis choice. Conversely, small numerical differences can arise without changing the scientific interpretation, but the audit should not decide that after the fact: the acceptance rule must be explicit.