What an End-to-End Audit Establishes
A workflow can finish without errors and still produce a report that cannot be trusted as a record of the analysis. It may have read an unrecorded input file, reused an output computed from older data, or run in an environment that differs from the one declared. An end-to-end reproducibility audit follows one analysis from its input files through its final outputs and report, checking that the evidence supports the claimed connection between them.
The goal is not to insist that every file produced by every run must be byte-for-byte identical. Some analyses include stochastic steps, hardware-dependent numerical operations, or output metadata such as timestamps. The audit instead asks what agreement the workflow claims, whether that claim is appropriate, and whether each material difference is explained. Exact equality may be required for input files or deterministic intermediate files; other outputs may need predeclared numerical tolerances or scientifically meaningful comparisons.
A checksum is a digest calculated from a file’s bytes. If two files have different cryptographic checksums, their bytes differ. Matching checksums are strong evidence that the bytes match, assuming a suitable cryptographic hash function and no deliberate manipulation. Checksums do not establish that a file is the correct biological input, that its contents were interpreted correctly, or that all required files were recorded. Those require additional provenance and validation checks.
Before beginning, state the audit target: the report or release being reviewed, the intended inputs, and the standard of agreement. A useful audit record identifies the code revision, configuration, environment, execution settings, input identities, output identities, and validation results. The audit assumes that the records being inspected are genuine and that the workflow’s dependency description is sufficiently complete to represent what each output uses. If either assumption fails, the audit should report that limitation rather than infer a clean run.
Follow the Evidence in Dependency Order
For a small count-matrix workflow, the dependency chain might be: raw matrix and feature and barcode files; quality-control and filtering steps; a normalized expression object; dimensionality reduction and clustering; summary tables and figures; then a rendered report. The audit follows this order because a discrepancy near the beginning can propagate to every downstream result. A final report alone rarely reveals which link failed.
Record the report identity, the run being reviewed, and the expected input files. Note whether the claim is byte-identical output, numerical agreement within stated tolerances, or agreement on specified scientific summaries.
Recalculate checksums, compare them with the recorded manifest, and check file roles, dimensions, and relevant reference or annotation versions. A matching checksum confirms bytes, not biological appropriateness.
Check the code revision and whether the working tree had uncommitted changes. Compare the effective parameters and software versions with those recorded for the run, including relevant system and parallel-execution details.
For each output, establish which inputs and settings produced it, whether its dependencies changed, and whether the pipeline recomputed it when necessary. Compare suspicious cached outputs with outputs from a clean regeneration.
Check expected files, dimensions, identifiers, summary values, and declared validation gates. Confirm that the report records the run and outputs actually inspected, then document unresolved differences and their consequences.
A pipeline’s cache is trustworthy only to the extent that its cache key represents every dependency capable of changing the target. If a normalized object depends on the matrix, a filtering threshold, and normalization code, a cache key that includes only the threshold and code is incomplete. The cache may appear current while holding an object computed from a different matrix. This is why “the pipeline reported that the target was up to date” is not sufficient evidence on its own.
Worked Example: Checking Inputs and Finding a Provenance Gap
Suppose a hypothetical workflow analyzes a compressed count matrix, a feature file, and a barcode file. Its manifest lists shortened checksum strings for illustration; in practice, the full digest would be recorded. The manifest also says that the matrix has 1,200 features and 800 cells.
| File | Manifest checksum | Recalculated checksum |
|---|---|---|
| counts.mtx.gz | 8c31 | 8c31 |
| features.tsv | 1a72 | 1a72 |
| barcodes.tsv | 4f90 | bd07 |
The matrix and feature checksums match. The barcode checksums do not: the recalculated value is bd07, not 4f90. The input set therefore does not match the recorded set in at least one file. That discrepancy must be investigated before claiming the report came from the manifest’s inputs. Checking the observed barcode count can narrow the issue, but it cannot make the mismatched checksum disappear.
Assume the recalculated barcode file contains 800 unique entries, just as the manifest says. This matching count does not establish identical contents: one barcode could have been replaced by another while the file still contained 800 entries. The audit should compare the actual file with the expected version or determine which version the run used.
There is also no recorded genome-build or feature-annotation version. That is a provenance gap because the feature identifiers cannot be interpreted fully from the available record. The audit’s conclusion is not that the expression analysis is necessarily incorrect. It is that input identity is unconfirmed for the barcode file, and feature interpretation cannot be reconstructed from the manifest alone. Both findings should be resolved or explicitly retained as limitations.
Regenerate and Compare Outputs
Regeneration means executing the workflow again from identified inputs and settings, with enough control to determine whether old intermediate files were reused. It is not enough to rerun the final report-rendering command if the report reads cached objects. Where possible, use a clean output location or a documented cache reset, preserve the original outputs, and record which targets were actually recomputed.
The comparison standard should be decided before inspecting a discrepancy. A checksum comparison is appropriate for files expected to be identical at the byte level. For structured outputs, compare both basic integrity (such as dimensions and identifiers) and relevant values. For stochastic analyses, compare the outputs using criteria suitable to the scientific claim; a changed cluster label is not automatically a changed result if labels were arbitrarily permuted, while a different number of retained cells may be consequential.
This relative difference can summarize a change in a numerical quantity, but it does not decide whether the change is acceptable. The denominator must be meaningful and nonzero; for a zero reference value, report the absolute difference instead. Any tolerance should be justified for the output and purpose, rather than selected after seeing how far the results moved.
Worked Example: Diagnosing a Stale Cached Output
In a hypothetical workflow, a quality-control object records 800 cells. The count matrix is then updated, and the new input contains 798 cells after the same documented filtering rule. The pipeline says the quality-control object is current. Inspection shows, however, that its cache key includes the filtering script and threshold but not the count-matrix checksum.
Because the matrix is an input to the quality-control object, omitting its identity from the key means that changing the matrix need not invalidate the cache. The “current” status is therefore not evidence that the object reflects the updated input. A clean regeneration is required to test that link.
The original output reports 800 cells; the regenerated output reports 798. The absolute difference is \(798-800=-2\) cells. Relative to the recorded output, the difference is
The regenerated output has 2 fewer cells, a 0.25% decrease relative to the cached output. The percentage is descriptive, not a justification for accepting the cached result. The cache key omits a direct dependency, and the cell counts disagree; the cached object is stale for the updated matrix. The appropriate resolution is to correct the dependency tracking, regenerate the downstream targets, and rerun output validation. The report should be regenerated too if it used that object.
Separate Environment Drift from Other Differences
Environment drift means that relevant execution conditions differ from those recorded or expected. Examples include a changed package version, a different numerical library, or a changed execution setting. Finding drift is evidence of a possible explanation, not proof that it caused a particular output difference. Likewise, finding identical package versions does not guarantee identical results if the inputs, code, configuration, or parallel execution differ.
A practical comparison starts by checking the earliest output that differs. If the inputs and initial integrity checks agree but a normalized object differs, inspect the code, parameters, and environment used to make that object. If all intermediate outputs match but a figure or report differs, investigate report inputs and rendering settings. This narrows the search without treating correlation between an environment change and an output change as a demonstrated cause.
Worked Example: Quantifying a Difference Across Environments
Consider two hypothetical runs with matching input checksums, the same recorded code revision, configuration, and random seed. Both retain 798 cells. The recorded environment uses one version of a numerical package; the regenerated environment uses a newer version. A quality-control summary gives a median library size of 486 counts in the recorded run and 480 counts in the regenerated run.
Relative to the recorded summary, the difference is
The regenerated median is 6 counts lower, or about 1.235% lower relative to the recorded median. This quantifies the discrepancy; it does not show that the package update caused it. The audit should also compare the intermediate objects used to calculate the median and verify that both summaries used the same cells and definition of library size.
Suppose the cluster assignments also differ, and their adjusted Rand index is 0.91. This indicates substantial but imperfect agreement under that comparison measure; it does not mean that 91% of cells have identical labels or establish that either clustering is correct. The environment version difference is a plausible lead. To test its role, one could rerun in the recorded environment while holding the verified inputs, code, configuration, and execution settings fixed. Until the discrepancy is explained or judged against a prespecified scientific criterion, the audit should report a reproducibility discrepancy rather than certify exact agreement.
Interpret the Audit and Report Its Limits
An audit should distinguish what was verified from what remains uncertain. For example, “the matrix and feature-file checksums matched; the barcode checksum did not; the environment versions differed; and the regenerated report used the new 798-cell object” is more informative than “the workflow is reproducible” or “the workflow failed.” A useful audit record includes the observations, how they were checked, the comparison standard, the disposition of each discrepancy, and any unresolved limitation.
Common failure modes include an incomplete manifest, a cache key that omits a dependency, a report that embeds old outputs, and an environment record that lists only top-level packages while omitting relevant system libraries. Output checks can also be too shallow: a file’s existence or a matching cell count does not prove that its values or identifiers agree. Diagnostics should therefore combine provenance checks with content checks and, where appropriate, regenerated comparisons.
There are limits to what a successful audit can establish. Matching checksums cannot show that the chosen reference is appropriate; matching outputs cannot establish that the statistical analysis is valid; and a documented difference does not automatically invalidate a scientific conclusion. Numerical tolerance, stochastic variability, and biological sensitivity require context-specific judgment. If the workflow cannot supply evidence for an important link, state that the link is unverified rather than filling the gap with an assumption.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A file’s checksum matches its manifest, but the manifest does not say which genome build was used. What is verified, and what remains uncertain?
- A cached target has the same script and configuration as before, but its input matrix checksum changed. What should the auditor check next, and why?
- A regenerated output has 995 cells compared with 1,000 in the recorded output. Calculate the relative difference using the recorded output as the denominator.
- Two runs have identical checksums for all inputs but different package versions and slightly different numerical summaries. What can and cannot be concluded?
- Why is a matching cell count insufficient to establish that two count matrices are identical?
Solutions. (1) The file’s bytes match the file represented by the recorded checksum, subject to the hash assumptions. The checksum does not identify or validate the genome build; that provenance remains uncertain. (2) Check whether the changed matrix is a dependency in the target’s cache key and whether the target was recomputed. If the dependency is omitted, the cache can be marked current despite representing the old matrix. (3) The relative difference is \((995-1000)/1000=-5/1000=-0.005=-0.5\%\), a decrease of 0.5%. (4) The input bytes match and environment drift is present alongside the numerical difference. The version change is a possible explanation, not demonstrated causation; isolate it with a controlled rerun and report the discrepancy until it is explained or assessed against an appropriate criterion. (5) Different matrices can have the same number of cells but different cell identifiers, counts, or ordering. Compare checksums or inspect the relevant contents and identifiers.