Why Reproducible Materials Matter
A validation report can state which cohorts were analyzed, how the analysis was conducted, and what results were obtained, yet still leave another analyst unable to check how those results arose. Reproducible materials close that gap. They connect the reported findings to identifiable versions of the code, data inputs or approved data references, cohort-specific definitions, and the recorded execution history.
The goal is not simply to publish a script. A script can be ambiguous about its inputs, rely on undocumented software, or silently use a different feature mapping in each cohort. For a multi-omics validation, small differences in sample identifiers, assay units, preprocessing, or feature availability can change which observations contribute to a result. The materials should make those choices inspectable without exposing data that cannot ethically or legally be shared.
There are two distinct questions. Can an authorized analyst reproduce the computation from its recorded inputs? And can a different analyst assess whether the design and decisions support the report's interpretation? Materials support both questions, but neither can be answered by code alone. Cohort provenance, analysis decisions, deviations, and uncertainty are reported separately; the earlier material on reporting validation design and cohort provenance, reporting analysis decisions and protocol deviations, and reporting replication outcomes and uncertainty addresses those report-level details. This tutorial focuses on the reproducibility of the materials underlying them.
The Reproducibility Chain
Treat each reported result as the endpoint of a traceable chain. A result should be linked to the cohort input, the transformations applied, the analysis configuration, the code and software environment, and the specific output from which the report transcribed or generated it. A useful abstract representation is
where \(R\) is a reported result, \(D\) denotes the data inputs or controlled-access data references, \(C\) is the code, \(\theta\) is the analysis configuration, \(E\) is the software environment, and \(\omega\) records random-state information when stochastic steps are used. For deterministic analysis, \(\omega\) is not needed. Reproducing \(R\) requires enough information to identify each relevant input and to execute the same defined computation. If any link is missing or mutable, the result may not be reproducible.
This relationship is a bookkeeping principle, not a claim that a particular statistic is correct. It explains why several materials are needed together. Code without exact input versions does not identify \(D\); a data file without a dictionary may not define the meaning or units of its columns; and a random seed without the software and algorithm context may not reproduce a stochastic result. Hashes and version identifiers help establish identity, but they do not explain scientific meaning or validate analytic choices.
Give the package a release identifier and record the code revision, configuration version, and date. Do not rely on a mutable label such as “latest.”
Record an access-controlled dataset identifier or file version, the cohort and assay, the eligible sample set, and relevant preprocessing status. Never imply that restricted data are publicly available.
Provide a data dictionary and cohort-specific metadata for identifiers, outcomes, covariates, omics features, units, coding, missingness, and transformations.
Capture the command or workflow entry point, environment, random-state settings where applicable, run status, logs, and output identifiers. Map each report result to the output that contains it.
A checksum can help detect a changed file: for instance, a cryptographic hash of a released input can be recalculated and compared with the manifest. A matching hash is evidence that the file matches the referenced byte sequence, not evidence that it is the correct cohort, correctly labeled, or scientifically suitable. In controlled-access settings, a data custodian may retain the actual file and verify its hash without disclosing the file.
What to Include in a Reproducible Package
| Material | Minimum useful content | What it helps establish |
|---|---|---|
| Versioned code | Released source revision, entry point, and dependencies | Which instructions were executed |
| Configuration | Feature lists, cohort filters, thresholds, model options, and explicit defaults | Which analysis choices were applied |
| Data dictionary | Variable name, meaning, type, units, coding, allowable values, and missing-value representation | How to interpret each input field |
| Cohort-specific metadata | Dataset version or access reference, sample counts, assay and platform details, feature coverage, and relevant preprocessing | Which cohort-specific inputs and definitions were used |
| Environment record | Language and package versions, operating-system or container information, and relevant numerical settings | Which computational context supported the run |
| Audit trail | Run identifier, timestamp, command, status, logs, deviations, and output locations or checksums | How a result was produced and where to inspect it |
A data dictionary should describe meaning rather than repeat column names. For an omics feature, document the identifier namespace and version, assay, measurement scale, units if applicable, and how duplicate or unmapped identifiers were handled. For an outcome, specify its definition, coding, time window, and missing-value convention. If a column is transformed, explain whether the released input is raw or already transformed and identify the step that produced the analyzed value.
Cohort-specific metadata matter because a shared script does not make cohorts interchangeable. Document the cohort-specific sample inclusion rule, available modalities, assay platform or protocol, and the mapping between local and harmonized feature identifiers. Record the number of samples at relevant processing stages, including exclusions and missing measurements. Do not replace a precise count with a generic statement such as “samples were filtered.” These records should agree with the validation report and with the actual inputs used by the code.
For versioning, preserve an immutable release or source revision and a machine-readable manifest that points to the relevant files and data references. A dependency lockfile or container image can make the environment more precise, but neither is a substitute for recording the operating context and testing that the workflow runs. A container may still depend on an external data source, and software can behave differently across hardware or numerical libraries. State a numerical tolerance when exact bitwise equality is not expected, and define in advance which outputs are compared.
Worked Examples
Worked Example: Checking a Cohort Manifest Against the Analysis
Consider a hypothetical validation package for Cohort Pine. Its manifest records 120 eligible participants, 116 with the outcome observed, and 18,400 RNA features in the harmonized feature table. The analysis log reports 116 participants entering the model and 18,400 features before the prespecified feature filter. The data dictionary states that the outcome is coded 0 or 1, and the cohort metadata identify the RNA values as log-transformed expression. Check whether the records agree and identify a useful audit check.
The participant counts are consistent if four of the 120 eligible participants have a missing outcome: \(120 - 116 = 4\). The model's reported input count of 116 therefore agrees with the manifest's observed-outcome count. The feature count also agrees at the pre-filter stage: both records say 18,400. The outcome coding and expression scale clarify how to interpret the input columns; neither count alone provides that information.
A concrete audit check is to compare the released sample-ID list used by the workflow with the list named in the manifest, then verify the count after the outcome-missingness filter. If the lists contain 120 unique eligible IDs and exactly four have no observed outcome, the expected model input count is 116. If the run instead reports 115, the discrepancy is not explained by the documented rule. The analyst should investigate and record the cause rather than silently changing the manifest or report. This check supports traceability; it does not determine whether the eligibility rule is scientifically appropriate.
Worked Example: Recording a Reproducible Randomized Step
A hypothetical validation workflow uses 200 bootstrap resamples to estimate an interval for a molecular signature's performance. The released configuration sets the random seed to 731, specifies 200 resamples, and records the sampling unit as participant. Suppose the run log shows that 200 resamples completed, with 198 successful fits and two failed fits. What should the package record, and what can a rerun reasonably be expected to match?
The audit record should include the configuration values, the software and relevant package versions, the participant-level resampling rule, and the count of successful and failed fits. It should also identify how failed fits were handled in the reported summary. The observed success fraction is
Thus, 99.0% of the attempted fits succeeded and 2 of 200 failed. The package must not describe all 200 as successful. With the same data, code, environment, and random-state setup, the analyst can test whether the workflow reproduces the recorded outputs. Exact agreement may still depend on the software's random-number implementation and numerical libraries. If those details do not guarantee bitwise-identical output, state a prespecified tolerance and compare the summaries under that rule. A seed alone is not sufficient documentation, and reproducing these numbers does not establish that the resampling procedure is suitable.
Worked Example: Tracing a Reported Result to Its Source
In a hypothetical report, the Cohort Cedar result is described as an estimated effect of 0.42 with standard error 0.15. A machine-readable results file has a row for Cedar with estimate 0.42, standard error 0.15, and 114 analyzed participants. The audit log points to results file `cedar_run_07`, but the manifest identifies `cedar_run_06` as the release output. Determine what is reproducible from these records and what must be resolved before release.
The report's two numerical values match the Cedar row in `cedar_run_07`, so there is a possible source for the reported result. But the audit log and manifest identify different run outputs. The package therefore does not yet establish which run produced the release result. The analyst should inspect both runs, compare their input references, configuration, and code revision, and determine which output was actually used for the report. If run 07 is authoritative, update the manifest and record why; if run 06 is authoritative, verify the report's values against it. Preserve the discrepancy and its resolution in the audit trail rather than overwriting history.
After resolving the mismatch, link the report row to the selected output using a stable run identifier and, where feasible, a checksum. The participant count of 114 should also agree with the cohort-specific inclusion and missingness records. These checks establish a coherent provenance path; they do not by themselves validate the effect estimate or its uncertainty.
Diagnostics, Alternatives, and Limitations
Reproducibility problems often appear as inconsistencies between the report, manifest, logs, and data dictionary. Useful checks include comparing sample counts at each analysis stage; confirming that feature identifiers and units match the documented mapping; checking that the configured cohort and modality match the reported result; and verifying that every output referenced in the report exists in the released package. A workflow should also be rerun from a clean environment using only the stated dependencies and inputs, when access permits. A successful run is valuable evidence, but does not prove that all reported results came from that run.
Common warning signs include a mutable code link, undocumented manual edits, an unexplained difference in sample count, a feature list recreated outside the workflow, software versions recorded only as “current,” or a final table with no link to run-specific output. If exact reproduction fails, first check inputs and access versions, then configuration, software environment, random-state handling, and undocumented steps. Record whether the difference is an expected numerical tolerance or a substantive discrepancy. Do not conceal failed runs or silently select the output that best matches the report.
Several approaches can support reproducibility. A dependency lockfile is lightweight and convenient for package versions; a container can capture more of the execution environment; and a workflow manager can make dependencies among processing steps explicit. A manually documented procedure may be necessary where data systems are restricted, but is more vulnerable to omitted steps. These approaches are complementary rather than interchangeable: a container does not define cohort eligibility, a lockfile does not preserve input data, and workflow software does not automatically create a complete audit trail. Choose and describe the approach that fits the data and computing constraints.
Data governance limits what can be released. Patient-level multi-omics data, linkage keys, and some cohort metadata may be restricted. In that case, provide a controlled-access route or a stable custodian reference, describe the access conditions, and make non-sensitive code, dictionaries, configuration, and synthetic test data available where appropriate. Synthetic data can help test that code executes, but cannot reproduce cohort-specific results. The report should distinguish what can be rerun publicly from what requires authorized access.
A reproducible package can establish how a reported result was generated, but cannot guarantee that the cohort was representative, that an assay measurement was valid, or that a model was well specified. It is not a substitute for the design, provenance, and uncertainty information needed to judge validation evidence. This distinction is essential: computational traceability makes scientific scrutiny possible; it does not replace that scrutiny.
Common Pitfall
A Practical Release Check
- Can an authorized analyst identify the exact cohort input or access-controlled data version used?
- Do the data dictionary and cohort metadata define the relevant variables, units, feature identifiers, filters, and missingness?
- Are code, configuration, dependencies, and stochastic settings versioned and recorded?
- Does the audit trail preserve run status, failures, deviations, and output identifiers?
- Can each reported result be traced to a specific run output, with documented tolerances for numerical comparisons?
- Are privacy restrictions and the limits of public reproduction stated plainly?
Check Your Understanding
Answer each question, then compare your response with the concise solutions below.
- A report and its manifest give different sample counts for one cohort. What should the analyst do before release?
- Why is recording a random seed alone insufficient to reproduce a bootstrap analysis?
- What does a matching checksum establish, and what does it not establish?
- A restricted cohort cannot be shared. Name two useful materials that can still be released and one way to identify the controlled data.
- Does successfully rerunning a workflow show that the validation conclusion is scientifically correct? Explain.
Solutions. (1) Reconcile the cohort input, filters, missingness, and run logs; identify the authoritative count and preserve the resolution in the audit trail instead of silently editing one record. (2) Reproduction also depends on the exact data, code, configuration, software and random-number implementation, and sometimes numerical libraries. (3) A matching checksum supports byte-level identity with the referenced file; it does not verify cohort meaning, labels, suitability, or analysis validity. (4) For example, release the code and data dictionary, and identify the restricted dataset through a stable custodian reference or access-controlled version identifier. (5) No. A rerun supports computational reproducibility under the recorded conditions; scientific validity also depends on design, measurement, assumptions, and interpretation.