Why Reproducibility Materials Matter
A comparative study can report its population, matching or weighting choices, diagnostics, effect estimates, and uncertainty clearly, yet still leave readers unable to determine how the results were produced. Reproducibility materials address that gap. They provide a traceable account of the computational steps and the conditions under which those steps can be inspected or rerun. This supports scrutiny of the analysis without implying that observational data identify a causal effect automatically.
Transparent reporting and computational reproducibility are related but not identical. A report is transparent when it gives enough information to understand what the investigators did, including choices and limitations. A computational result is reproducible when another analyst can use the same data, code, and sufficiently compatible computing environment to obtain the same reported outputs, allowing for stated numerical tolerances. A study may be transparent even when legal or ethical restrictions prevent independent access to its data. In that case, the restrictions and any available route to access should be explicit.
The goal is not to publish every internal file or to promise exact numerical identity across all computers. The goal is to make the analytical path understandable, checkable, and as independently executable as the data and computing conditions allow. Reproducibility materials complement earlier reporting of analysis samples, balance diagnostics, uncertainty, and sensitivity analyses; they do not replace those results.
What to Record
A useful reproducibility record documents each link between the final tables or figures and the steps that generated them. At minimum, state the software and version used for data preparation and analysis. Name relevant packages, libraries, or other dependencies and their versions. If analysis relied on a particular operating system, specialized software, or an important external resource, identify it when it could affect execution or results. “Analyses were conducted in R” is not enough to identify the computational environment; include the R version and the versions of consequential packages.
Make the analysis code available where permitted, and state where it is stored and how it can be accessed. A repository should have a stable identifier or versioned release where possible. Explain how to run the analysis, which files or parameters are required, and which outputs the code generates. If a random-number generator affects a procedure, report the software’s seed-setting approach and seed value when feasible. A seed improves repeatability within a compatible environment, but does not guarantee identical results across different software versions or implementations.
Describe the data precisely enough for a reader to understand what the code expects: the source, relevant dates or release, format, and any transformations needed before analysis. Do not assume that a publicly available source is immutable. Give a release identifier, access date, or other locator that distinguishes the data version used. If data cannot be shared, explain why, identify the data custodian or access process when possible, and say whether qualified researchers can apply for access. State which parts of the analysis cannot be independently reproduced without the restricted data.
Finally, distinguish the planned analysis from the analysis actually conducted. Record changes to eligibility rules, exposure or outcome definitions, propensity-score specification, matching or weighting choices, exclusions, outcome models, or uncertainty calculations. For each material deviation, give the reason, when the decision was made if known, and which results it affects. A concise deviation log is more informative than a general statement that “minor changes were made.”
A Traceable Reporting Workflow
A practical way to design the materials is to work backward from each central result. For every estimate, table, or figure, identify the input data, the code that created it, the required software environment, and the instructions needed to rerun that code. This chain exposes missing links before publication. For example, a figure should not depend on undocumented manual edits to a spreadsheet after the analysis script finishes.
List the principal estimates, tables, figures, and diagnostics that readers should be able to verify. Link each to the analysis step that produced it.
Identify data sources and versions, describe derived files, and state whether the data are public, restricted, or available only through an application process.
Name software and versions, consequential dependencies, any required configuration, and how to run the code. Record random seeds when relevant.
Check that code, results, and the manuscript reflect the same analysis decisions. List and explain deviations rather than silently substituting a different workflow.
This workflow rests on a simple condition: the stated materials must correspond to the analysis that generated the reported results. If code is an abandoned draft, a data file is a different release, or a package version is omitted despite affecting the output, a reader cannot reliably follow the chain. The record should therefore be checked against the final analysis files, not assembled from memory after the fact.
Worked Example: Reporting a Versioned Analysis Environment
Suppose a hypothetical study uses matching to compare 240 treated patients with 360 comparison patients. Its report includes a matched-sample outcome estimate and a balance figure. The analysis was run in R 4.3.2 using MatchIt 4.5.5 and ggplot2 3.5.0. The analysis script sets a random seed of 1842 before a bootstrap procedure used for the reported interval.
A vague statement such as “analyses were conducted in R” omits the package versions and the seed. A more useful record states that the matching and figure code is available in a named, versioned repository release; specifies R 4.3.2, MatchIt 4.5.5, and ggplot2 3.5.0; and gives the command or instructions needed to run the scripts. It also identifies which script produces the estimate and which produces the balance figure.
If the code contains set.seed(1842) before the bootstrap, the report can say that the seed was 1842 and identify the software version used for resampling. This helps make that computation repeatable. It does not establish that every output will be identical under every future R or package version: random-number generation and numerical routines can change. The versioned environment and seed together are more informative than either alone.
Before making a reproducibility claim, the investigators should run the released code from the documented starting point and compare the generated estimate and figure with the reported versions. If the outcome estimate is reproduced but the figure requires a manual edit, that edit must either be automated and documented or disclosed as a limitation. The example reports no numerical estimate because none is supplied; inventing one would not improve reproducibility.
Data Restrictions, Deviations, and Dependencies
Data access statements should distinguish several situations that are often collapsed into “data available on request.” Data may be openly downloadable, available under a license, controlled by a custodian, or unavailable for sharing because of privacy, consent, contract, or law. State which situation applies. If access requires approval, report the responsible organization, the general application route, and any known eligibility constraints. Do not promise access on behalf of a custodian. If no independent access is permitted, say so plainly and identify which results cannot be independently recalculated from shared materials.
When individual-level data cannot be released, code and documentation may still be shared, provided they do not disclose protected information. A synthetic or example data set can illustrate file structure and demonstrate that scripts run, but it does not reproduce estimates from the restricted study data. Make that distinction explicit. Likewise, aggregate results alone usually cannot substitute for the data needed to regenerate an analysis.
Dependencies include more than named statistical packages. The workflow may require a particular file format, external data dictionary, operating system, database connection, or licensed software. Document the dependencies that a reader needs to execute the analysis, and avoid exposing credentials, private endpoints, or other sensitive configuration. Where practical, a locked environment file or container specification can record versions in a machine-readable form. Such tools support execution but do not remove the need for a plain-language description of the environment and access limits.
Worked Example: Restricted Registry Data With Public Code
Consider a hypothetical comparative study using a health registry whose data cannot be publicly shared under its data-use agreement. The investigators can release their analysis scripts, a data dictionary, and a small synthetic file with the same variable names and data types, but cannot release patient records.
The availability statement should say that patient-level registry data are restricted by the agreement, identify the registry custodian, and describe the application process if outside researchers may seek access. It should separately state that the analysis code and data dictionary are available in a versioned repository. The synthetic file should be labeled as illustrative and not as the study data.
A reader can then inspect the code and test whether it runs on the synthetic file, but cannot verify that the registry data were cleaned or transformed as described, or independently recalculate the study’s effect estimate. The accurate conclusion is that the workflow is partially inspectable, while independent reproduction of the numerical results depends on obtaining authorized access to the registry data. Calling the result “fully reproducible” would overstate what the shared materials allow.
A deviation record should be concrete enough to distinguish a changed plan from an undocumented discrepancy. For instance, “we modified the analysis” is not informative. State what was planned, what was done, why, and which outputs were affected. If a planned matching specification failed to achieve adequate overlap and a different specification was used, identify both specifications and connect the change to the reported diagnostics and estimates. The explanation does not itself establish that the alternative is unbiased; it makes the decision visible for evaluation.
Worked Example: Documenting a Change After a Failed Run
In a hypothetical study, the analysis plan specified 1:1 nearest-neighbor matching without replacement. When the prespecified algorithm was applied, 28 of 150 treated participants could not be matched under the stated caliper. The team then used a wider caliper, which retained all 150 treated participants but changed the matched sample and its balance diagnostics.
A transparent report records both the planned and implemented calipers, the algorithm, and the number retained under each specification: 122 treated participants under the original caliper and 150 under the wider one. It explains that the wider caliper was chosen to retain more treated participants, and reports the relevant balance and outcome results for the implemented analysis. If the original result is also shown as a robustness analysis, label it as such rather than presenting it as the sole planned analysis.
The difference in retained treated participants is \(150-122=28\). This arithmetic makes the sample change explicit, but it does not determine which caliper is preferable. Readers need the stated rationale and diagnostics to assess the trade-off. The report should not describe the wider-caliper result as if it had followed the original plan.
Diagnostics and Common Failure Modes
Reproducibility problems can often be detected before release. A strong check is to run the workflow in a clean environment using only the files and instructions intended for readers. Confirm that paths do not point to a researcher’s local computer, that required inputs are identified, and that generated outputs correspond to the reported tables and figures. Compare the outputs, noting any expected variation from numerical tolerances or stochastic procedures.
| Warning sign | Why it matters | Improvement |
|---|---|---|
| Software named without versions | Packages and defaults can change across releases. | Report software and consequential dependency versions. |
| Code posted without run instructions | Readers may not know the required inputs, order, or configuration. | Provide a brief execution guide and expected outputs. |
| “Data available on request” without details | Access may be impractical, prohibited, or dependent on approval. | Identify restrictions, custodian, and access route, or state that access is unavailable. |
| Results differ from released code | The repository may not represent the final analysis. | Rerun and reconcile the code, outputs, and report before release. |
| Unreported manual changes | Some results cannot be regenerated from the code. | Automate the step or describe it as a limitation. |
A common mistake is to treat posting code as equivalent to making a study reproducible. The mistaken reasoning is: “The script is public, so anyone can reproduce the result.” But a script without accessible data, dependency information, or execution instructions may not run, and a script that runs on synthetic data cannot regenerate estimates from restricted data. The correction is to make a precise claim: say which materials are available, what a reader can reproduce with them, and which barriers remain.
Another pitfall is to report a seed as a guarantee of exact output. A seed controls a random-number stream under particular software conditions; it does not make results invariant to software changes, parallel execution, or different implementations. State the seed and environment, and describe any remaining source of variation. More broadly, a reproducible computation is not proof that the study’s design or causal interpretation is valid. Reproducibility makes analytical work more inspectable; it does not remove confounding, establish positivity, or guarantee correct measurement.
Choosing an Appropriate Level of Sharing
The appropriate sharing route depends on data sensitivity, licensing, and technical needs. A public repository is suitable for materials that can be openly shared and benefits from versioned releases. A controlled-access repository may support independent analysis when data cannot be public. A journal supplement can be useful for concise documentation, but may be less suitable for maintaining a complete computational environment or large codebase. These approaches can be combined: for example, code can be public while data access is controlled.
No single format replaces a clear availability statement. A repository link alone does not explain whether it contains final code, what data it expects, or whether the study estimates can be regenerated. Conversely, a well-written statement without accessible materials does not enable independent execution. Match the claim to the materials: distinguish “code available,” “data available under conditions,” “workflow tested on synthetic data,” and “results independently reproducible with authorized access.”
Check Your Understanding
Answer each question, then compare with the solutions.
- A report says only “analyses were conducted using statistical software.” Name two kinds of information that would make this description more useful.
- Why is a public analysis script insufficient to reproduce an estimate when the study data are restricted?
- A planned analysis used one matching specification, but the final report uses another. What should a reproducibility record include about this change?
- What does recording a random seed help with, and what does it not guarantee?
- A synthetic data file has the same fields as restricted study data. Can running code on it reproduce the study’s estimate? Explain.
Solutions. (1) Give the software name and version, plus consequential package or dependency versions; execution instructions are also useful. (2) The script cannot regenerate the estimate without the study data, and the report should identify the access restriction and any application route. (3) State the planned and implemented specifications, the reason for changing them, and which samples, diagnostics, or results are affected. (4) A seed supports repeatability of stochastic steps in a compatible computational environment, but does not guarantee identical results across versions or implementations. (5) No. It can test that code runs and illustrate the required data structure, but it does not contain the study observations needed to regenerate the study estimate.