Why Reproduce a Submitted Run?
A submitted analysis may include model code, data, configuration files, and a report, but those artifacts alone do not establish that another reviewer can execute the analysis from its documented instructions. Reproducing the run means independently carrying out those instructions from a controlled starting point and recording what was required. The purpose here is to establish the execution path, not to decide whether the resulting outputs agree with the submitted outputs; that comparison is a separate review task.
A clean starting point matters because a run can appear reproducible only because the original analyst's machine supplied an unrecorded file, environment variable, cached intermediate, or manual change. Independent execution should make such dependencies visible. The earlier material on reconstructing analysis provenance and verifying software and environment metadata helps identify the package and execution environment to use; this tutorial focuses on carrying out the documented run without silently filling gaps.
“Clean” does not mean deleting or changing the submitted evidence. Preserve the received package unchanged, work from a controlled copy, and keep the reproduction record separate from both. The working location should be new or demonstrably empty before the package is staged. Do not reuse a previous run directory: stale outputs can be mistaken for newly generated files or can influence a script that discovers files automatically.
A Controlled Execution Workflow
Before starting, identify the run instructions, required inputs, expected entry point, and any specified output location. Confirm the software and environment information against the submitted documentation, but do not infer that a missing instruction is authorized merely because a familiar default seems reasonable. Record the package identifier or checksum, the location of the untouched source package, the clean working path, and the date and operator for the reproduction record.
Keep the submitted package unchanged. Make a controlled working copy in a new directory, and record which files were staged and any files excluded under explicit instructions.
List the specified commands or application actions in order, required inputs, configuration settings, and stated prerequisites. Note ambiguities and missing dependencies before deciding how to proceed.
Use the documented entry point and settings. Record the command or action, working directory, time, and whether it completed, stopped, or produced a warning. Do not edit files just to make the run succeed.
If an undocumented choice or change is necessary, record the problem, the exact intervention, its rationale, and who authorized it. When an intervention would change analysis inputs or method settings and no authority is documented, pause and request clarification rather than guessing.
Record the final status, any generated artifacts, warnings, failures, and unresolved issues. Preserve the log and working copy separately from the original package so a reviewer can distinguish submitted evidence from reproduction evidence.
A useful execution record is factual rather than evaluative. It should allow another reviewer to tell what was attempted, in what order, with which stated inputs, and where the actual procedure departed from the documentation. For instance, “changed the input path from A to B because the documented file was absent” is more useful than “fixed the path.” The former identifies both the problem and the intervention; the latter conceals the decision.
Worked Example: Reproducing a Documented Run Without Editing It
Suppose a hypothetical package contains a run guide, a model script named fit.mod, a dataset named analysis.csv, and a configuration file that specifies seed 418. The guide instructs the reviewer to run the script from the package’s run directory. A new empty directory is created, the four package files are copied into it, and the original package is left untouched.
The reviewer records the package identifier, the new working path, and the four staged filenames. The guide identifies fit.mod as the entry point and specifies the working directory; the configuration supplies the seed. The reviewer follows those directions without changing the script or configuration, then records the invocation, start and finish times, completion status, and any warnings.
If execution completes, the record supports the claim that the documented procedure was executed in a clean location. It does not yet establish that the newly generated outputs match the submitted outputs, nor does it validate the model. Those are separate review questions. If the script instead requests a file that is not among the documented inputs, the reviewer records the request and stops; searching personal folders for a plausible substitute would introduce an undocumented input.
The key decision is not whether a convenient workaround exists. It is whether the workaround is specified or authorized. In this case no missing dependency occurred, so no intervention was necessary and the execution record can state that the run followed the documented procedure.
Why the Clean-State Condition Matters
A run can be represented conceptually as a procedure applied to inputs under an execution environment. Let \(I\) denote the documented inputs, \(E\) the execution environment, and \(C\) the documented procedure and settings. The run is then \(R = f(I,E,C)\). If a reviewer also relies on an unrecorded file \(U\), the actual execution is instead \(R = f(I,E,C,U)\). A successful result in the latter case does not show that the documented package is sufficient: it shows only that the package plus the hidden dependency was sufficient.
This gives the clean-state rule a direct justification. Starting in an empty location removes prior run artifacts as candidate inputs. Staging only the specified package makes the intended inputs identifiable. Recording each intervention makes any added dependency explicit. Together, these controls help distinguish a reproducible documented procedure from a run that succeeds only because the reviewer supplied something extra.
This reasoning does not require that every execution be bit-for-bit deterministic. Some software uses random-number generation, parallel computation, or platform-dependent numerical routines. The reproduction record should follow the documented controls, such as a stated seed or execution setting, and record deviations or warnings. Whether differences in generated outputs are acceptable is not decided by merely completing this execution procedure.
Worked Example: A Missing Input Path
In a hypothetical package, the run guide says to execute the analysis using inputs in the folder data. The staged package contains model.mod and settings.cfg, but no data folder. The configuration names ../shared/cohort.csv, a relative path that points outside the new working directory. No instruction describes where shared refers to or how to obtain the file.
The reviewer first confirms the observed facts: the expected data folder is absent; the configuration points to ../shared/cohort.csv; and that file is not present in the staged package. The reviewer records these facts and the attempted documented command. The run cannot proceed with the documented inputs as staged.
A possible intervention would be to copy a file with the same name from another location or edit the configuration to point elsewhere. Neither action is justified by the guide. The reviewer therefore does not choose a file based on its name, and does not alter settings.cfg. The execution status is recorded as blocked by an unspecified input location, and clarification is requested from the submitter.
This is a completed review action even though the model did not run: the reviewer has established a reproducible reason why execution could not continue without an undocumented choice. If an authorized clarification later identifies the exact file and path, that direction is recorded, the clean working location is prepared again, and the resulting execution is labeled as using the clarified instruction rather than as having followed the original guide unaided.
Recording and Classifying Interventions
Not every deviation has the same consequence. A typographical correction to a command may be operationally minor, while replacing a dataset, changing a seed, or modifying a model setting can change the analysis itself. Record all departures from the documented procedure, then describe their likely scope without claiming an effect that has not been assessed.
| Situation | Appropriate response | Record to retain |
|---|---|---|
| A documented file is present and the documented command runs | Proceed as written | Input listing, command, working path, status, and warnings |
| A required file is absent or a path is ambiguous | Pause; do not select a substitute by guesswork | Expected and observed paths, error or request, and clarification sought |
| A setting is required but not specified | Request an authorized value; do not infer a default when it affects the run | Setting needed, available documentation, and disposition |
| A technical adjustment is authorized | Make only the stated adjustment and retain the original package | Exact change, reason, authorization, and files affected |
When an intervention is authorized, record it before or as it is made, not from memory after the run. Include the original value or state, the changed value or action, the reason, the source of authorization, and whether a fresh clean working copy was used. If the intervention alters analysis inputs or settings, the record should make that fact prominent. Do not describe an adjusted run as an execution of the unmodified documented procedure.
Worked Example: An Unstated Random-Number Seed
A hypothetical run guide specifies the command and lists the model and dataset, but does not mention a random-number seed. The model configuration contains no seed setting. The software reports that its default seed is used, and the run completes. The reviewer has no instruction to select a particular seed.
The reviewer records that no seed was specified in the guide or configuration, notes the software’s reported default behavior, and records the run as completed with that behavior. The reviewer does not insert a seed merely because another value might make future executions more consistent. Adding a seed would be an intervention, not faithful execution of the submitted procedure.
If the software does not reveal its seed or default behavior, that limitation is recorded as unknown rather than guessed. The run may still have been attempted as documented, but the record must not imply that stochastic settings were fully controlled. If the reviewer needs a controlled seed for a separate sensitivity or rerun exercise, that action should be explicitly authorized and documented as a modified procedure.
Diagnostics, Alternatives, and Limits
Several signs indicate that the execution process may not be clean or adequately documented: generated files are already present before the run; a script reads from a path outside the staged package; a command depends on an unrecorded environment variable; the software requests a value absent from the instructions; or the run succeeds only after a manual file edit. These are not reasons to conceal the problem by “making it work.” They are evidence to record and, when necessary, reasons to pause.
A clean-room or containerized execution can reduce some environmental dependencies, while a virtual machine can reproduce a specified operating system more fully. These approaches are useful when the submitted procedure calls for them or when an authorized environment has been selected. They are not automatic substitutes for the documented setup: creating a container definition or installing an unstated dependency can itself introduce choices. A second analyst following the same documented procedure is another useful independent check, but it does not remove the need to preserve a run record or to resolve missing instructions.
This workflow has limits. It establishes what could be executed under the recorded conditions, not whether the source data are correct, the model matches its specification, the analysis population is appropriate, or the results are scientifically valid. It also cannot prove that every hidden dependency has been found. The value is practical and evidentiary: a controlled starting point, explicit procedure, and candid intervention log make undocumented reliance easier to identify and review.
Common Mistake
A common mistake is to repair a missing path or choose an unstated default, then report that the submitted run reproduced successfully. That reasoning confuses “I obtained a successful execution” with “the documented procedure was sufficient.” The correction is to separate the two facts: record whether the run proceeded as documented, and record any intervention that enabled it to proceed. If the intervention is not authorized or could alter analysis inputs or settings, stop and seek clarification rather than treating a plausible guess as part of the submission.
Check Your Understanding
Answer each question, then check the brief solutions below.
- Why should a reviewer avoid reusing a directory from a previous run?
- A required input is missing, and a similarly named file is available in the reviewer’s home directory. What should the reviewer do?
- The submitted guide omits a random-number seed, but the reviewer adds one to make the run consistent. How should this action be characterized?
- What information should be recorded when a technical adjustment is authorized?
- Does a completed clean-state execution establish that the reproduced outputs match the submitted outputs or that the model is scientifically valid?
Solutions. (1) A previous directory may contain stale files that influence execution or be mistaken for newly generated artifacts. (2) Record the missing input and pause for clarification; do not select the home-directory file by guesswork. (3) It is an undocumented intervention and a modified procedure, not execution exactly as submitted; record it and obtain authorization where required. (4) Record the original state, exact adjustment, reason, authorization, affected files, and whether execution began from a fresh working copy. (5) No. Execution establishes that the documented or explicitly modified procedure ran under recorded conditions; output agreement and scientific validity require separate review.