Reproducible Simulation Records

Why Simulation Records Matter

A design-assurance result is evidence about a trial design under specified assumptions. A reported operating characteristic, such as a simulated rejection rate, depends not only on the design but also on the scenario inputs, implementation, software environment, and random-number generation. If those details are missing, another analyst may be unable to determine what was run, repeat it, or explain why a rerun differs.

A reproducible record is therefore more than a seed written in a notebook. It connects a result to the exact inputs and computational process that produced it. This is especially important for adaptive trials, where a small implementation or configuration change can alter how interim data trigger a decision and therefore affect the reported design performance.

What a Run Record Should Contain

A useful record separates the specification of a run from its execution and its results. The specification says what should be simulated; execution metadata says how a particular run was carried out; results summarize what it produced. Keeping these distinct makes it possible to identify whether two results came from the same design and settings, even if they used different random-number streams.

• Identity and purpose: a unique run identifier, date and time, responsible analyst or process, and a short description of the design-assurance question.
• Code revision: a version-control commit identifier or an archived source-code snapshot, including any uncommitted changes. A branch name alone is not enough because the branch can move.
• Software environment: programming language and version, relevant package or library versions, operating system, and, when exact numerical agreement matters, details such as processor or numerical-library configuration.
• Scenario inputs: scenario identifiers and all numerical and categorical inputs, with units and conventions. Record both values and their meanings; for example, whether a rate is per patient or per patient-year.
• Random-number configuration: generator or library, seed or initial state, and stream or substream assignments. For parallel runs, record how work was divided and how each worker’s stream was assigned.
• Run settings: replication count, stopping rules, parallel worker count, batching, tolerances, and any options affecting the computation.
• Outputs and status: completion or failure status, output-file locations or identifiers, summary estimates and their units, and any warnings or exceptions.

For an adaptive-trial simulation, scenario inputs might include treatment effects, event rates, accrual assumptions, outcome variability, and the settings that govern interim decisions. Store the actual configuration used, not merely a description such as “the standard scenario.” A clear record also distinguishes a requested setting from the value the software actually used after defaults or overrides were applied.

Why a Seed Alone Is Not a Reproducibility Record

A simulation’s output can be represented schematically as a function of its code \(C\), environment \(E\), inputs \(I\), random-number setup \(R\), and run settings \(S\):

A seed helps specify \(R\), but it does not specify the other arguments. Even with the same seed, changing the code, scenario, generator implementation, or order in which random numbers are requested can change \(Y\). In parallel computation, a single seed may not determine which stream supplies each task unless the stream assignment and execution strategy are also recorded.

A useful way to understand this is to imagine a deterministic generator that turns an initial state into a sequence of random-number values. The simulation consumes that sequence as it runs. If the code makes an extra random draw in one branch, later draws can be assigned to different simulated quantities. Thus identical initial states do not guarantee identical outputs when the code path or draw order differs.

This representation is a justification for recording the generator, state, code, inputs, and settings together: the result depends on the full computation, not on the seed in isolation. Exact reproduction also depends on whether the relevant software operations behave identically in the environments being compared.

Recording Parallel Randomness and Run Settings

Parallel simulation can reduce elapsed time, but it makes random-number provenance more demanding. Random Number Streams for Parallel Simulation established why workers need a controlled stream arrangement rather than an undocumented collection of independently chosen seeds. A reproducible record should retain the generator and stream method, the master seed or initial state, the mapping from workers or tasks to streams, and the parallel settings used for the run.

Suppose a hypothetical simulation uses four workers to generate 20,000 trials. A record that says only “seed 731” leaves open how the 20,000 trials were divided, which stream each worker used, and whether a change in worker count changes task-to-stream assignments. Record those choices explicitly. If results are intended to be stable under changes in worker count, the run design must use an assignment scheme that supports that goal; the record cannot make an unstable assignment stable after the fact.

Reproduction, Regeneration, and Numerical Agreement

“Reproducible” can mean different levels of agreement. Exact regeneration means the same run can be executed from retained inputs and artifacts and produces the same outputs, often bit for bit. Numerical reproduction means the rerun agrees to a stated tolerance. Conceptual reproduction means the same design, scenarios, and analysis are implemented again, but independent code or software may produce different random draws and therefore different Monte Carlo estimates.

These distinctions matter because floating-point calculations can differ slightly across processors, compilers, library versions, or parallel reduction orders. A small numerical discrepancy does not automatically imply a different design. Conversely, a similar final percentage does not establish that the same scenario, code, or run settings were used. The record should state which level of agreement is expected and, where relevant, the tolerance used to compare numerical outputs.

Practical Integrity Checks and Common Failure Modes

A record is most useful when its pieces can be cross-checked. Use a unique run ID consistently in configuration files, logs, and output files. Preserve the exact configuration file and source revision, and record a checksum or other content identifier for archived artifacts. A checksum can help detect whether a file has changed; it does not explain the file’s contents or prove that the computation was correct.

Before treating a result as independently regenerable, check that the run record answers these questions:

• Can the code revision be retrieved, and are uncommitted changes accounted for?
• Are all scenario inputs, units, defaults, and overrides explicit?
• Are the generator, initial state, and parallel stream assignments specified?
• Are replication counts and stopping or batching rules recorded?
• Can each reported summary be linked to the corresponding output and run ID?
• Is the expected degree of agreement on rerun stated?

A common mistake is to reason, “The seed is recorded, so the result is reproducible.” The correction is that a seed addresses only part of the random-number setup. Without the code revision, scenario values, generator, software environment, and run settings, another analyst may not be able to generate the same sequence, run the same model, or even identify the intended computation.

Another failure is to document only the final estimate. A percentage without the event count, denominator, scenario identity, and run configuration is difficult to audit and may conceal that two results used different settings. Retain both the summarized results and the underlying output or a clearly identified archived artifact, subject to applicable data-governance requirements.

Choosing a Practical Recording Strategy

A manually maintained log can be simple and useful for small exploratory runs, but it is vulnerable to omissions and transcription errors. A machine-generated manifest, created from the configuration and execution environment, can improve completeness and consistency. Neither approach is sufficient by itself if it fails to preserve the source revision, actual inputs, or output links.

Environment capture also involves trade-offs. A package lockfile records package versions and dependencies, while a container or archived environment may capture more of the software stack. These tools improve the chance of re-creating an environment but do not necessarily reproduce hardware behavior, external services, or all system libraries. Store the environment description alongside the run record and specify whether bitwise equality is required or a tolerance-based comparison is acceptable.

The goal is not to record every detail that could ever affect a computer. It is to capture enough information to make the result traceable, make a meaningful rerun possible, and diagnose discrepancies. The appropriate detail depends on the intended use of the result, the computational setup, and the consequences of a mistaken design-assurance conclusion.