Why an Operating-Characteristic Study Needs an Archive
A reported operating characteristic is the result of a chain of choices: a data-generating scenario, a trial design, an analysis model, a decision rule, and a computational implementation. A table of final percentages cannot reveal whether those choices were implemented as intended. An archive makes the chain inspectable and gives another analyst enough information to rerun the study, understand differences, and verify summaries against saved results.
This is not simply a matter of keeping the script that launched the simulation. Code can depend on unrecorded settings, software versions, input files, or random-number behavior. It can also contain local edits that were never committed to version control. A useful archive therefore preserves both the computational materials and the scientific specification: what scenarios were considered, what each replicate did, and how the resulting records became the reported summaries.
The archive should let a reviewer answer, without guessing: which exact scenarios were simulated; which analysis and decision rule were applied; how many replicates were completed; what software and settings were used; which outputs were generated; and how each reported result can be checked. The purpose is not to prescribe one file format. It is to make the necessary information explicit and retrievable.
Specify the Study Before Preserving Its Code
Start with a human-readable specification that is independent of the code. For every scenario, record the data-generating assumptions and parameter values, trial design, endpoint definition, and any nuisance-parameter settings. Separately document the analysis model, analysis prior, estimand, and decision rule. State the operating characteristic being estimated, such as the probability of a specified decision under a particular scenario. If scenarios have weights for an average, preserve those weights and their rationale.
This separation matters because a program can execute successfully while implementing the wrong scenario or decision rule. A specification gives reviewers a target against which to check the program. Use stable identifiers, such as “scenario_001,” rather than relying on row order or descriptive labels that may change. Keep a data dictionary defining each field, its units, allowed values, and whether it describes data generation or analysis.
Preserve the actual decision rule in operational terms, not only a brief label. For example, “declare success if the posterior probability that the treatment effect exceeds the clinically relevant threshold is greater than 0.975” identifies a different rule from “declare success if the posterior probability of any benefit exceeds 0.975.” Archive the precise threshold, inequality, parameter scale, and direction. Similarly, document missing-data, stopping, allocation, and analysis details whenever they affect the simulated trial.
Worked Example: Turning a Scenario into an Auditable Specification
Suppose a hypothetical binary-outcome trial simulation has two arms, 150 participants per arm, and a control event probability of 0.20. In one scenario, the treatment event probability is 0.14. The archived analysis uses a specified Bayesian model and declares success when the posterior probability of a treatment benefit greater than the prespecified clinical threshold exceeds 0.975.
A sufficient scenario record identifies the scenario as, for example, “scenario_001”; records the two event probabilities, allocation ratio, sample sizes, and binary endpoint; and states whether the generation model includes any additional variation, such as overdispersion. A separate analysis record gives the model, prior distributions, estimand, clinical threshold, and exact decision inequality.
To check that the design fields agree, total enrollment is \(150+150=300\), and the allocation ratio is \(150/150=1\). Those simple checks can catch a mismatch between a written description and a configuration file. If the code instead reads a total sample size of 150 and allocates 75 per arm, the archive’s specification makes the discrepancy visible rather than leaving it to be inferred from a final percentage.
The example is hypothetical. The point is to preserve enough detail to reconstruct the intended simulation, not merely to save a scenario name and a result.
Code, Environment, and Random-Number Settings
Save the exact source snapshot used for the reported run, including configuration files and scripts that generate inputs, execute replicates, summarize results, and create tables or figures. A version-control commit identifier is useful provenance, but it is not sufficient by itself: the working directory might have had uncommitted edits, or important files might not have been tracked. Record the commit and whether the working tree was clean; also preserve the run’s actual source snapshot or an immutable release containing it.
Record the language and version, relevant package or library versions, operating system, and computational settings that can affect results. A dependency lockfile or environment specification can help recreate the software environment, but note any dependencies that are not fully captured, such as external services or locally installed software. If parallel computing is used, preserve the backend, worker configuration, and method used to allocate random-number streams.
A seed alone does not guarantee identical results across environments. Random-number generators differ, software updates can change algorithms, and parallel execution may assign random draws differently when task order changes. Preserve the generator name and version where available, the seed or seed-generation procedure, stream or substream allocation, and relevant parallel settings. Record whether the goal is exact, bit-for-bit reproduction in a fixed environment or a rerun that should agree within Monte Carlo variation. These are distinct standards.
Worked Example: What a Seed Does and Does Not Guarantee
A hypothetical study is run with seed 48172 using a named pseudorandom-number generator, a recorded software version, and one worker. A later analyst reruns the archived code with the same seed but a newer generator implementation and four workers.
The seed value matches, but the computational setup does not. The newer implementation may map the seed to a different stream of draws, and parallel task scheduling may assign draws to replicates differently. Therefore, identical seed values alone do not establish that identical trial-level records should result.
For an exact-reproduction target, the archive should specify the generator and software versions and the execution settings, and the rerun should use them. For a statistical-reproduction target, the analyst should instead confirm that the same scenario and decision procedure were executed and compare results with their expected simulation variability. The distinction prevents a changed sequence of random draws from being misdiagnosed as evidence of a changed scientific specification.
Preserve Results and Make an Audit Trail
Keep the replicate-level records needed to check the reported summaries, subject to privacy, storage, and governance constraints. In a simulation, records can usually be designed to avoid personal information, but they may still be large. If retaining every detailed record is impractical, preserve a clearly justified set of sufficient per-replicate fields, the summaries, and the code that produced them. Document any limitation that prevents independent reconstruction.
Each output should carry enough metadata to identify its run, scenario, replicate range, and completion status. Preserve logs of commands, start and end times, warnings, errors, and interrupted or resumed jobs. Distinguish completed runs from partial outputs so that a summary cannot silently include some replicates twice or omit a failed batch. Give results stable names and record which files feed each reported table or figure.
Checksums provide a compact way to detect whether archived files have changed: a checksum is calculated from a file’s contents, and a newly calculated value can be compared with the value recorded at archiving. A match supports the conclusion that the file is unchanged under the chosen checksum procedure; it does not establish that the file is scientifically correct. Record checksums for key inputs, source snapshots, and final outputs, together with the checksum method. For text-based results, also document how records are ordered and serialized if byte-for-byte comparisons are intended.
Assign stable scenario and run identifiers; record the estimand, generation assumptions, analysis, and decision rule.
Preserve source and configuration files, environment information, random-number settings, and parallel-computing details.
Record commands, completion status, warnings, replicate counts, and the relationship between outputs and scenarios.
Recompute selected summaries, check file integrity, and document whether the archive supports exact or statistical reproduction.
A simple numerical check illustrates the value of preserving replicate-level decisions. Suppose a hypothetical scenario has 400 completed trial replicates, of which 236 satisfy the archived decision rule. The estimated probability of that decision is
Under independent replicate generation with the same scenario and decision procedure, the estimated Monte Carlo standard error is
An auditor can check the numerator, denominator, and resulting estimate directly from the saved records. If the archived result file instead contains 234 successes out of 400, the displayed estimate should be \(234/400=0.585\), not 0.590. This check does not by itself prove the simulation code was correct; it verifies consistency between records and summary.
Worked Example: Auditing a Reported Summary
Suppose an archived hypothetical results file has one row per replicate, with a field coded 1 when the decision rule is met and 0 otherwise. It contains 500 rows for one scenario, and summing that field gives 285. The report claims an estimated decision probability of 0.570.
Recompute the estimate from the file:
where \(I_i\) is 1 if replicate \(i\) met the rule and 0 otherwise. The reported value agrees with the saved records. Next, check that the scenario identifier appears on all 500 rows, there are no duplicate replicate identifiers, and the run log confirms 500 completed replicates. If one duplicate row is found and a distinct replicate is missing, the arithmetic can still appear plausible while the file no longer represents 500 unique simulations. An audit therefore checks identifiers and run metadata as well as the summary calculation.
Archive Structure, Diagnostics, and Limits
A practical directory layout separates specifications, source, environment, run metadata, and results, while a top-level README explains how they fit together. For example:
| Archive area | Contents to preserve |
|---|---|
| Specification | Scenario definitions, estimands, decision rules, and data dictionary |
| Source | Run scripts, analysis code, configuration files, and source revision |
| Environment | Language and package versions, dependency lockfile, operating system, and computing settings |
| Run metadata | Run identifier, seed and stream settings, commands, logs, and completion status |
| Results | Replicate-level or justified reduced records, summaries, and checksums |
Common warning signs include a seed with no generator information, output files with no scenario identifiers, a result table that cannot be linked to source code, missing dependency versions, and logs that do not distinguish completed from partial runs. Another warning sign is a figure that cannot be regenerated from the archived result files or whose plotted values differ from the report. Investigate these failures before treating the archive as independently auditable.
No archive can make a flawed model or an inappropriate scenario specification valid. Reproducibility addresses whether an analysis can be traced and repeated; it does not establish that the assumptions are clinically plausible or that the implementation is free from error. Nor is it always feasible to retain every intermediate object. The archive should state what was retained, why any details were omitted, and which checks remain possible.
Check Your Understanding
Answer each question using the archive principles above.
- A report gives one estimated decision probability for each scenario. Name four types of information that should be archived to make those results interpretable and auditable.
- Why is preserving a seed without the random-number generator and execution settings not enough to promise an identical rerun?
- A saved results file has 300 replicate rows, 171 of which meet the decision rule. What estimate should be reported, and what basic file checks would you perform?
- What does a matching checksum establish, and what does it not establish?
- Explain the difference between exact reproduction and statistical reproduction.
Solutions. (1) Appropriate items include scenario-generation assumptions, trial design, analysis model and prior, estimand, decision rule, software environment, random-number configuration, run logs, and replicate-level results. (2) Different generator implementations, software versions, or parallel stream assignments can produce different draws from the same seed; the seed alone does not specify the full computational process. (3) The estimate is \(171/300=0.570\). Check the scenario identifier, unique replicate identifiers, expected number of completed replicates, and consistency with run logs. (4) A matching checksum supports that the file contents have not changed under the recorded checksum method; it does not show that the file or analysis is correct. (5) Exact reproduction targets identical numerical output under a sufficiently matched computational setup. Statistical reproduction targets a rerun of the same specified procedure whose results agree within expected simulation variability, even if individual draws differ.