Reproducibility Is Not the Same as Independence
A simulation result should be reproducible: another analyst, or the same analyst later, should be able to recover the computational result from the recorded inputs and settings. But reproducing a run is not the same as generating a new independent replicate. Confusing these goals can make a simulation appear much more precise than it is.
A pseudorandom number generator (PRNG) is a deterministic algorithm that produces a sequence of numbers intended to behave like random draws for the simulation. A seed initializes its internal state. Given the same generator, initial state, and sequence of calls, the generator returns the same sequence. If the data-generating model and analysis are deterministic conditional on those values, the entire simulated result is repeated as well.
The practical aim is therefore twofold: preserve enough information to replay each result, while assigning streams so that distinct replicates do not inadvertently replay the same random draws. The earlier material on simulation workflows and replicate counts gives the broader setting: Monte Carlo summaries are meaningful only when the simulation runs correspond to the intended replicate-generating process.
Why Resetting a Seed Repeats a Run
Represent the internal generator state before a draw by \(S_k\). A deterministic generator updates state according to a transition rule \(T\), and produces a value from the state using an output rule \(G\):
If the same seed initializes the same state \(S_0\), the first generated value is the same. Applying the same deterministic transition repeatedly gives the same state at every subsequent position, and therefore the same sequence of generated values. If the simulation makes the same calls in the same order, uses those values in the same way, and the analysis is deterministic, its output must also be identical.
This is a replay, not a collection of new independent realizations. In practice, a simulation may involve multiple random components, such as covariates, random effects, residual errors, and missingness. A documented stream-allocation method should make clear which random values belong to which replicate and component. It should also make it possible to reproduce the allocation rather than rely on undocumented calls to a global generator.
Worked Example: Detecting a Replayed Replicate
Suppose an analyst wants to estimate a design's probability of meeting a prespecified success criterion. The analyst runs the same deterministic simulation and analysis four times, resetting the same seed before each run. In the first run, the generated case meets the criterion, so the success indicator is 1.
Because the seed, generator, call pattern, scenario, and deterministic analysis are all unchanged, each reset replays the same case. The four indicators must therefore be \(1,1,1,1\), not a mixture of successes and failures. Their arithmetic mean is
The calculation is correct as an average of the four recorded indicators, but it is not a valid estimate based on four independent simulated cases. The four runs contain only one distinct realization repeated four times. If the first case had failed, the indicators would instead be \(0,0,0,0\), with mean 0. A mixed sequence such as \(1,1,0,1\) would contradict the stated conditions; it would require investigating whether the seed reset, call pattern, or analysis actually differed, or whether some part of the pipeline was nondeterministic.
Allocating Streams to Replicates
A sound stream strategy needs to answer a simple question: how does each intended replicate obtain its random numbers without duplicating another replicate's draws? Well-designed PRNG systems may provide stream-splitting, jump-ahead, or counter-based methods. Such methods can assign distinct, reproducible portions of a generator's output to different replicate identifiers. Their guarantees depend on the particular generator and implementation, so the method must be checked in its documentation.
Avoid assuming that seeds such as 1001, 1002, and 1003 automatically create independent streams. For some generator families, direct seeding this way can be an acceptable documented approach; for others it may not provide the desired guarantees. The point is not that sequential seed values always fail, but that the numerical difference between seeds is not itself evidence of non-overlap or independence. Use a supported stream mechanism when available, and record exactly how stream identifiers are mapped to runs.
A global stream is simple: all stochastic components consume values from one sequence. This is easy to reproduce if the call order remains fixed, but inserting a new random draw early in the code can shift the values used by every later component. Per-replicate streams isolate one replicate from another. Separating streams by component as well can make debugging and code changes more controlled, provided each replicate-component pair receives a unique, documented stream assignment.
A stream key can be built from identifiers such as scenario, replicate, and component. The allocation rule must be unambiguous and collision-free for the planned simulation. If a design comparison deliberately uses common random numbers, matching streams across designs is intentional for the comparison; that deliberate pairing should not be confused with accidentally assigning the same stream to two replicates that are meant to be independent.
Worked Example: Four Replicates From Distinct Streams
Consider a hypothetical simulation estimating the probability that a candidate design achieves a target precision. The analyst uses a PRNG with a documented stream-splitting facility and creates four distinct stream identifiers from one recorded master seed. Assume the generator's documented procedure assigns non-overlapping streams suitable for the intended independent replicates. The success indicators from the four streams are \(1,1,0,1\).
The estimated success probability is the mean of the indicators:
Thus, in this hypothetical run, three of the four independently generated cases meet the success criterion. The estimate is 0.7500, or 75.00%, for this set of four replicates. This result differs appropriately from replaying one seed four times: the stream-allocation method generates distinct cases, rather than reproducing one case. Four replicates are used here only to demonstrate the calculation, not to recommend a replicate count for a study.
For a reproducible record, the analyst saves the generator name and version, master seed, stream-splitting method, the four stream identifiers, the scenario settings, and the simulation and analysis code versions. If another analyst reconstructs the same four streams and executes the same deterministic workflow, the same indicators should be recovered.
Making a Run Reproducible in Practice
A seed is necessary information for replay, but it is rarely sufficient. Changes to the generator, software version, parallel execution, or order of random-number calls can change results even when the displayed seed is unchanged. Record enough context to identify both the generator state and the computation that consumed its output.
Record the scenario, data-generating model, replicate identifiers, and any settings that affect the random calls.
Save the PRNG family, implementation and version, seed or master seed, and stream-allocation method. Use a supported method for separate streams rather than relying on undocumented seed arithmetic.
Map each scenario-replicate-component combination to its intended stream. Check that streams intended for distinct independent replicates are not being reused.
Record code revision, software and package versions, relevant analysis settings, and parallel-processing configuration. These can affect the order of calls or numerical results.
Replay a selected replicate to confirm that its output is recovered, and check that different replicate identifiers receive the intended distinct streams.
Parallel execution needs particular care. If workers inherit identical generator state, separate workers can accidentally produce identical sequences. A reliable parallel setup assigns streams through the generator's documented parallel mechanism; it does not assume that launching a new worker automatically creates a new stream. Save the worker-to-stream assignment or the deterministic rule that reconstructs it.
Worked Example: Auditing a Parallel Stream Plan
A hypothetical study has two scenarios, three replicates per scenario, and two random components per replicate: random effects and residual error. The team plans to run the six scenario-replicate combinations on parallel workers. Its documented generator supports splitting streams and assigning substreams by a unique key.
The team defines the key as the tuple \((\text{scenario ID},\text{replicate ID},\text{component ID})\). For example, the random-effects stream for scenario A, replicate 2 has key \((A,2,\text{effects})\), and the residual-error stream for that same replicate has key \((A,2,\text{residual})\). The team verifies that the allocation procedure maps these keys to distinct streams, and saves the master seed, keys, generator version, and allocation rule.
If the task scheduler later assigns replicate \((A,2)\) to a different worker, that replicate still receives the stream defined by its key, not whichever state happens to be next in a worker's local sequence. This makes results less sensitive to the order in which tasks finish. It also prevents the common error of initializing every worker with the same seed and thereby duplicating random sequences.
The plan is reproducible because each replicate's random inputs can be reconstructed from the recorded key and generator settings. It avoids accidental reuse because each intended stream has a unique key and the generator's documented allocation is checked. Merely writing down six different worker seeds, without checking the generator's stream guarantees, would not establish the same conclusion.
Diagnostics, Alternatives, and Limitations
Repeated results can be a warning, but they are not conclusive proof of a seed error. Some performance outcomes may naturally be identical across independent replicates, particularly when the outcome is a coarse indicator or when a design performs very consistently. Diagnose the random-number setup itself, not only the final summary.
- Compare generated inputs. For selected replicate identifiers, inspect a small set of generated random inputs or data summaries. Exact repetition across many supposedly distinct replicates can indicate stream reuse.
- Check the allocation map. Confirm that each replicate has the intended stream identifier and that parallel workers were not initialized from identical states.
- Replay a recorded run. With the same environment and stream assignment, rerun one replicate and verify that its outputs match. A mismatch can point to changed software, call order, hidden randomness, or nondeterministic computation.
- Check every source of randomness. A secondary library, stochastic optimizer, or analysis routine may use a separate generator. Seeding only the main generator does not control those sources.
The main alternatives involve different trade-offs. One global sequence is compact and straightforward to record, but changes in call order can affect later results. Per-replicate streams improve isolation and parallel allocation. Per-component streams can further limit the effects of changes in one part of the simulator, at the cost of a more elaborate allocation record. In design comparisons using common random numbers, a stream may be intentionally shared across candidate designs for corresponding replicates; the mapping must preserve that intent while still distinguishing separate replicate identifiers.
No stream strategy makes results independent of modeling assumptions, and a fixed seed does not validate the simulation model. Nor does one identical outcome prove that random values were duplicated. Some environments also introduce nondeterminism through thread scheduling, parallel reductions, or numerical libraries. In such cases, a replay may be statistically equivalent without being bit-for-bit identical. State which level of reproducibility is expected and test it in the actual computational environment.
Common Pitfall
Correction: Resetting the same seed with the same generator and call pattern reproduces the same sequence. Under a deterministic simulation and analysis, it reproduces the same result. Use a documented stream-allocation strategy to generate distinct replicates, while retaining the seed and stream information needed to replay each one.
The opposite mistake is to treat any repeated final statistic as proof of a replay. Independent replicates can yield equal summaries by chance. Verify the stream assignments and, where appropriate, generated inputs before drawing that conclusion. Reproducibility allows a result to be checked; correct stream management ensures that the set of results corresponds to the simulation experiment that was intended.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A deterministic simulation is run three times after resetting the same seed, with the same generator and call pattern. The first run has success indicator 0. What indicators should the next two runs have?
- Why does using three different integer seeds not, by itself, prove that three replicates use independent streams?
- Name four items, in addition to a seed, that should be recorded to support reproducibility.
- In a parallel simulation, two workers produce suspiciously identical generated inputs. Give one plausible cause and one diagnostic or correction.
Solutions. (1) Both remaining indicators should be 0: under the stated deterministic conditions, resetting the same seed replays the same random sequence and result. (2) Independence and non-overlap depend on the PRNG and the seed-to-stream method; numerical differences between seed values do not establish those properties. Use and document a supported stream mechanism. (3) Suitable items include the generator family and version, stream-allocation method and identifiers, simulation code revision, scenario settings, software or package versions, and parallel configuration. (4) Both workers may have inherited or been initialized with the same generator state. Inspect their stream assignments and generated inputs, then assign distinct streams with the generator's documented parallel-stream procedure.