Why Parallel Simulation Needs a Stream Plan
Simulation studies of adaptive trials can require many independent replicates, scenarios, or design comparisons. Running these jobs in parallel can reduce elapsed time, but it creates a risk: if random numbers are assigned according to which worker happens to run next, changing the number of workers or the order of execution can change the simulated trial data. Then a computational change may look like a statistical change.
A stream plan makes the assignment explicit. Each simulation job receives random inputs based on a stable identifier, not on the worker that executes it. This supports two useful properties: jobs do not reuse portions of the generator sequence, and the random inputs assigned to a given job remain the same when execution is rescheduled. Those properties require a suitable generator and implementation; choosing different integer seeds by itself does not establish either one.
Streams, Jobs, and the Generator
A pseudorandom number generator is a deterministic algorithm that produces a sequence of values intended to behave statistically like random draws. Given its initial state, the sequence is reproducible. A seed initializes that state; it is not itself a guarantee of independence. A stream is a long segment of a generator’s sequence, and a substream is a segment reserved within a larger stream according to the generator’s documented design. A simulation job is a defined unit of work, such as one replicate under one scenario.
These are distinct requirements. Reproducibility does not imply nonoverlap: two jobs can be restarted from the same state and reproduce the same sequence. Nonoverlap does not by itself imply statistical independence: consecutive portions of a poor generator can have undesirable dependence. In practice, use a well-studied generator whose stream or substream mechanism has documented properties, and follow its prescribed initialization and allocation method.
A useful abstraction is to define the random input for job \(j\), replicate \(r\), and draw position \(k\) as \(U_{j,r,k}=G(K,j,r,k)\), where \(G\) is the specified generator mechanism and \(K\) is a master key or initial state. This notation describes the desired mapping, not a claim that every generator accepts these exact arguments. Some generators provide jump-ahead or stream-splitting operations; counter-based generators can derive outputs from a key and counter. Use the actual documented interface for the chosen generator.
The identifiers must describe the statistical work, not the machine layout. For example, label jobs by scenario and replicate, such as (scenario 3, replicate 417). Do not label them “worker 2’s next job.” If a scheduler assigns replicate 417 to a different worker, its identifier and random inputs should remain unchanged.
Allocation and Schedule Invariance
A robust allocation begins with a fixed mapping from job identifiers to streams or substreams. Each job then consumes its assigned sequence according to a defined order. The job must not draw from a shared mutable generator whose state is advanced by whichever worker requests a value first. Such a shared state makes outputs depend on execution timing.
Choose the generator, version or implementation, and documented stream-allocation method. Record the master seed or key and relevant parameters.
Choose identifiers for the statistical units of work, usually scenario and replicate. Do not use worker number, process start order, or completion order.
Map each job to a documented stream, substream, or counter range with enough capacity for its maximum random-number use.
Ensure the job initializes from its assigned portion and does not draw from another job’s state. Record the random inputs or job-level outputs needed for checks.
Run a small set of the same jobs with different worker counts and orders. Compare each job by identifier, not just the final aggregate.
The justification is direct. Let \(R_j\) denote the complete sequence of random inputs assigned to job \(j\), and let \(f_j\) be its deterministic simulation computation. Its result is \(Y_j=f_j(R_j)\). If the allocation makes \(R_j\) a function only of the fixed generator specification and \(j\), then changing the schedule leaves \(R_j\), and hence \(Y_j\), unchanged. This conclusion assumes that job computation itself is deterministic given its inputs: hidden random draws, shared mutable state, nondeterministic data access, or schedule-dependent decisions can break it.
The implication concerns each job’s result. Aggregation can introduce a separate issue: floating-point addition is not exactly associative, so summing the same values in a different order can change the last few digits. A schedule-invariance check should therefore compare job-level results first. If bit-for-bit identical aggregate output is required, use a fixed reduction order or a numerically stable, deterministic summation method.
Worked Example: Assigning Replicates Independently of Workers
A hypothetical design-assurance analysis has 12,000 replicates for each of 4 scenarios. The team plans to use a generator with a documented facility for creating nonoverlapping substreams. It assigns one substream to each scenario-replicate pair. How many substreams are required, and what should happen when the team changes from 8 workers to 20?
There are \(4\times12{,}000=48{,}000\) distinct jobs. Define job identifiers as \((s,r)\), where \(s\in\{1,2,3,4\}\) is the scenario and \(r\in\{1,\ldots,12{,}000\}\) is the replicate. The allocation must provide 48,000 job substreams, or a documented equivalent that maps all these identifiers to nonoverlapping portions.
Suppose the fixed mapping orders jobs by scenario and then replicate. The substream index is \(q=(s-1)12{,}000+r\). For scenario 3, replicate 417, \(q=(3-1)12{,}000+417=24{,}417\). The job uses the substream assigned to index 24,417 regardless of which worker runs it.
Changing from 8 workers to 20 changes only the execution assignment. It must not change \(q\), the substream initialization, or the sequence consumed by job \((3,417)\). The mapping is valid only if the chosen generator documents that these substreams are nonoverlapping and each has enough outputs for the job’s maximum draw count. The arithmetic for assigning indices does not prove those generator properties.
Common Random Numbers Across Scenarios
When comparing designs or scenarios, common random numbers (CRN) deliberately align selected random inputs across conditions to reduce Monte Carlo variability in a paired comparison. Stream allocation must preserve that alignment rather than accidentally giving each scenario unrelated inputs. The earlier material on paired simulation and common random numbers establishes why paired outputs can be informative; here the implementation question is how to keep the pairing stable when jobs run in parallel.
A practical scheme identifies a replicate first and then organizes its random inputs by component. For each replicate \(r\), a shared component—such as a baseline covariate draw—can be generated from a replicate-specific stream and reused across scenarios. Scenario-specific components can use separate, explicitly assigned streams. This avoids relying on the order in which a program happens to request draws. The exact coupling depends on the intended comparison: CRN should be applied to inputs that are meant to be shared, not indiscriminately to every random quantity.
Worked Example: Preserving a Paired Comparison Across Scenarios
In a hypothetical simulation, two trial designs are compared over 5 replicates. For illustration, a shared baseline input \(B_r\) and a scenario-specific input \(E_{s,r}\) are assigned by replicate and scenario. The simulated outcome is \(Y_{s,r}=B_r+E_{s,r}\). Suppose the allocated values are:
| Replicate \(r\) | Shared \(B_r\) | \(E_{1,r}\) | \(E_{2,r}\) |
|---|---|---|---|
| 1 | 0.20 | 0.10 | 0.30 |
| 2 | 0.70 | 0.20 | 0.10 |
| 3 | 0.40 | 0.50 | 0.20 |
| 4 | 0.90 | 0.10 | 0.40 |
| 5 | 0.30 | 0.30 | 0.20 |
The paired differences, defined as design 2 minus design 1, are \(D_r=Y_{2,r}-Y_{1,r}\). Since \(B_r\) is shared, it cancels: \(D_r=(B_r+E_{2,r})-(B_r+E_{1,r})=E_{2,r}-E_{1,r}\). The five differences are \(0.20,-0.10,-0.30,0.30,-0.10\), whose mean is \(\bar{D}=(0.20-0.10-0.30+0.30-0.10)/5=0.00\).
If replicate 4 is run on another worker, its shared baseline input must still be 0.90 and its two scenario-specific inputs must still be 0.10 and 0.40. Assigning streams by worker could break that alignment. This small illustration shows the mapping principle; it does not establish that CRN reduces variance for every comparison. That depends on the covariance induced by the shared inputs and the estimand.
Verification, Diagnostics, and Limits
Check the allocation at the level where errors arise. A useful diagnostic run selects several job identifiers and records their assigned stream identifiers, initial states or keys, and job-level outputs. Run those same jobs serially, with multiple worker counts, and in deliberately different orders. Compare the records by job identifier. Identical job-level random inputs and results support schedule invariance; they do not prove the generator’s statistical quality or prove nonoverlap for every possible job.
Investigate a mismatch by checking whether stream initialization depends on worker identity, whether any code path draws from a global shared generator, whether a job consumes a variable number of draws and then shares its state, and whether stream capacity was exceeded. Also check whether retries reuse a job’s stream in a way that duplicates a replicate unintentionally. A variable number of draws within a job is not inherently a problem if that job has an isolated stream and adequate capacity, but it can cause overlap if subsequent jobs are allocated by advancing a shared state.
Avoid assuming that two different seeds automatically create independent sequences. Seed hashing or seed lists may be adequate only when the generator’s documentation supports the method and the chosen seed space and initialization procedure are appropriate. Prefer generator-provided stream splitting, jump-ahead, or counter-based construction with stated guarantees. Nonoverlap is a structural property of allocation; statistical independence is a property assessed under the generator’s design and should not be asserted merely from different seed values.
Stream methods also have limits. A substream may be finite, and a job that consumes more outputs than allotted can compromise separation. Counter-based methods require careful, collision-free encoding of identifiers and counters. Reproducibility can change across generator versions, libraries, or numerical environments if the generator or transformation algorithms change. These details belong in the computational specification; stream allocation alone does not make an entire simulation study reproducible.
Worked Example: Diagnosing a Schedule-Dependent Result
A hypothetical simulation estimates a 30-day failure rate using 10,000 replicates. A serial run reports 0.0821, while an 8-worker run reports 0.0794. The team needs to determine whether this difference is Monte Carlo variation or a stream-allocation problem.
First compare a fixed set of job identifiers, for example replicates 1 through 100, between runs. For each replicate, compare its stream assignment and simulated failure indicator. If the same replicate receives different inputs, the allocation is schedule-dependent; the aggregate difference cannot be interpreted as ordinary variation from independent reruns.
Suppose the check shows replicate 17 has different inputs, and inspection finds each worker initialized its generator using the worker number. Replicate 17 was on worker 2 in the serial configuration but worker 6 in the parallel configuration. The diagnosis is a mapping error: worker identity, not replicate identity, determines the stream. The correction is to assign streams using the stable replicate identifier and rerun the comparison.
If all sampled job-level inputs and outputs match, then investigate aggregation order and other sources of nondeterminism. If the runs used genuinely different master keys, unequal aggregate estimates may instead reflect Monte Carlo variability, but that is not a schedule-invariance check: it is a comparison of different randomizations. These explanations should not be conflated.
Common Mistakes
- Using one shared generator across workers. The next draw may depend on timing. Give each statistical job an isolated, documented allocation.
- Giving every worker a different seed and calling the streams independent. Distinct seeds alone do not establish nonoverlap or independence. Use a generator-supported splitting method.
- Assigning streams by worker number. A job can receive different random inputs when worker count changes. Assign by scenario and replicate identifiers.
- Checking only the aggregate estimate. Similar totals can conceal changed job-level assignments. Compare fixed jobs by identifier.
- Expecting identical output from every software environment. Generator versions, transforms, and floating-point reductions can matter. Define the computational scope of the reproducibility claim.
The central design choice is simple but consequential: random inputs belong to simulation jobs, not to processors. A documented generator allocation, stable job identifiers, and job-level schedule checks make parallel execution easier to audit and protect paired comparisons from accidental loss of alignment.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A team has 3 scenarios and 2,500 replicates per scenario. How many scenario-replicate jobs are there, and what should identify a job?
- Why does assigning a different integer seed to each worker not, by itself, establish nonoverlapping streams?
- A job produces different results when moved from worker 1 to worker 4. Give one likely cause and a suitable diagnostic.
- Under what condition do shared random inputs preserve a paired comparison across scenarios?
- Why might job-level outputs match while the final aggregate differs in its last digits?
Solutions. (1) There are \(3\times2{,}500=7{,}500\) jobs. Identify each by a stable scenario-replicate pair, not by worker. (2) Different seeds do not establish how the generator’s sequences relate; nonoverlap requires a documented generator allocation method or other justified construction. (3) A likely cause is worker-dependent initialization or use of a shared mutable generator. Compare the job’s stream assignment and random inputs across workers while keeping its identifier fixed. (4) The inputs intended to be shared must be generated or assigned identically for the same replicate across the compared scenarios, with other inputs allocated according to the comparison design. (5) Floating-point addition can produce slightly different rounding when values are summed in a different order; a fixed reduction order can address this when bitwise aggregate equality is required.