Why Parallel Randomness Needs a Design
A simulation may produce the same result when run twice on one processor yet produce different replicate results when its calculations are distributed across workers. The cause is often not the pharmacometric model. It is the way random numbers are assigned: parallel workers can consume numbers in a different order, or a shared generator can be accessed unpredictably. A reproducible parallel simulation therefore needs a rule that connects each simulated task to its random-number stream, rather than relying on the order in which workers happen to run.
Here, a task is a logically defined unit of simulation whose random draws and output can be identified independently of the worker that executes it. A task might be a replicate, a virtual patient, or another stable unit chosen by the simulation design. A worker is a computing process or thread that executes tasks. Workers are execution resources; they should not define the scientific identity of a replicate.
A pseudorandom number generator is deterministic: once its algorithm and initial state are fixed, it produces the same sequence. A seed initializes that state. Different seed values alone do not establish that two sequences are independent or nonoverlapping. Parallel stream methods should use a generator or library that documents a way to create separated streams or substreams, such as a supported splitting or spawning procedure, or a counter-based generator keyed by task identity. The statistical independence of pseudorandom streams is an engineering approximation supported by the generator's design, not a guarantee obtained merely by choosing different integers.
Assumptions and Design Requirements
The procedures in this tutorial rely on several conditions. First, the simulation uses a specified generator and a documented stream-allocation method intended for parallel use. Second, every stochastic task has a stable, unique identifier that does not change when the task is scheduled on another worker. Third, each task's random draws are obtained from its assigned stream, not from a shared mutable global generator. Finally, the simulation code, model inputs, software implementation, and numerical environment are held fixed when exact reproducibility is being tested.
The central design choice is whether to attach a stream to a worker or to a task. Worker-level streams can be reasonable when each worker always receives the same tasks in the same order. But if the number of workers changes, or tasks are dynamically reassigned, the sequence consumed by a given task can change. Task-level streams avoid this dependency: a task obtains its stream from its identity, regardless of which worker executes it.
A robust implementation can use a master key or root state to derive child streams using the generator's supported stream facility. Conceptually, the stream address is formed from a fixed root and the task identifier. If tasks have distinct stochastic components, stable labels can identify component-specific substreams as well. The labels and derivation rule must be fixed and unambiguous. Do not initialize each worker by copying the same generator state: that creates identical sequences, not independent streams.
Assigning Streams to Stable Tasks
Choose identifiers such as replicate 1 through replicate \(n\). The identifier must be tied to the simulation task, not to its worker or its position in a changing execution queue.
Specify the generator, root seed or key, and supported stream-splitting or spawning rule. Record the mapping from task identities to streams in a deterministic way.
When a worker begins a task, it uses that task's stream. It does not inherit an uncontrolled state from the previous task run on the same worker.
Store each result with its task identifier. Sort or join by identifier for comparison; do not rely on the order in which workers finish.
Run the same tasks with different worker counts and execution orders, then compare results by task identifier and investigate any mismatch.
If each task has multiple random components, a task may use one stream sequentially, or derive substreams from stable component labels such as “covariate sampling” and “residual error.” The second arrangement can prevent adding a draw in one component from shifting the random numbers used by another. It does not make components statistically independent by magic: the substream construction must be supported by the generator and applied consistently.
Worked Example: Why Worker Streams Can Change Replicates
Suppose four hypothetical simulation tasks each need one uniform draw, and the generator's first eight outputs from a worker's stream are \(0.12, 0.83, 0.41, 0.67, 0.25, 0.94, 0.38, 0.56\). For illustration, each task reports \(Y=100U\), where \(U\) is its draw. A two-worker implementation assigns tasks 1 and 3 to worker A, and tasks 2 and 4 to worker B. Each worker starts from the same illustrative sequence.
With two workers: worker A gives \(U_1=0.12\) to task 1 and \(U_2=0.83\) to task 3; worker B gives \(U_1=0.12\) to task 2 and \(U_2=0.83\) to task 4. Thus task results are \(Y_1=100(0.12)=12\), \(Y_2=12\), \(Y_3=83\), and \(Y_4=83\). The repeated values also reveal the flaw: copying an identical initial state duplicates sequences.
Now imagine the implementation instead gives each worker a properly separated stream. This avoids the direct duplication problem, but worker-stream assignment still makes a task's draws depend on which worker receives it. If task 3 moves from worker A to worker B, it can receive a different position in B's stream. Therefore, separate worker streams alone do not guarantee task-level invariance to reassignment.
Correction: assign a stream to each task identity using the generator's supported splitting method. Task 3 then uses its designated stream whether it runs on worker A or worker B. The numerical values depend on the chosen generator and stream method, but the allocation rule removes worker assignment as a cause of change.
Why Task-Level Results Are Invariant
The justification is direct. Let \(i\) be a stable task identifier, let \(K\) be a fixed root key or state, and let \(G(K,i)\) denote the stream assigned to task \(i\) by a deterministic stream-allocation method. Let \(F\) be the task's computation, including its use of that stream, and let \(X_i\) be its output. Then
Changing the worker assignment or execution order changes neither \(K\), \(i\), the task inputs, nor the function \(F\). Therefore it changes neither the stream \(G(K,i)\) nor the task output \(X_i\), provided the computation itself is deterministic conditional on its inputs and random stream. This establishes task-level invariance under those assumptions. It does not establish that two distinct streams are perfectly independent, nor does it guarantee identical results across different software versions or numerical hardware.
A shared generator has a different structure. Its state advances after each draw, so the value received by a task depends on how many draws other tasks consumed first. If the task order changes, the generator state at a task's start may change. That is precisely the dependence the task-level allocation is designed to remove.
Worked Example: An Order-Dependent Shared Generator
Consider two hypothetical tasks, A and B, sharing a sequential generator that produces \(0.20, 0.70, 0.40, 0.90\). Each task draws two values and computes their mean. If A runs first, it receives \(0.20\) and \(0.70\), so
Task B then receives \(0.40\) and \(0.90\), giving
If B runs first, B instead receives \(0.20\) and \(0.70\), so \(\bar{U}_B=0.45\), while A receives \(0.40\) and \(0.90\), so \(\bar{U}_A=0.65\). The overall pair of means is unchanged, but the values attached to task identities have switched. If the tasks represent distinct simulation replicates, that is not task-level reproducibility.
With task-specific streams, suppose the designated draws for A are \(0.20, 0.70\), and those for B are \(0.40, 0.90\). The means remain \(\bar{U}_A=0.45\) and \(\bar{U}_B=0.65\) whether A or B executes first. This example illustrates allocation logic; in practice the streams must be produced by an appropriate stream facility, rather than manually assigning these values.
Testing Worker-Count and Order Invariance
A useful test changes the execution plan while keeping the scientific task definition fixed. Compare outputs by task identifier, not by row position or completion order. For deterministic task code and a fixed numerical environment, exact equality is appropriate. Where calculations use parallel floating-point reductions or other order-sensitive arithmetic, a small numerical difference may occur even if random draws are identical; in that case compare the random inputs exactly where available and use a prespecified numerical tolerance for derived quantities.
Worked Example: Comparing Two Execution Plans
Four hypothetical tasks each draw two values from their own designated streams and calculate \(Y_i=10U_{i1}+U_{i2}\). The recorded draws are:
| Task | \(U_{i1}\) | \(U_{i2}\) | Calculation | \(Y_i\) |
|---|---|---|---|---|
| 1 | 0.20 | 0.30 | \(10(0.20)+0.30\) | 2.30 |
| 2 | 0.40 | 0.10 | \(10(0.40)+0.10\) | 4.10 |
| 3 | 0.60 | 0.80 | \(10(0.60)+0.80\) | 6.80 |
| 4 | 0.90 | 0.20 | \(10(0.90)+0.20\) | 9.20 |
In execution plan A, tasks run in the order 1, 2, 3, 4 on two workers. In plan B, they run in the order 4, 2, 1, 3 on three workers. The expected comparison is made after matching by task identifier: task 1 must still have \(Y_1=2.30\), task 2 must have \(Y_2=4.10\), task 3 must have \(Y_3=6.80\), and task 4 must have \(Y_4=9.20\). The sum is \(2.30+4.10+6.80+9.20=22.40\) in either plan, but matching individual outputs is the stronger check; equal sums alone could conceal swapped or altered task results.
If the task-level values differ, inspect the stream mapping, task identifiers, accidental use of a shared generator, and whether every stochastic component uses the task's designated stream. If the random draws match but outputs do not, investigate nondeterministic computation or numerical differences. Agreement is evidence of execution invariance for the tested conditions, not proof that the simulation model is scientifically valid.
Diagnostics, Alternatives, and Limits
Several failures are detectable with targeted checks:
- Duplicate streams: identical draws or unusually repeated task outputs can result from initializing workers with copied states. Verify that the parallel stream method creates distinct streams.
- Order dependence: task results change after shuffling task order. Search for shared global random state or stream initialization based on queue position.
- Worker dependence: results change with the number of workers. Check whether streams are attached only to workers rather than to stable tasks.
- Unstable identity: results change because an identifier was assigned from the current execution order. Derive identifiers from the intended simulation design, not dispatch sequence.
- False reassurance from summaries: equal means, totals, or histograms do not show that each task received the intended stream. Compare outputs by task identity, and compare draws where feasible.
Worker-specific streams are simpler and may be adequate when task assignment is fixed and the scientific requirement is reproducibility only for that fixed configuration. Task-specific streams are preferable when tasks may be reassigned or when results must be invariant to worker count and scheduling. Counter-based generators can make random values functions of a key and an indexed counter, which is useful for this purpose when supported and used with a carefully defined mapping. Each approach depends on correct implementation; none makes a poor generator, unsuitable model, or incorrect task definition statistically sound.
Reproducibility also has a scope. Holding the root key, generator, task identities, inputs, code, and numerical environment fixed can support exact repeatability. Changing a generator implementation, software version, task definition, or floating-point execution path may change results. Record enough configuration to identify the intended setup, and test the invariance that the workflow actually requires. Do not interpret identical output as evidence that random-number streams are statistically independent or that the simulation is unbiased.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Why can a shared sequential generator make task outputs depend on execution order?
- A simulation produces different results when moved from two workers to four, although each worker has a different seed. Give one likely design issue.
- For a task-specific stream design, what should remain fixed when task order changes?
- Two execution plans have the same sum of replicate outputs but differ for several task identifiers. Has task-level invariance been demonstrated? Why or why not?
- If task-level random draws match exactly across worker counts but a derived floating-point summary differs slightly, what should be checked?
Solutions. (1) The generator advances after each draw, so a task's assigned values depend on the state left by tasks that ran earlier. (2) Streams may be attached to workers, so changing worker assignments changes which stream positions tasks consume; a shared global generator is another possibility to investigate. (3) The root key or state, stream-allocation rule, task identifier, task inputs, and task computation should remain fixed; only worker assignment and execution order change. (4) No. Equal sums can conceal changed or swapped task results; compare each output by its stable task identifier. (5) Check for order-sensitive floating-point operations or other nondeterministic computation, and compare derived values using an appropriate, prespecified tolerance while confirming that the random inputs match.