Why Stream Design Matters in a MAMS Simulation
A MAMS simulation repeatedly generates trial data, applies interim arm-dropping rules, and records terminal decisions or other operating characteristics. The random numbers determine not just the outcomes, but sometimes which arms continue and how much data are generated later. A run that cannot be reproduced is difficult to debug, verify, or extend when a design choice changes.
A recorded seed is part of the solution, but it is not the whole solution. Reproducibility also depends on the random-number generator, the way streams are assigned to simulation tasks, and whether the implementation consumes random numbers in a stable way. These details matter especially when replicates are run in parallel or when branching decisions cause some parts of a trial simulation to be skipped.
For clarity, let \(s\) index a design scenario, \(r\) a replicate within that scenario, \(k\) a simulation component, and \(j\) a draw within that component. Examples of components include treatment allocation, outcome generation, and accrual. Denote a draw by \(U_{s,r,k,j}\). An implementation need not literally store each draw; the notation describes the intended assignment of randomness.
The goal is to define this assignment before running the simulation. A replicate should receive a reproducible source of randomness based on its stable identifiers, rather than on which worker happens to process it or how many draws a previous replicate happened to use.
Constructing a Reproducible Stream Plan
A robust plan begins with one recorded master seed and a specified generator that supports independent stream creation or stream splitting. Use that generator's documented mechanism to derive separate streams for scenarios, replicates, or components. In abstract form, the plan is:
This is a hierarchy of identifiers, not a prescription to make up new integer seeds by adding numbers to the master seed. Naive seed arithmetic can create collisions or relationships between streams that the generator does not guarantee to be independent. Use a documented stream-splitting, substream, or counter-based facility where available. If the software does not provide one, consult its documentation for a recommended parallel random-number method rather than assuming that different seeds are enough.
Stable identifiers are essential. For example, scenario 3, replicate 27, outcome component should map to the same stream whether that replicate runs first, last, or on a different worker. Do not use a worker number as the replicate's seed: worker assignment can change with the number of workers, scheduling, or a restarted run. Similarly, do not rely on a single global generator whose state advances as whichever task runs next.
Record its name, algorithm, and relevant implementation or software version. Choose a generator with a documented method for creating parallel streams or substreams.
Define scenario, replicate, and, if needed, component identifiers before launching jobs. Identifiers should not depend on worker assignment or execution order.
Use the generator's supported stream mechanism to map each task identifier to its stream. Ensure that different tasks are assigned distinct streams according to that mechanism.
Record the master seed, generator and version, stream-assignment rule, scenario parameters, replicate identifiers, software environment, and code version.
Rerun selected replicate identifiers and confirm that their generated data and trial decisions match under the same recorded conditions.
Separate component streams can make a simulation easier to diagnose. If allocation uses one component stream and outcomes another, changing the allocation routine need not shift the outcome draws for that replicate. This separation is a design choice, not a universal requirement: more streams require a clear, documented mapping, and some generator APIs structure streams differently. The essential requirement is that the mapping be deliberate and reproducible.
Worked Examples
Worked Example: Replaying a Simulated Outcome Sequence
Suppose a hypothetical MAMS simulation includes a binary outcome with event probability \(p=0.20\). For one patient, the simulation generates a uniform draw \(U\) and records an event if \(U<p\). A particular recorded replicate stream produces the first five draws \(0.12, 0.73, 0.08, 0.91, 0.34\).
The event indicators are \(I(U<p)\), where \(I(\cdot)\) equals 1 when the condition is true and 0 otherwise. Thus the indicators are \(1,0,1,0,0\), because only \(0.12\) and \(0.08\) are less than \(0.20\). The simulated event proportion for these five observations is
To replay this result, it is not sufficient to write down “seed 123” without context. The run record must also identify the generator and version, how the replicate stream was obtained from that seed, and the event-generation rule. Under the same recorded setup, the generator returns the same five draws, so the same two events and the same simulated proportion follow. This reproduces a simulated result; it does not make the hypothetical data real or remove Monte Carlo uncertainty.
Worked Example: Why Interim Branching Can Shift a Shared Stream
Consider a hypothetical two-stage simulation with two experimental arms. A simple implementation uses one stream for all outcomes. At the first interim, it generates one draw for each arm. If an arm continues, it then generates that arm's second-stage outcome; if it stops, that draw is skipped. For illustration, suppose the stream begins \(0.14, 0.68, 0.22, 0.91, 0.31, 0.47\), and an outcome is an event when its draw is less than \(0.30\).
The first two draws give stage-one outcomes: arm A has an event since \(0.14<0.30\), and arm B does not since \(0.68\not<0.30\). Suppose the interim rule stops arm A and continues arm B. If skipped draws are not consumed, arm B's stage-two outcome uses the next draw, \(0.22\), and is an event. If a small code change instead generates both second-stage outcomes before discarding the stopped arm's data, arm A consumes \(0.22\) and arm B receives \(0.91\), which is not an event.
The change in simulated arm B data is not evidence that the trial rule changed; it is a consequence of assigning draws by position in a shared sequence. A component stream for each arm, or a stable stream for each arm-stage task, prevents a draw skipped for arm A from shifting arm B's assigned draws. Which organization is best depends on the simulation specification, but the chosen mapping must be explicit and held fixed for the comparison being run.
Worked Example: Reproducing Results Across Parallel Workers
Suppose a hypothetical simulation has four replicates, each assigned a stable replicate identifier \(r=1,2,3,4\) and its own stream. Two workers process the tasks. In one run, worker 1 processes replicates 1 and 3 while worker 2 processes 2 and 4. In a later run, the scheduler assigns replicate 4 to worker 1 and replicate 1 to worker 2. Because each replicate's stream is determined by its identifier and not its worker, each replicate receives the same random numbers in both runs.
Suppose the first run records terminal rejection indicators, in replicate order, as \(1,0,1,0\). The estimated rejection proportion is
The second run must reproduce the same indicator for each replicate, so it gives the same estimate, \(0.500\). If instead each worker is assigned a seed and that worker processes tasks from a shared generator, changing the worker count or task order can change which random numbers a replicate receives. Stable task-to-stream assignment, rather than a fixed worker-to-seed assignment, is what makes this example reproducible under rescheduling.
Why the Construction Works, and What to Record
The key justification is determinism. Let \(G\) denote a fixed generator, \(q(s,r,k)\) the stream-allocation rule, and \(F\) the simulation code for a replicate. If the same task identifiers receive the same stream state and the same code follows the same draw order, then the inputs to \(F\) are identical. Consequently the generated data and decisions are identical. Parallel scheduling does not enter this argument if task \( (s,r,k) \) selects its stream by identifier rather than by execution order.
This is a reproducibility argument, not a proof that streams are mathematically independent. A PRNG is deterministic, and its stream-splitting guarantee is a property claimed and documented for a particular generator. The simulation relies on that guarantee being appropriate for the number and length of streams it uses.
A useful run record includes the master seed; generator algorithm and software version; stream or substream allocation rule; scenario and replicate identifiers; trial parameters and randomization settings; code version; and relevant environment details. Exact bit-for-bit replay may also depend on numerical libraries, hardware, or parallel operations whose order affects floating-point calculations. If those conditions are not fixed, the run may still be statistically reproducible in the broader sense of giving comparable results, without every stored value matching bit for bit.
Diagnostics, Alternatives, and Limitations
Reproducibility can fail in recognizable ways. Rerun a few selected replicate identifiers and compare the generated data, interim decisions, and terminal decisions. If results differ under the same recorded setup, check whether a stream was initialized more than once, whether task assignment depends on worker order, whether a random draw occurs outside the assigned stream, or whether a software or generator version changed. Also check that each replicate has a unique identifier and that restarting a partial run does not accidentally reuse streams for different replicates.
- Repeated streams: If multiple replicates start from the same state, they may produce identical sequences. Confirm that the stream-allocation mechanism gives distinct task streams.
- Worker-based seeding: A result can change when the number of workers changes. Assign streams by stable task identifiers instead.
- Hidden global state: A helper function may draw from a global generator rather than the replicate's assigned stream. Pass or select the intended stream explicitly.
- Branch-dependent draw counts: Arm dropping and other conditional logic can change how many values are drawn. Use a stable assignment strategy so one branch does not unintentionally shift randomness assigned to another component.
- Incomplete records: A seed without the generator, stream rule, and code context may not be enough to recreate a run.
A simple single-stream run is a reasonable alternative for serial exploratory work when its draw order is fixed and the entire run is rerun from the beginning. It is less robust for independently restarting replicates or changing the parallel schedule. A stream per replicate improves task-level replay; component-level streams can further isolate changes within a replicate. These alternatives do not by themselves change the statistical estimand or improve the precision of a Monte Carlo estimate. They control how randomness is organized and how reliably a simulation can be rerun.
This topic is distinct from deliberately coupling simulations to compare candidate designs using common random numbers. That technique has its own goals and requires coordinated randomness across designs. For ordinary reproducibility, do not assume that the same seed for two scenarios creates a useful or valid coupling; use the intended stream plan and document it.
The main limitation is that reproducibility is conditional on the recorded computational setup. It does not guarantee that the model is scientifically appropriate, that the implementation correctly applies MAMS decision rules, or that Monte Carlo error is small. Those questions require separate checks. Reproducible randomness makes results auditable; it does not make them correct by itself.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why is a seed by itself generally insufficient to reproduce a simulation across software versions?
- A replicate is assigned to a worker using the worker's seed. What can go wrong when the number of workers changes, and what is a better assignment principle?
- In a two-arm MAMS simulation, why might dropping one arm at an interim change the later random outcomes for the other arm if both use one shared stream?
- What does a generator's stream-splitting facility establish, and what does it not establish?
Solutions. (1) Different generator algorithms or versions can map the same seed to different sequences, and the stream-allocation method and draw order also affect which values are used. Record those details, not just the seed. (2) The tasks assigned to a worker can change, so a replicate may receive a different sequence. Map streams to stable scenario and replicate identifiers, independently of worker assignment. (3) Dropping an arm can skip draws; if the other arm receives the next value from the same sequence, its assigned draws shift. Separate stable component or task streams can prevent that unintended shift. (4) It provides a documented method for creating distinct streams with properties intended for parallel use. It does not make a PRNG truly random, prove statistical independence in an absolute sense, or guarantee that the simulation code uses the streams correctly.