Why Pair Simulation Replicates?
When several candidate multi-arm multi-stage (MAMS) designs are under consideration, a planning question may be whether one design has lower expected enrollment than another. Expected enrollment is an average over the possible trial outcomes and stopping decisions under a specified scenario. Simulation estimates that average, but the estimate has Monte Carlo error: another set of random replicates would generally produce a slightly different result.
The earlier material on Monte Carlo precision for expected sample size establishes how to quantify simulation error for one design. For a comparison, the key choice is whether to simulate the designs independently or to make each replicate for one design use the same underlying random inputs as the corresponding replicate for the other. This pairing, often called using common random numbers, can make the enrollment outcomes move together. If that dependence is positive, it can reduce the Monte Carlo error of their difference.
The pairing does not change the expected enrollment of either design, provided each design still receives random inputs with its correct distribution. It changes the joint distribution of the two simulated outcomes, and therefore the precision of their estimated difference. Pairing is not automatically beneficial: the covariance can be small or negative, in which case it may do little or even increase the error.
Paired Estimation and Its Assumptions
The target difference is \(\Delta=E(N_A)-E(N_B)\), with expectations taken under the specified planning scenario. The paired estimator is unbiased for this difference when each design’s replicate outputs have the correct marginal distribution: \(E(\widehat{\Delta})=E(N_A)-E(N_B)\). The shared inputs may induce dependence within each pair, but pairs must be independent and identically distributed across replicates, with finite variance, for the usual sample-based standard error to apply.
A valid comparison therefore requires more than using the same seed or matching rows in two output files. The simulation must preserve each design’s own model, stage sizes, decision rules, and enrollment accounting. Shared random inputs should represent corresponding sources of uncertainty, such as outcomes for the same simulated participant or common nuisance parameters, while each design applies its own analysis and stopping rules. When designs use different numbers of participants or stages, the coupling must still yield the correct inputs for each design.
Why Pairing Can Reduce Monte Carlo Error
For one paired replicate, the variance of the difference is determined by the two marginal variances and their covariance. Across \(R\) independent, identically distributed pairs (replicates generated by the same scheme, with finite variance), the variance of the paired estimator is the variance of a single paired difference divided by \(R\).
If the two designs were simulated independently using \(R\) replicates each, the estimated difference would instead have variance \(\{\operatorname{Var}(N_A)+\operatorname{Var}(N_B)\}/R\). Thus, for the same number of simulations per design, pairing reduces variance relative to independent simulation exactly when \(\operatorname{Cov}(N_A,N_B)>0\). It leaves variance unchanged when covariance is zero and increases variance when covariance is negative.
This condition concerns the covariance of enrollment outcomes under the chosen coupling, not whether the designs are similar in name or structure. For example, a replicate with generally favorable outcomes might tend to lead to high enrollment in both designs, yielding positive covariance. But different stopping rules can make the outcomes respond in opposite ways to shared inputs. A short pilot simulation can estimate whether the proposed coupling is helping.
Worked Examples
Worked Example: Correcting for Replicate Pairing
Consider a hypothetical comparison in which each design enrolls 40 participants initially and enrolls 40 more if its interim rule says to continue. Five paired replicates use common underlying simulation inputs. The scenario-specific interim statistics can differ by design; A continues when its statistic is below 0.5, while B continues when its statistic is below 0. For B, a statistic of 0.2 does not meet the continuation rule.
Suppose the simulated statistics and resulting enrollment are:
| Replicate | Statistic for A | Statistic for B | Enrollment A | Enrollment B | Difference A − B |
|---|---|---|---|---|---|
| 1 | −0.4 | −0.4 | 80 | 80 | 0 |
| 2 | 0.2 | 0.2 | 80 | 40 | 40 |
| 3 | 0.8 | 0.8 | 40 | 40 | 0 |
| 4 | 1.1 | 1.1 | 40 | 40 | 0 |
| 5 | −0.2 | 0.1 | 80 | 40 | 40 |
The design-specific means are \(\bar{N}_A=(80+80+40+40+80)/5=64\) participants and \(\bar{N}_B=(80+40+40+40+40)/5=48\) participants. The estimated difference is \(64-48=16\) participants. The paired differences are \(0,40,0,0,40\), with mean 16. Their sample variance is
Therefore the paired Monte Carlo standard error is \(SE(\widehat{\Delta})=\sqrt{480/5}=\sqrt{96}\approx9.798\) participants. This is the estimated simulation error for the difference, not a measure of uncertainty in the assumed clinical scenario.
For comparison, the sample variance of A’s five enrollments is \(480\) participants\(^2\), while B’s is \(320\) participants\(^2\). Treating the two sets as independent would give \(SE=\sqrt{480/5+320/5}=\sqrt{160}\approx12.65\) participants. Here the pairing lowers the estimated standard error. Notice that correctly applying B’s \(Z<0\) rule matters: the \(0.2\) replicate does not continue B, and the resulting paired differences must be recalculated accordingly.
Worked Example: A Coupling That Makes Precision Worse
Suppose a hypothetical four-replicate comparison gives paired enrollments \(N_A=(40,40,80,80)\) and \(N_B=(80,80,40,40)\). Both sample means are 60 participants, so the estimated difference is zero. Each sample variance is \(533.3\) participants\(^2\): for A, for instance, the squared deviations from 60 sum to \(1600\), and \(1600/(4-1)=533.3\).
The paired differences are \((-40,-40,40,40)\). Their mean is zero and their sample variance is \[ s_D^2=\frac{4(40^2)}{3}=2133.3\ \text{participants}^2, \] so the paired standard error is \(\sqrt{2133.3/4}\approx23.09\) participants. The sample covariance is \(-533.3\) participants\(^2\): the deviations for A and B have opposite signs on every replicate. The variance identity gives \(1600/3+1600/3-2(-1600/3)=6400/3\approx2133.3\), agreeing with the direct calculation.
Independent simulation with four replicates per design would have estimated standard error \(\sqrt{533.3/4+533.3/4}\approx16.33\) participants. This hypothetical coupling is counterproductive because high enrollment for one design coincides with low enrollment for the other. The example shows why “common random numbers” is a strategy to assess, not a guarantee of improved precision.
Worked Example: Using a Pilot to Quantify the Gain
A hypothetical pilot with \(R=400\) paired replicates estimates means of 126 and 119 participants for designs A and B. Across replicates, the sample standard deviations are \(s_A=18\) and \(s_B=15\) participants, and the sample covariance is \(s_{AB}=180\) participants\(^2\). The estimated difference is \(126-119=7\) participants. The paired-difference variance is estimated by
Thus \(SE(\widehat{\Delta})=\sqrt{189/400}\approx0.6874\) participants. With independent simulation at the same replicate count, the estimated standard error would be \(\sqrt{(18^2+15^2)/400}=\sqrt{549/400}\approx1.172\) participants. For these pilot results, pairing produces a smaller estimated Monte Carlo standard error because the sample covariance is positive. The pilot estimates suggest a difference of 7 participants, with about 0.69 participants of Monte Carlo standard error; they do not establish that the designs differ by precisely 7 participants in practice.
Practical Checks and Failure Modes
A paired analysis is only as credible as the pairing and the replicate-level outputs. Check that the two designs’ marginal simulation mechanisms are correct before interpreting a variance reduction. Then inspect the paired differences themselves, not just the two separate enrollment averages. A large positive covariance is useful only if it arises from a defensible shared-input scheme and is stable enough to estimate.
State the scenario, the candidate designs, the enrollment definition, and the direction of the contrast. For this tutorial, \(\widehat{\Delta}\) is A minus B, so a positive estimate means higher estimated expected enrollment for A.
Use shared random inputs only for corresponding uncertainty sources, while applying each design’s own stage structure, decision rules, and enrollment calculation.
For each replicate, calculate \(D_r=N_{A,r}-N_{B,r}\). Compute \(s_D\) across replicates and report \(s_D/\sqrt{R}\), along with the estimated difference in participants.
Compare the paired standard error with the independent-simulation benchmark based on the two marginal variances. Investigate a small or negative covariance rather than assuming that pairing improved precision.
Several problems can undermine the comparison. Misapplying a continuation boundary, counting shared-control participants twice, or using a different enrollment definition across designs changes the quantity being simulated; pairing does not repair such errors. Pair indices can also be misleading if the supposed shared inputs are not actually aligned. For instance, a simulation that consumes a variable number of random draws after an early stopping decision can cause later draws in the two runs to represent unrelated events. Random streams indexed by replicate and by the relevant source of variation can help maintain intentional alignment.
Monte Carlo standard errors are estimates, particularly when replicate counts are small or enrollment distributions are highly skewed. Inspect the distribution of \(D_r\), look for unusual values or coding-related spikes, and assess whether the estimated standard error stabilizes as the number of pairs grows. If decisions depend on a very small estimated difference, more replicates may be needed even when pairing reduces error. Simulation precision also does not cover uncertainty about assumed treatment effects, recruitment, attrition, or other scenario inputs.
How Pairing Compares With Alternatives
| Approach | How the difference is estimated | Precision implication | Main caution |
|---|---|---|---|
| Independent simulation | Subtract two means from separate replicate sets | Variance is the sum of the two mean variances | May use simulation effort inefficiently when outcomes could be positively coupled |
| Common random numbers with paired analysis | Average replicate-level differences | Includes the covariance term; improves precision when covariance is positive | Can be less precise if covariance is negative, or misleading if pairing is invalid |
| More replicates | Increase the number of simulations for either approach | Usually reduces Monte Carlo standard error at a rate proportional to \(1/\sqrt{R}\) | Costs computation and does not correct a flawed model or coupling |
Independent simulation remains a reasonable choice when constructing meaningful pairs is difficult or when pilot results indicate that pairing does not help. Increasing \(R\) is another way to reduce Monte Carlo error, but pairing can be computationally attractive when it produces a clear positive covariance. The two strategies are not mutually exclusive: a well-designed paired simulation can also use enough replicates to meet a chosen precision target.
The central limitation is that variance reduction is specific to the scenario and coupling. A shared-input design that works well for one effect configuration or pair of candidate designs may not work as well for another. Each comparison should retain its own replicate-level differences and uncertainty estimate. A smaller Monte Carlo standard error means the simulated difference is estimated more precisely under the chosen model; it does not mean the model’s assumptions are more realistic.
Check Your Understanding
Answer the questions, then compare your work with the solutions.
- A paired simulation has \(R=36\) replicates and a sample standard deviation of paired enrollment differences of 12 participants. What is the paired Monte Carlo standard error?
- State the condition under which common random numbers reduce the variance of an estimated difference compared with independent simulations using the same number of replicates per design.
- In a pilot, \(s_A=20\), \(s_B=16\), the sample covariance is \(160\) participants\(^2\), and \(R=100\). Find the paired standard error and the independent-simulation benchmark.
- Why can common random numbers increase the Monte Carlo standard error instead of reducing it?
Solutions. (1) \(SE=12/\sqrt{36}=2\) participants. (2) The paired covariance must be positive: \(\operatorname{Cov}(N_A,N_B)>0\). (3) The paired-difference variance estimate is \(20^2+16^2-2(160)=336\) participants\(^2\), so the paired standard error is \(\sqrt{336/100}\approx1.833\) participants. The independent benchmark is \(\sqrt{(20^2+16^2)/100}=\sqrt{6.56}\approx2.561\) participants. (4) Negative covariance makes the term \(-2\operatorname{Cov}(N_A,N_B)\) positive, increasing the variance of the paired differences relative to independent simulation.