Variance Reduction With Common Random Numbers

Why Share Random Inputs?

When two trial designs are evaluated by simulation, the quantity of interest is often a contrast: for example, the difference in power, expected sample size, or probability of trial failure. Simulation error can obscure that difference, particularly when the candidate designs have similar operating characteristics. One way to improve precision is to simulate both designs using shared underlying random inputs, so that each pair of replicate outcomes reflects the same simulated sources of variation as far as the designs permit.

This approach is called common random numbers (CRN). The objective is not to make the two designs produce identical outcomes. It is to induce positive covariance between their replicate-level results while leaving the marginal simulation distribution for each design correct. If the covariance is positive, variation in the difference can be smaller than it would be under independent simulation. But sharing inputs does not guarantee positive covariance, so the variance of the comparison must be checked.

The Variance of a Simulated Contrast

Suppose the target is the difference between two design-specific means, \(\Delta = E(Y_A)-E(Y_B)\). In each of \(R\) replicates, simulate both outcomes and calculate \(D_r=Y_{Ar}-Y_{Br}\). The estimator is the average replicate-level difference, \(\bar{D}\). Assume the replicate pairs are independent and identically distributed across \(r\), and that both outcomes have finite variances. Within a replicate, the outcomes may be dependent because of their shared inputs.

The key identity follows by expanding the variance of a difference. It shows exactly how the dependence induced by CRN affects precision.

Because the \(R\) replicate differences are independent, \(\operatorname{Var}(\bar{D})=\operatorname{Var}(D_r)/R\). If both designs have the same marginal variances under CRN as they would under independent simulation, then their individual Monte Carlo variances are unchanged; the difference is that the CRN covariance term is no longer zero. Positive covariance reduces the variance of the difference, while negative covariance increases it.

In practice, the covariance is not known. Calculate the observed differences \(D_r\), their sample standard deviation \(s_D\), and estimate the Monte Carlo standard error (MCSE) of the contrast by \(s_D/\sqrt{R}\). This estimate uses the paired replicate differences, not the standard errors of the two design estimates considered separately.

Worked Example: Comparing Mean Outcomes

Common Random Numbers for Operating Characteristics

An operating characteristic such as power is a probability. For design A, let \(I_{Ar}=1\) if replicate \(r\) meets the specified success criterion and \(I_{Ar}=0\) otherwise; define \(I_{Br}\) in the same way. Then the simulated power estimates are the corresponding sample proportions, and their difference is the average of \(I_{Ar}-I_{Br}\). The same variance identity applies to these binary outcomes.

For one Bernoulli outcome with success probability \(p\), its variance is \(p(1-p)\). If the within-replicate probability that both designs succeed is \(q=P(I_A=1,I_B=1)\), then their covariance is \(q-p_Ap_B\). Consequently, the variance of the paired difference depends on which successes and failures occur together, not just on the two power estimates.

Constructing and Checking the Coupling

A useful CRN scheme shares random quantities that represent common sources of uncertainty in the two designs. For example, if both designs are evaluated under the same hypothetical distribution of patient characteristics, a replicate can begin with the same generated patient characteristics for both designs. Where a design requires random outcomes conditional on those characteristics, the simulation must still generate outcomes according to that design’s specified model.

Sharing the same numerical random draw everywhere is not, by itself, a justification. Designs can have different sample sizes, stages, or decision paths. The simulation must translate shared inputs into each design’s own required random variables while preserving each design’s marginal distribution. If the mapping changes a design’s intended distribution, the comparison is biased for the intended estimand, no matter how small its MCSE appears.

A practical assessment compares the paired-difference variance with the variance expected under independent replicates. The empirical variance of \(D_r\) is the direct measure of comparative precision. The sample correlation between the two replicate outcomes can help explain that variance, but correlation alone is not the goal: the variance also depends on the marginal variances and, for binary outcomes, on the success probabilities. Check the marginal operating-characteristic estimates against their expected simulation definitions and inspect whether unusual decision paths or boundary events dominate the differences.

When Pairing Fails to Help

CRN is not a variance-reduction guarantee. The relevant condition is positive covariance between the two outcomes for the contrast being estimated. Even a sensible shared-input design may produce little covariance if the outcomes respond differently to the common inputs. In an adaptive trial, for instance, candidate designs may follow different paths after an interim decision; their outcomes can then be weakly related despite sharing some initial simulated quantities.

As a diagnostic, examine the distribution of \(D_r\), not only its mean. A few large differences can inflate \(s_D\); clusters of replicates that take different adaptive paths can reveal why the coupling is weak. Compare the observed variance of paired differences with an independent benchmark based on the marginal variances, \(s_A^2+s_B^2\), where \(s_A\) and \(s_B\) are the sample standard deviations of the replicate outcomes. This comparison is informative when the estimates are based on the same number of replicates and the marginal outcomes have finite variances. It should be treated as a diagnostic, not as a proof that the chosen coupling is valid.

If the pairing is ineffective, independent simulation remains a valid way to estimate the contrast, using the variance of the two independent estimates. Increasing \(R\) can also reduce Monte Carlo error, although it does not repair an invalid marginal model. An analytic variance calculation may be useful where available, but it may not capture the full behavior of a complex adaptive simulation. CRN’s advantage is therefore conditional: it can improve comparative precision without changing the target, but only when its construction is valid and the resulting replicate differences are less variable.

Common Mistakes and Practical Limits

• Assuming shared inputs must reduce error. They do so only when they produce sufficiently positive covariance for the particular outcomes being contrasted. Estimate the variance of paired differences.
• Using separate standard errors for a paired contrast. Subtracting two estimates does not erase their covariance. Use the standard deviation of \(D_r\) divided by \(\sqrt{R}\).
• Confusing precision with correctness. A small MCSE says the simulation estimate is stable under repeated simulation from the implemented model. It does not establish that the model or code represents the intended trial.
• Assuming every random input should be identical. Shared inputs need a defensible interpretation and must preserve each candidate design’s marginal distribution. Inputs that cannot be aligned appropriately should not be forced into a common mapping.
• Claiming improved precision for each design separately. CRN primarily targets a comparison. The marginal Monte Carlo precision for each design is determined by its own replicate outcomes and number of replicates.

The main alternative for a design contrast is to simulate the designs independently and combine their estimated variances. That approach avoids dependence induced by the coupling and is straightforward, but it can be less precise when a valid CRN scheme produces positive covariance. Paired simulation with common inputs instead estimates the contrast directly through replicate differences. Neither approach excuses checking the marginal model, and neither removes uncertainty about whether the simulation assumptions reflect the trial question.