Why Pair Simulations Across Designs?
A simulation comparison asks how candidate designs differ in performance under a specified scenario. The difference can be obscured by Monte Carlo variation: one set of simulated participants may happen to be unusually difficult to estimate, while another set happens to be easier. Common random numbers address this problem by evaluating the candidate designs using shared simulated inputs, such as the same virtual participants, covariates, parameter values, and random effects. When those shared features affect both designs in similar ways, they can make the design comparison less noisy.
The key is not that the candidate designs must produce identical data. They may have different sampling times, numbers of observations, or analysis results. Instead, each paired replicate starts from the same underlying simulated situation, while each design is implemented according to its own specification. The comparison then asks how the designs perform for that common situation.
Let \(Y_{A,r}\) and \(Y_{B,r}\) denote a performance outcome for designs A and B in replicate \(r\). The outcome might be an estimation error, a measure of precision, or an indicator of meeting a prespecified criterion. The target difference is \(E(Y_A)-E(Y_B)\), and its simulation estimate from \(R\) paired replicates is the mean of the replicate-level differences.
The pairing matters because the two outcomes within a replicate are generally correlated. If both designs tend to perform poorly on the same challenging simulated participants, then their difference may vary less than either outcome on its own.
Variance Reduction: The Justification
Assume the paired replicates are independent and identically generated across \(r\), each design’s outcomes follow its intended simulation distribution, and the outcome has finite variance. Within a replicate, write \(\Delta_r=Y_{A,r}-Y_{B,r}\). The variance of a difference follows from the variance identity for two random variables:
Because the \(R\) paired differences are independent replicates, the variance of their average is the variance of one paired difference divided by \(R\). Thus,
This equation explains both the potential benefit and its limitation. Positive covariance reduces the variance of the difference; zero covariance gives no variance reduction relative to independent simulation with the same marginal variances; negative covariance increases the variance. CRN is therefore a coupling strategy with a possible benefit, not a guarantee of improvement.
For comparison, if designs are simulated independently, their covariance is zero, so the variance of the difference of their sample means is \(\{\operatorname{Var}(Y_A)+\operatorname{Var}(Y_B)\}/R\). The paired strategy has lower variance precisely when the induced covariance is positive. This comparison assumes the same number of replicates and unchanged marginal distributions.
Worked Example: Comparing Paired and Independent Simulation Noise
In a hypothetical simulation, two designs are evaluated on five shared sets of virtual participants. Their replicate-level precision scores are shown below; higher scores represent better precision. For this illustration, the target is the difference in mean scores, A minus B.
| Replicate | Design A score | Design B score | A minus B |
|---|---|---|---|
| 1 | 8.0 | 7.5 | 0.5 |
| 2 | 11.0 | 10.0 | 1.0 |
| 3 | 9.0 | 8.5 | 0.5 |
| 4 | 12.0 | 10.5 | 1.5 |
| 5 | 10.0 | 8.5 | 1.5 |
The mean score for A is \((8+11+9+12+10)/5=10.0\), and for B it is \((7.5+10+8.5+10.5+8.5)/5=9.0\). The estimated difference is therefore \(10.0-9.0=1.0\) score unit. Equivalently, the paired differences average to \((0.5+1.0+0.5+1.5+1.5)/5=1.0\).
To see the effect on Monte Carlo variation, the sample variance of the paired differences is \[ s_{\Delta}^{2} = \frac{(-0.5)^2+0^2+(-0.5)^2+0.5^2+0.5^2}{5-1} = 0.25. \] The standard error of the paired mean difference, using the replicate differences, is \(\sqrt{0.25/5}=0.2236\) score units.
For comparison, the sample variance of A’s scores is \(2.50\), and that of B’s scores is \(1.50\). If the two sets of five outcomes had been generated independently, the estimated standard error of their mean difference would be \[ \sqrt{\frac{2.50}{5}+\frac{1.50}{5}} = \sqrt{0.80} = 0.8944 \] score units. In this hypothetical example, the shared participants make the paired outcomes move together, so the differences are much less variable. The result supports an estimated advantage of 1.0 score unit for A on this performance measure; the smaller paired standard error describes Monte Carlo precision, not a change in the designs’ underlying performance.
How to Construct a Meaningful Pair
A simulation replicate can be viewed as a set of underlying random inputs passed through the design-specific data-generation and analysis procedures. For example, let \(U_r\) represent the scenario’s parameter values, participant covariates, random effects, and any other inputs that can appropriately be shared. Each design maps those inputs to its own outcome:
The functions \(h_A\) and \(h_B\) differ because the candidate designs differ. To preserve valid comparisons, use the same input \(U_r\) only where it represents the same underlying feature in both designs. Design-specific features—such as the observation schedule—must still be generated and applied as specified for that design.
Choose a performance outcome and define the direction of the difference, such as A minus B. The meaning and units of the outcome should be clear before simulation.
Decide which elements represent the same simulated situation across designs, for example a participant’s covariates, random effects, or a parameter draw from the scenario.
Generate observations and run the analysis pipeline required by each candidate design. Sharing participants does not mean forcing the designs to have the same observations or results.
Store both design outcomes for each shared replicate and calculate their difference within that replicate. Do not discard the pairing when summarizing the comparison.
In pharmacometric design simulations, sharing a participant’s covariates and random effects can be especially useful when those features strongly affect estimation difficulty. For schedules with different observation times, however, residual errors require care. A shared participant can have a coherent underlying concentration-time profile, but the residual error at one design’s observation time is not automatically the same quantity as the residual error at a different time. Generate each design’s observations under its specified error model. If error processes are coupled across designs, the coupling must preserve the correct error distribution and dependence structure for each design.
Worked Example: Paired Estimation Errors for Shared Participants
Consider a hypothetical comparison of two sampling designs for estimating a model parameter whose true value is 1.00. In five simulation replicates, each design is applied to the same simulated participants and random effects within the replicate. The resulting estimates are:
| Replicate | Estimate A | Estimate B | Absolute error A | Absolute error B | A error minus B error |
|---|---|---|---|---|---|
| 1 | 0.82 | 0.90 | 0.18 | 0.10 | 0.08 |
| 2 | 1.08 | 1.02 | 0.08 | 0.02 | 0.06 |
| 3 | 0.95 | 1.01 | 0.05 | 0.01 | 0.04 |
| 4 | 1.20 | 1.10 | 0.20 | 0.10 | 0.10 |
| 5 | 0.90 | 0.96 | 0.10 | 0.04 | 0.06 |
The mean absolute error for A is \((0.18+0.08+0.05+0.20+0.10)/5=0.122\) parameter units. For B it is \((0.10+0.02+0.01+0.10+0.04)/5=0.054\) parameter units. The estimated difference, A minus B, is \(0.122-0.054=0.068\) parameter units. Calculating the paired differences directly gives the same result: \((0.08+0.06+0.04+0.10+0.06)/5=0.068\).
Because lower absolute error is preferable, the positive difference indicates that design B had lower mean absolute error in these five hypothetical paired replicates. This is an estimate from a finite simulation, not proof that B is better in every scenario or for every simulated participant. The shared replicate construction makes the comparison focus on the designs’ differing errors under matched simulation conditions.
Checking Whether the Coupling Helps
A practical diagnostic is to inspect the paired outcomes and the paired differences, not just the separate summaries for each design. If unusually high or low outcomes tend to occur together, the covariance is positive and pairing is likely to reduce variation in the difference. If there is little co-movement, the variance reduction may be negligible. If high outcomes for one design tend to coincide with low outcomes for the other, covariance can be negative and CRN can make the comparison noisier.
Worked Example: When Shared Inputs Help—or Hurt
Suppose the replicate-level outcome variances are 4 for design A and 9 for design B. With independent simulations, the variance of the difference from one replicate is \(4+9=13\). If shared inputs induce correlation \(0.60\), then the covariance is \(0.60\sqrt{4\times9}=0.60(6)=3.6\). The paired variance becomes \[ 4+9-2(3.6)=5.8. \] For \(R=100\) replicates, the variance of the estimated mean difference is \(5.8/100=0.058\), compared with \(13/100=0.13\) for independent simulations. The reduction is \(1-5.8/13=0.554\), or about 55.4% in variance.
Now suppose a different coupling induces correlation \(-0.30\). Its covariance is \(-0.30(6)=-1.8\), so the paired variance is \(4+9-2(-1.8)=16.6\), which is greater than 13. For 100 replicates, the mean-difference variance is \(0.166\), also greater than the independent value \(0.13\). Thus, sharing random inputs is useful only when the resulting dependence is favorable for the comparison.
Alternatives, Limitations, and Common Mistakes
The main alternative is independent simulation: generate a separate set of random inputs for each design and compare their estimated mean performance. This is straightforward and does not require constructing a meaningful coupling, but it forgoes any covariance benefit. Another option is partial sharing: pair the inputs that naturally describe the same underlying participants or scenario while generating design-specific features separately. Partial sharing can be more defensible than forcing a match between quantities that do not correspond.
CRN does not correct a misspecified data-generating model, an inappropriate performance estimand, or errors in the analysis pipeline. It also does not make the simulated participants more representative, and it cannot compensate for too few replicates in every setting. Its benefit depends on the outcome being compared and the dependence created by the chosen coupling; a coupling that helps one performance measure may not help another.
A common mistake is to assume that using identical random inputs must improve precision. That reasoning ignores the covariance term: negative covariance raises the variance of the paired difference. The correction is to assess how the chosen inputs affect both outcomes and to examine the resulting paired differences. Another mistake is to reuse a residual error draw as though it represented the same observation in two designs with different observation times. The correction is to share only genuinely corresponding latent inputs and to generate design-specific observations according to each design’s model.
Check Your Understanding
Answer each question using the definitions and variance relationship above. Concise calculations are sufficient.
- What must remain unchanged for a CRN comparison to preserve the validity of each candidate design’s simulated performance?
- If the covariance between paired outcomes is positive, what happens to the variance of the replicate-level difference compared with the same marginal variances under independent simulation?
- For paired outcomes with variances 16 and 25 and covariance 5, calculate the variance of their difference.
- A proposed CRN setup reuses the exact same residual error for observations at different times in two sampling schedules. What is the concern, and what should be checked?
- Can CRN increase Monte Carlo noise? State the condition in terms of covariance.
Solutions. (1) Each design must retain its intended marginal distribution, including its specified data-generation and analysis procedures. (2) Positive covariance subtracts \(2\operatorname{Cov}(Y_A,Y_B)\) from the sum of the marginal variances, so the difference variance decreases. (3) \(\operatorname{Var}(Y_A-Y_B)=16+25-2(5)=31\) outcome-units squared. (4) The errors may represent different observation times and therefore may not be corresponding quantities; check that any coupling preserves the specified residual distributions and dependence structure, and generate design-specific observations as required. (5) Yes. Negative covariance makes \(-2\operatorname{Cov}(Y_A,Y_B)\) positive, increasing the variance above the independent-simulation value.