Why Compare Designs in Pairs?
Simulation studies often compare candidate designs by estimating the same operating characteristic for each—for example, power, Type I error, average sample size, or probability of trial failure. If both designs are evaluated under the same data-generating scenario, a direct comparison is usually more useful than two isolated summaries. The central quantity is the replicate-level difference: the value for one design minus the value for the other in a matched replicate.
A match means more than giving two unrelated simulation runs the same row number after they have finished. The replicate outputs must correspond under a prespecified scheme: each pair represents the two designs’ results under the same scenario and a meaningful matched simulation condition. This pairing can make the estimated contrast more precise when the two outputs tend to move together. It does not guarantee greater precision; the empirical variability of the paired differences determines that.
The order of subtraction must be declared and used consistently. Here, positive \(\bar{D}\) means the output is larger for design A. Whether that is favorable depends on the measure: greater power may be desirable, whereas greater failure probability or average trial duration may not be.
Monte Carlo Uncertainty for a Paired Contrast
Suppose the matched differences \(D_1,\ldots,D_R\) are independent and identically distributed draws from the replicate-level contrast distribution under a fixed scenario. Assume that this distribution has a finite variance. The sample variance of the differences is
The Monte Carlo standard error (MCSE) of \(\bar{D}\), which measures simulation error in estimating the contrast, is estimated by
This is uncertainty from using a finite number of simulation replicates. It is not the standard error of an effect estimate within a single simulated trial, nor does it represent uncertainty about whether the chosen data-generating scenario is realistic.
Why Matching Can Improve Precision
For a single matched replicate, the variance identity is
This follows by expanding the variance of a difference: \(\operatorname{Var}(Y_A-Y_B)=\operatorname{Var}(Y_A)+\operatorname{Var}(-Y_B)+2\operatorname{Cov}(Y_A,-Y_B)\), and using \(\operatorname{Var}(-Y_B)=\operatorname{Var}(Y_B)\) and \(\operatorname{Cov}(Y_A,-Y_B)=-\operatorname{Cov}(Y_A,Y_B)\). When matched outputs have positive covariance, the variance of their difference is smaller than it would be if that covariance were zero. The benefit is not automatic: negative covariance can increase the variance of the difference.
For comparison, if the designs are simulated in separate independent sets of \(R\) replicates, the estimated difference of means has estimated variance \(s_A^2/R+s_B^2/R\), where \(s_A^2\) and \(s_B^2\) are the two sets’ sample variances. In a paired analysis, the estimated variance is \(s_D^2/R\). These formulas answer different questions about the simulation output: use the paired formula only when the replicate correspondence is genuine and the differences are the analysis units.
Worked Example: Comparing Type I Error Under a Null Scenario
In a hypothetical simulation under a null scenario, two candidate designs use the same rejection rule. Each of 10 matched replicates records whether the trial rejects the null hypothesis. Across the pairs, both designs reject in 2 replicates, only A rejects in 2, only B rejects in 1, and neither rejects in 5. Calculate the estimated difference in Type I error rates, A minus B, and its MCSE.
The outcome for each design in a replicate is a binary indicator. Therefore, the paired difference can only be \(-1\), \(0\), or \(1\). The stated pair counts give two differences of \(1\), one difference of \(-1\), and seven differences of \(0\). The estimated rejection rates are \(4/10=0.40\) for A and \(3/10=0.30\) for B, so the estimated difference is \(0.10\), or 10 percentage points. Because the scenario is null, this is a difference in estimated Type I error rates—not a difference in power.
The mean paired difference is \(\bar{D}=0.10\). Its sum of squared deviations is \(2(1-0.10)^2+1(-1-0.10)^2+7(0-0.10)^2=2.90\). Thus,
The estimated contrast is 10 percentage points, with MCSE about 17.95 percentage points. The large MCSE relative to the contrast shows that 10 replicates do not provide a precise estimate. This is an illustration of the calculation, not a reliable design comparison. The paired binary outcomes are internally consistent: no binary-indicator difference such as 0.02 is possible in a single replicate.
Continuous Outputs and Replicate-Level Differences
For a continuous operating characteristic, \(Y_{Ar}\) and \(Y_{Br}\) might be simulated estimates of a success probability, sample size, or duration. These are replicate-level quantities, not necessarily individual patient outcomes. Their paired difference can take many values, so small decimal contrasts can be entirely appropriate. Always identify what one replicate-level value represents before interpreting its difference.
Worked Example: Comparing a Continuous Operating Characteristic
Suppose six hypothetical matched replicates produce estimated success probabilities for A of 0.70, 0.74, 0.68, 0.73, 0.69, and 0.72, and for B of 0.68, 0.70, 0.68, 0.70, 0.70, and 0.70. Compute the average paired contrast A minus B and its MCSE.
The differences, expressed in percentage points, are \(2,4,0,3,-1,2\). Their mean is \(\bar{D}=10/6=1.6667\) percentage points. The sum of squared differences is \(2^2+4^2+0^2+3^2+(-1)^2+2^2=34\). The sum of squared deviations from the mean is \(34-6(1.6667^2)\approx17.3333\), so
The estimated success-probability contrast is 1.6667 percentage points in favor of A, with MCSE about 0.7601 percentage points. The sample size of six is small, so the estimated MCSE itself may be unstable. A precise-looking decimal result does not remove that limitation.
For comparison, the sample variances of the A and B outputs are 5.600 and 1.0667 in squared percentage points. Treating the design outputs as independent would give an estimated standard error of \(\sqrt{5.600/6+1.0667/6}\approx1.054\) percentage points. The paired MCSE is smaller here because the paired outputs vary together. This comparison is informative only because the matched pairs were defined as part of the simulation plan.
Comparisons Across Multiple Scenarios
Design comparisons often cover several data-generating scenarios rather than one. First estimate a paired contrast within each scenario. Then combine those estimates using weights that reflect the stated target of the comparison. If scenario \(j\) has prespecified weight \(w_j\), with \(\sum_j w_j=1\), the target contrast estimate is \(\sum_j w_j\bar{D}_j\). Equal numbers of replicates per scenario do not imply equal scenario weights. The weighting rule defines the summary being estimated.
If replicates are independent across scenarios, and scenario \(j\) has \(R_j\) pairs and sample variance \(s_{Dj}^2\), the estimated variance of the weighted contrast is \(\sum_j w_j^2s_{Dj}^2/R_j\). This formula accounts for the separate Monte Carlo uncertainty within each scenario. It does not account for uncertainty in the scenario weights themselves; those are treated as fixed.
Worked Example: A Prespecified Weighted Contrast
Consider two hypothetical scenarios. Scenario 1 has weight 0.60 and four paired differences, in percentage points, of \(1,2,0,1\). Scenario 2 has weight 0.40 and four paired differences of \(-1,-2,-3,-2\). Find the weighted contrast and its MCSE, assuming independent replicates across scenarios.
The scenario means are \(\bar{D}_1=1.00\) and \(\bar{D}_2=-2.00\) percentage points. The weighted estimate is \(0.60(1.00)+0.40(-2.00)=-0.20\) percentage points. In each scenario, the squared deviations sum to 2, so each sample variance is \(2/(4-1)=0.6667\) squared percentage points. Therefore,
Under these scenario weights, the estimated contrast is \(-0.20\) percentage points, with MCSE about 0.2944 percentage points. The negative sign means the weighted average output for A is lower than for B; whether that is desirable depends on the output. An unweighted average of the two scenario means would be \(-0.50\) percentage points and would answer a different question.
Diagnostics, Alternatives, and Limitations
Before interpreting a paired estimate, check the simulation record and the difference distribution. Confirm that every pair uses the intended scenario, that both design outputs were produced under the planned matching rule, and that the subtraction order is unchanged. Inspect a table or plot of the \(D_r\) values for outliers, skewness, unexpectedly many identical values, or dependence across replicate indices. For binary outcomes, verify that every difference is \(-1\), \(0\), or \(1\). For scenario-weighted results, check the weights and compute contrasts within scenarios before combining them.
The usual MCSE formula relies on independent replicate differences with a finite variance. If replicates are dependent, \(s_D/\sqrt{R}\) may understate or overstate uncertainty; the dependence structure must be reflected in the uncertainty calculation. If the differences are extremely skewed or rare events dominate them, a small \(R\) can also make a variance-based MCSE unstable. The remedy is not to report more digits: use an adequate replication plan, examine stability as replication increases, and report the limitations.
An alternative is to run independent simulation sets for the two designs and compare their sample means using the independent-samples variance formula. This avoids requiring a meaningful pair-level correspondence, but it can be less precise when a legitimate pairing would produce positive covariance. Conversely, a nominal pairing that does not reflect the simulation plan is not a valid reason to use the paired formula. Do not choose between paired and unpaired calculations after inspecting which gives the preferred conclusion.
A paired simulation comparison addresses Monte Carlo precision for a contrast under specified conditions. It does not establish that the scenarios cover all clinically relevant possibilities, that the candidate designs are otherwise acceptable, or that a simulated result will hold in practice. Those are separate questions of scenario choice, design evaluation, and scientific judgment.
Name the operating characteristic, the subtraction order, and the scenario or scenario weights.
Prespecify how each pair corresponds and preserve the scenario and replicate identifiers in the simulation records.
Compute \(D_r\), \(\bar{D}\), and \(s_D/\sqrt{R}\); for multiple scenarios, combine scenario estimates and variances using the stated weights.
Inspect differences, confirm units and signs, and distinguish Monte Carlo error from uncertainty about the design or assumptions.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In a null scenario, design A rejects in 12 of 100 replicates and design B in 10 of 100. What kind of difference is being estimated?
- What values can a replicate-level difference take when both design outputs are binary indicators?
- Why can the MCSE from paired differences be smaller than the independent-samples MCSE?
- Two scenarios have weights 0.25 and 0.75 and estimated contrasts of 4 and 0 percentage points. What is the weighted contrast?
- Name one condition under which \(s_D/\sqrt{R}\) may not be a valid MCSE.
Solutions. (1) It is a difference in estimated Type I error rates, because the scenario is null; the observed rates are 12% and 10%, so A minus B is 2 percentage points. (2) Only \(-1\), \(0\), or \(1\). (3) Positive covariance between matched outputs reduces the variance of their difference through the term \(-2\operatorname{Cov}(Y_A,Y_B)\). (4) \(0.25(4)+0.75(0)=1\) percentage point. (5) For example, replicate differences are dependent; then the independent-replicate variance formula does not generally apply.