Why Compare Estimators on the Same Simulated Data?
A simulation study can compare two procedures by applying each to repeated datasets generated under the same hypothetical model. The comparison is clearest when, in each replicate, both procedures receive the same dataset. This creates a pair of results: one from each estimator under the same simulated conditions. Differences within these pairs can remove some of the variation that would otherwise obscure the comparison.
Suppose the quantity of interest is \(\theta\), and two estimators are \(\hat{\theta}_A\) and \(\hat{\theta}_B\). The study may compare their bias, variance, mean squared error, or another performance measure. The paired approach is not a new definition of those measures. It is a design for estimating the difference between their expected performance while accounting for the fact that both results come from the same simulated dataset.
For example, if \(g\) is squared error, a positive difference means A has greater squared error on average, so B has lower mean squared error. If the measure is an estimate itself, the sign instead describes which procedure tends to produce the larger estimate. Always state the measure and the subtraction order before interpreting the sign.
Estimating the Difference and Its Monte Carlo Error
Assume the replicate datasets are generated independently from the same specified data-generating model, and both procedures are applied in every replicate. The paired differences \(D_1,\ldots,D_R\) are then independent and identically distributed, provided the simulation procedure is kept the same across replicates. Their sample mean estimates the expected performance difference.
Here \(s_D^2\) estimates the variance of one replicate-level difference, while \(\widehat{\operatorname{SE}}(\bar D)\) estimates the Monte Carlo standard error of the reported average difference. It describes simulation error in estimating the expected difference; it is not the standard deviation of the underlying estimator across individual datasets, nor does it measure uncertainty about whether the chosen data-generating model is appropriate.
A compact justification follows from independence across replicates. The variance of a sum of independent random variables is the sum of their variances, so \(\operatorname{Var}(\bar D)=\operatorname{Var}(D_r)/R\). Replacing the unknown variance of a replicate difference with its sample estimate \(s_D^2\) gives the stated estimated standard error. The standard deviation of \(D_r\) is estimated from deviations around \(\bar D\), using \(R-1\) in the denominator.
Why Pairing Can Improve Precision
Write \(A_r\) and \(B_r\) for the performance values from the two procedures in replicate \(r\), so \(D_r=A_r-B_r\). The variance identity for a difference is \(\operatorname{Var}(A_r-B_r)=\operatorname{Var}(A_r)+\operatorname{Var}(B_r)-2\operatorname{Cov}(A_r,B_r)\). When both procedures respond similarly to unusually favorable or unfavorable simulated datasets, their covariance is positive. Subtracting their paired results then cancels some of this shared variation.
This identity explains the potential gain, not a guarantee. If the covariance is zero, pairing gives no variance reduction relative to the independent-results variance calculation with the same marginal variances. If it is negative, the paired differences can be more variable. The actual paired differences should be inspected and analyzed; pairing should not be assumed to help merely because it was used.
Choose a common performance measure and a clear subtraction order, such as squared error from A minus squared error from B.
Use the same simulated dataset as input to both procedures in that replicate.
Subtract the two performance values within each replicate before averaging across replicates.
Compute \(\bar D\), \(s_D\), and \(s_D/\sqrt{R}\), then interpret the sign in the chosen performance measure's context.
Worked Example: Comparing Two Estimators of a Portfolio Quantity
A hypothetical simulation uses annual claim counts \(N\sim\operatorname{Poisson}(20)\) and independent claim severities \(X\sim\operatorname{Exponential}(\text{mean }500\text{ dollars})\). In each of five independent simulated datasets, two procedures estimate the same portfolio quantity, in dollars. Their results are listed below. Find the average paired difference \(A-B\), its estimated Monte Carlo standard error, and compare that error with the calculation that treats the two columns as independent.
| Replicate | \(A_r\) | \(B_r\) | \(D_r=A_r-B_r\) |
|---|---|---|---|
| 1 | 96 | 100 | -4 |
| 2 | 104 | 100 | 4 |
| 3 | 99 | 101 | -2 |
| 4 | 107 | 101 | 6 |
| 5 | 94 | 98 | -4 |
Method. Use paired replicate differences, because both procedures were applied to the same simulated dataset in each row. The differences sum to \(-4+4-2+6-4=0\), so \(\bar D=0/5=0\) dollars.
The squared deviations from zero are \(16,16,4,36,16\), with sum \(88\). Thus \(s_D^2=88/(5-1)=22\) dollars squared, and \(\widehat{\operatorname{SE}}(\bar D)=\sqrt{22/5}=\sqrt{4.4}\approx2.098\) dollars. The paired results show no average difference in this small simulation, but the standard error indicates nonzero Monte Carlo variation.
For comparison, the means of the two columns are both \(100\). Their sample variances are \(s_A^2=118/4=29.5\) and \(s_B^2=6/4=1.5\). Treating the columns as independent would give an estimated standard error of \(\sqrt{(29.5+1.5)/5}=\sqrt{6.2}\approx2.490\) dollars. The paired covariance is \(s_{AB}=18/4=4.5\), so the paired variance is \(29.5+1.5-2(4.5)=22\), agreeing with the direct calculation from \(D_r\). In this example the positive covariance reduces the estimated standard error.
Common mistake avoided: using the independent-columns formula for results that were paired. It omits the covariance term. The independent calculation is appropriate for two genuinely independent sets of replicates, but it is not the standard error calculation for the paired differences in this design.
Pairing Performance Measures, Not Just Estimates
The paired quantity need not be the raw estimator value. If the objective is to compare mean squared error, calculate squared error separately for each procedure within each replicate, then subtract those values. This respects the comparison target: the expected difference in squared error. Averaging the estimates first and then squaring their difference would answer a different question.
Worked Example: Comparing Mean Squared Error
In a hypothetical severity study, independent samples are generated from an exponential loss model with mean \(\theta=100\) dollars. Two estimators are applied to each of four simulated samples. Their outputs are A: \(92,108,97,105\) and B: \(96,104,99,101\), in dollars. Compare their simulated mean squared errors using paired replicate differences, defined as squared error for A minus squared error for B.
The true value is \(100\) dollars. For A, the squared errors are \((92-100)^2=64\), \((108-100)^2=64\), \((97-100)^2=9\), and \((105-100)^2=25\). For B, they are \((96-100)^2=16\), \((104-100)^2=16\), \((99-100)^2=1\), and \((101-100)^2=1\).
The paired differences are therefore \(48,48,8,24\) dollars squared. Their mean is \(\bar D=(48+48+8+24)/4=32\) dollars squared. The simulated MSE for A is \((64+64+9+25)/4=40.5\); for B it is \((16+16+1+1)/4=8.5\). Their difference is \(40.5-8.5=32\), as it must be.
Deviations of the paired differences from 32 are \(16,16,-24,-8\). The sum of their squares is \(256+256+576+64=1{,}152\), giving \(s_D^2=1{,}152/3=384\) dollars to the fourth power. The estimated Monte Carlo standard error is \(\sqrt{384/4}=\sqrt{96}\approx9.798\) dollars squared. The estimated average MSE is lower for B in these replicates, by \(32\) dollars squared. The standard error quantifies simulation variability in that estimated gap.
Common mistake avoided: comparing \((\bar{\hat{\theta}}_A-\theta)^2\) with \((\bar{\hat{\theta}}_B-\theta)^2\) and calling the difference an MSE comparison. MSE averages squared errors across datasets; it is not the squared error of the average estimate.
When the Pairing Is Valid
A paired calculation requires a meaningful match between the two results in each replicate. Both procedures must be evaluated on the same underlying simulated case, or on cases deliberately coupled by a clearly specified simulation design. The replicate pairs must then be independent across \(r\) for the standard error formula above to apply directly. If simulation runs share random inputs across replicates, or if a procedure's randomization creates dependence that is not handled by the design, the simple formula may not describe the Monte Carlo error.
The procedures need not be deterministic. If either uses additional randomization, the estimand should include that randomness, and the simulation should specify whether random-number streams are independent or deliberately coordinated between procedures. A pairing strategy that changes the procedures' marginal behavior would change the comparison target and should not be used.
Pairing is also distinct from forcing the two procedures to have equal results or equal variances. Each procedure retains its own output on each dataset. The benefit, when present, comes from analyzing their within-replicate difference and its variance, including the covariance induced by the shared dataset.
Check Your Understanding
For each problem, use the stated subtraction order and calculate the estimated Monte Carlo standard error from the paired differences.
- Two premium estimators are applied to each of four shared simulated portfolios. The differences \(A-B\), in dollars, are \(-1,3,0,2\). Find \(\bar D\) and its estimated standard error. (Answer: \(\bar D=1\) dollar; \(\widehat{\operatorname{SE}}(\bar D)=\sqrt{5/6}\approx0.9129\) dollars.)
- Two severity procedures produce paired differences \(A-B\) of \(0.2,-0.1,0.3,-0.2,0.3\), in thousands of dollars. Find the mean difference and estimated standard error. (Answer: \(\bar D=0.1\) thousand dollars; \(\widehat{\operatorname{SE}}(\bar D)=\sqrt{0.011}\approx0.1049\) thousand dollars.)
- On four shared datasets, the squared-error differences \(A-B\) are \(5,9,7,3\), in dollars squared. Which method has the smaller simulated MSE, by how much, and what is the estimated standard error of the difference? (Answer: B has the smaller simulated MSE by \(6\) dollars squared; \(\widehat{\operatorname{SE}}(\bar D)=\sqrt{5/3}\approx1.2910\) dollars squared.)
- For \(R=10\) paired replicates, suppose the sample variances of the two performance columns are \(16\) and \(9\), and their sample covariance is \(6\). Calculate the paired and independent-replicate standard error estimates for the difference of their means. (Answer: paired \(=\sqrt{13/10}\approx1.1402\); independent \(=\sqrt{25/10}\approx1.5811\).)
- A study has six paired replicate differences \(4,2,-2,6,0,2\). Calculate the mean difference and estimated standard error. If replicate differences have unchanged variance in a larger run, estimate how many replicates are needed for a standard error of \(1\) or less. (Answer: \(\bar D=2\), \(\widehat{\operatorname{SE}}(\bar D)=\sqrt{4/3}\approx1.1547\); approximately \(8\) replicates are needed.)