Why Benchmark Size Is a Design Decision
A simulation benchmark can rank two methods differently simply because its estimated performance difference is noisy. More replications reduce Monte Carlo uncertainty, but they do not repair unrealistic scenarios, a poorly chosen performance measure, or a comparison that does not answer the scientific question. The design task here is narrower: choose enough independent simulation replications that uncertainty in method differences is small relative to differences that matter in practice.
The earlier material on Monte Carlo replication counts introduced the role of replication number in simulation error. Here the target is the difference between methods, considered across priority scenarios. This distinction matters because a benchmark may estimate each method's performance tolerably well while still estimating their difference imprecisely, especially when the two estimates are noisy or correlated.
Define the Estimand and Precision Target
For a particular scenario, let \(Y_{Ar}\) and \(Y_{Br}\) be the performance scores for methods A and B on simulation replicate \(r\), with larger scores defined as better. The replicate-level difference is \(D_r=Y_{Ar}-Y_{Br}\), and the target is the expected difference \(\Delta=E(D_r)\) under the specified scenario-generating mechanism. If smaller scores are better, the sign convention should be reversed or stated explicitly before analysis.
This paired definition applies when both methods are evaluated on each same replicate. If the methods are evaluated on separate, independent replicates, then the difference of their means has estimated standard error \(\sqrt{s_A^2/R_A+s_B^2/R_B}\), where \(s_A\) and \(s_B\) are the separate sample standard deviations. Do not use the paired standard error for unpaired runs, or the unpaired formula when the paired replicate differences are the intended sampling units.
The estimand is scenario-specific: it averages over the randomness in generating data and, if applicable, algorithmic randomness. Replicates must be independently generated according to the intended simulation design. When the same random data set is used for both methods, the within-replicate difference is the natural unit of analysis; a shared replicate can induce useful covariance, but does not make the replicate differences independent of one another.
From Desired Precision to a Replication Count
Suppose a two-sided confidence interval for \(\Delta\) is to have half-width no greater than \(h\). With a pilot estimate \(s_D\) of replicate-level variability, a normal-approximation planning calculation is
Here \(z_{1-\alpha/2}\) is the standard normal quantile for the desired confidence level; for a 95% interval it is approximately 1.96. The chosen \(h\) is in the same units as the performance difference. If a difference of 0.05 in a recovery score is the smallest practically important contrast, for example, an interval half-width near 0.05 is a possible precision target. It does not itself guarantee that a conclusion about practical importance will be correct: the estimate's location relative to the practical threshold also matters.
The calculation follows directly from the standard error. With known replicate-difference standard deviation \(\sigma_D\), the sample mean has standard deviation \(\sigma_D/\sqrt{R}\), since the \(R\) differences are independent and each has variance \(\sigma_D^2\). A normal interval has half-width \(z_{1-\alpha/2}\sigma_D/\sqrt{R}\). Solving the requirement that this be at most \(h\) yields the displayed count. In practice, \(\sigma_D\) is unknown and replaced by a pilot estimate.
When the standard deviation is estimated from the benchmark, a \(t\)-interval is usually used: \(\bar{D}\pm t_{1-\alpha/2,R-1}s_D/\sqrt{R}\). Thus, the critical value depends on \(R\), and a practical plan uses the normal formula for an initial count, then checks the resulting \(t\)-interval width and increases \(R\) if necessary. This interval is exact under independent normally distributed replicate differences; with enough independent replicates it is often a useful approximation even when the differences are not normal. Strong skewness, rare failures, or a few extreme values can make that approximation unreliable.
Worked Example: Planning for a 0.05-Unit Half-Width
Hypothetical pilot. A paired pilot benchmark produces replicate-level differences in a recovery score with sample standard deviation \(s_D=0.12\). The team wants a 95% confidence interval with half-width at most 0.05 score units.
The normal-approximation calculation is
Since \(R\) must be a whole number, rounding up gives an initial count of 23 paired replicates. The calculation uses the pilot standard deviation of the differences, so it already reflects the covariance between method scores within a replicate.
For a small count, refine with the \(t\) critical value. At \(R=23\), the 95% critical value with 22 degrees of freedom is about 2.074, giving the planning half-width \(2.074(0.12)/\sqrt{23}\approx0.0519\), which exceeds 0.05. At \(R=25\), the critical value with 24 degrees of freedom is about 2.064, so the half-width is \(2.064(0.12)/\sqrt{25}=0.049536\). Thus 25 is a suitable \(t\)-based plan under these assumptions.
Worked Example: A More Variable Priority Scenario
Hypothetical pilot. A second, scientifically important scenario has paired replicate-difference standard deviation \(s_D=0.18\). The target remains a 95% interval half-width of 0.05 performance-score units.
Using the normal approximation,
Rounding up gives 50 as the initial normal-approximation count. Refining the count using the \(t\) critical value gives 53: with \(R=53\), \(df=52\), and \(t_{0.975,52}\approx2.00665\), the half-width is approximately \(2.00665(0.18)/\sqrt{53}=0.0496\). The higher-variance scenario therefore determines the replication count if both scenarios must meet the same half-width target and use a common number of replicates.
The count is not chosen by averaging the two scenario standard deviations. Each scenario has its own uncertainty target; a common count must be large enough for the most demanding scenario. If computational constraints require different counts by scenario, report those counts and the resulting precision separately.
Pairing, Priority Scenarios, and Practical Separation
A benchmark often runs each method on the same simulated data replicate. This pairing is useful when performance scores move together across replicates: unusually difficult data may lower both methods' scores, while unusually easy data may raise both. The variance of the difference is
Positive covariance reduces variance of the paired difference. This can make a paired benchmark substantially more precise than estimating two means from unrelated runs. Pairing is appropriate only if the shared replicate represents the intended comparison and both methods are evaluated under comparable conditions. It must not be simulated as an arbitrary statistical shortcut.
Worked Example: Using the Paired Pilot Variance
Hypothetical pilot. In a priority scenario, method A's score has standard deviation 0.20 and method B's score has standard deviation 0.18 across paired replicates. Their pilot correlation is 0.75. Plan for a 95% interval half-width of 0.04 score units.
The estimated variance of a paired difference is
Thus \(s_D\approx\sqrt{0.0184}=0.1356\). The normal-approximation count is
so the initial count is 45. Refining with the \(t\) critical value gives about 47 paired replicates: with \(df=46\), \(t_{0.975,46}\approx2.013\), and \(2.013(0.1356)/\sqrt{47}\approx0.0398\). For comparison, ignoring pairing would use a variance estimate \(0.0400+0.0324=0.0724\), giving the normal-approximation count \((1.96\sqrt{0.0724}/0.04)^2\approx173.83\), or 174. The paired plan is smaller because the hypothetical pilot indicates strong positive correlation. The unpaired calculation is not the correct primary analysis when matched replicate differences are the estimand.
Precision and practical importance should be planned together. Let \(\delta\) be a prespecified threshold for a practically meaningful difference, with its direction defined in advance. If the benchmark is intended to support a claim that method A exceeds B by more than \(\delta\), a suitable inferential rule might require the lower confidence limit for \(\Delta\) to exceed \(\delta\). A narrow interval centered near zero may rule out large benefits; a wider interval centered at the same estimate may not. Replication planning based only on detecting a difference from zero can therefore miss the actual decision target.
A precision target is often preferable when the central goal is to report an informative interval regardless of its location. If a design instead requires a high probability of concluding that \(\Delta>\delta\) when a specified alternative value \(\Delta_1>\delta\) is true, that is a power calculation. Under a normal approximation, the relevant separation is \(\Delta_1-\delta\), not simply \(\Delta_1\). The required count depends on the chosen error rate, desired power, and variability. These goals are related but not interchangeable.
A Practical Planning Workflow
Define the performance score, its units and direction, the method contrast, the simulation mechanism, and whether methods share each replicate.
Choose a confidence level and a maximum acceptable interval half-width, or specify the practical threshold and desired probability of making the intended conclusion.
Use the standard deviation of paired differences for paired comparisons, or the separate variances for independent groups. Include every priority scenario in this assessment.
Use the normal formula for a starting count, then check the \(t\)-based interval width at that count. Increase the count until the target is met.
Recalculate under plausible larger variability, inspect pilot data for skewness or failures, and document scenario-specific counts and precision targets.
Diagnostics, Alternatives, and Limitations
A planned count should be checked against the realized benchmark. For each priority scenario, report the number of valid replicates, the estimated difference, its Monte Carlo standard error, and an interval. Inspect the replicate-level differences, not only the two method-specific score distributions. Extreme skew, clusters of algorithm failures, or a standard deviation much larger than the pilot value are signals that the initial plan may no longer deliver its target precision. Do not silently discard failed runs: define in advance whether failure is itself an outcome, and account for it consistently.
Pilot variability is uncertain, especially when the pilot is small. Planning exactly at its point estimate can understate the final count. A conservative option is to use an upper plausible value for the standard deviation or to examine several variability scenarios. For example, multiplying the pilot standard deviation by a factor \(c\) multiplies the normal-approximation count by \(c^2\). A 20% increase in the assumed standard deviation therefore raises the count by a factor of \(1.2^2=1.44\). This sensitivity check is often more informative than reporting a single seemingly exact count.
The formula assumes independent replicates from the target simulation mechanism. Correlated draws, repeated reuse of the same generated data, or changes to the data-generating process can make the nominal standard error misleading. Also, the usual interval describes Monte Carlo uncertainty conditional on the specified simulation design; it does not quantify uncertainty about whether that design represents real applications.
The main alternative to equal replication across scenarios is allocating different counts by scenario. Equal counts make comparisons and summaries simpler and protect against undersampling a scenario, but may spend computation inefficiently when variability differs greatly. Unequal allocation can focus computation where the target is hardest to meet, provided all counts are disclosed and the summary estimand is defined appropriately. A second alternative is to report a precision curve: show the expected or observed interval width over a range of replication counts rather than asserting that one count is universally adequate.
If many scenarios or many method contrasts are examined, separate 95% intervals do not automatically provide 95% simultaneous coverage for all claims. A benchmark whose conclusion depends on selecting the best result across many comparisons may need simultaneous intervals or an explicitly defined multiplicity strategy. Increasing \(R\) narrows Monte Carlo intervals but does not by itself resolve multiplicity, selection bias, or scenario imbalance.
Common Pitfall
Check Your Understanding
Answer each question, then compare with the solutions below.
- A paired pilot has \(s_D=0.10\). Using a 95% normal approximation and target half-width 0.04, what initial replication count is required?
- Why can paired simulation replicates reduce uncertainty in a method difference?
- A benchmark uses 40 replicates in each of four scenarios, but one priority scenario has much greater variability than the others. What should be checked before concluding that the benchmark is adequately sized?
- Does an interval half-width target of 0.05 guarantee that a method difference is practically meaningful? Explain.
Solutions. (1) \(R\geq(1.96(0.10)/0.04)^2=(0.196/0.04)^2=4.9^2=24.01\), so round up to 25 as the initial normal-approximation count; a \(t\)-based check may require more. (2) When both methods are evaluated on the same replicate, positive covariance between their scores reduces the variance of their difference: \(\operatorname{Var}(Y_A-Y_B)=\operatorname{Var}(Y_A)+\operatorname{Var}(Y_B)-2\operatorname{Cov}(Y_A,Y_B)\). The paired differences must still be independent across replicates. (3) Check the interval half-width for that scenario using its own replicate-difference variability, and assess whether the pilot adequately represented its failures and variability. A common count is adequate only if the precision target is met for each required scenario. (4) No. The half-width controls interval precision, not the location of the interval. Practical importance depends on where the interval lies relative to a prespecified practical threshold.