Why Replication Counts Need a Precision Target
A simulation result is itself an estimate: its value depends on which simulated datasets happened to be generated. Running more replications reduces this Monte Carlo variability, but consumes computing time. The useful planning question is therefore not simply “How many replications are customary?” It is “How much numerical uncertainty can we tolerate in the performance estimates that matter?”
A replication is one generated dataset under a specified scenario, followed by the planned analysis or analyses. It is not an individual participant, feature, or omics modality within that dataset. If a scenario has \(R\) replications, its estimate is based on \(R\) simulated datasets. The scenario matrix determines which conditions are examined; replication planning determines how precisely the quantities of interest can be estimated within those conditions.
A basic plan assumes that, within a scenario, the replicate-level quantities \(G_1,\ldots,G_R\) are independent and generated under the same simulation design, with finite variance. If their standard deviation is \(\sigma_G\), then the variance of their average is \(\sigma_G^2/R\). Replacing the unknown \(\sigma_G\) with the sample standard deviation \(s_G\) gives the estimated MCSE \(s_G/\sqrt{R}\). For a proportion, a more specific formula is available.
Planning for a Proportion
Many simulation summaries are proportions: for example, the fraction of replications in which an algorithm converges, recovers a specified structure, or declares a signal. If \(I_r\) is 1 when the event occurs on replicate \(r\) and 0 otherwise, the estimate is \(\hat{p}=R^{-1}\sum I_r\). Under independent, identically distributed replications, \(I_r\) is Bernoulli with event probability \(p\), so its variance is \(p(1-p)\). Consequently, the estimated MCSE is \(\sqrt{\hat{p}(1-\hat{p})/R}\).
Here \(\delta\) is the largest acceptable MCSE in proportion units. Since \(p\) is not known before running the simulation, a pilot estimate can be used, or a conservative bound can be used: \(p(1-p)\leq 0.25\), with equality at \(p=0.5\). The bound gives a replication count adequate for the requested standard-error target at any possible proportion. Round the required count up to an integer.
Worked Example: Precision for a Recovery Proportion
Suppose a pilot suggests that a method recovers the target structure in about 80% of datasets. The design requires the estimated recovery proportion to have MCSE no greater than 0.01, or one percentage point. Using the pilot value in the planning formula gives
Thus, plan for at least 1,600 independent datasets in this scenario. At that count, if the true proportion is 0.80, the MCSE is approximately \(\sqrt{0.16/1600}=0.01\). If there is no credible pilot value, the conservative calculation is \(R\geq 0.25/0.0001=2500\). The smaller pilot-based count is reasonable only if the pilot is representative and the target is specifically the standard error, rather than a confidence-interval half-width.
An MCSE target and a confidence-interval half-width are not interchangeable. With a normal approximation, an approximate 95% Monte Carlo interval has half-width \(1.96\times\operatorname{MCSE}\). Therefore, a requirement that this half-width be at most \(h\) calls for approximately \(R\geq 1.96^2p(1-p)/h^2\). This interval describes simulation error in estimating the scenario’s performance quantity; it is not a statement that the simulation model or real-world method performance is correct.
Planning for Means and Method Differences
A continuous performance estimate, such as a loss or a numerical recovery score, is often averaged across replications. For replicate-level values with standard deviation \(\sigma_G\), the same variance argument yields \(\operatorname{MCSE}(\bar{G})\approx s_G/\sqrt{R}\). If the goal is a 95% Monte Carlo interval with half-width no larger than \(h\), a normal-approximation planning calculation is \(R\geq(1.96s_G/h)^2\). This requires a representative estimate of \(s_G\), usually from a pilot or prior comparable simulations.
Worked Example: A Continuous Performance Estimate
In a pilot, a continuous recovery score has sample standard deviation \(s_G=0.12\) score units. The investigators want an approximate 95% Monte Carlo interval for the mean score with half-width at most 0.01 score units. The planning calculation is
Rounding up gives 554 replications. At this count, the estimated half-width is approximately \(1.96(0.12/\sqrt{554})=0.00999\) score units. This calculation assumes the pilot standard deviation is representative and that the mean’s sampling distribution is adequately approximated by a normal distribution at the chosen count. If the score is strongly skewed or occasionally extreme, a pilot may not estimate its variability reliably; more pilot runs and direct examination of the replicate-level values are warranted.
For comparing two methods, the relevant target is often the mean of their replicate-level differences, not the separate precision of each method’s score. When both methods are applied to the same simulated dataset, define \(D_r=G_{A,r}-G_{B,r}\). The estimated mean difference is \(\bar{D}\), and its MCSE is \(s_D/\sqrt{R}\). Pairing matters because the methods encounter the same underlying simulated difficulty on each replicate. Their outcomes can be correlated, and the variability of the difference can be smaller than it would be if each method were evaluated on unrelated datasets.
The covariance term explains why using matched datasets can improve precision when the method outcomes tend to move together. The comparison must preserve the pairing in its calculation; subtracting two separately estimated standard errors does not give the MCSE of the difference.
Worked Example: Planning a Paired Method Comparison
A pilot uses 200 matched datasets to compare two methods on a binary recovery outcome. Both methods recover the target in 100 of 200 datasets. They both succeed in 70 datasets and both fail in 70; method A alone succeeds in 30, and method B alone succeeds in 30. Set \(D_r=1\) when only A succeeds, \(D_r=-1\) when only B succeeds, and \(D_r=0\) when they agree. The pilot therefore has 30 values of 1, 30 values of -1, and 140 values of 0, with \(\bar{D}=0\).
The sample variance of the paired differences is
Suppose the target is a paired-difference MCSE no greater than 0.02. Using the pilot variance, the required replication count is
Round up to 754 matched datasets. For comparison, if the outcomes from the two methods were treated as independent Bernoulli estimates, the pilot marginal variances would each be \(0.5(1-0.5)=0.25\). Their difference would have variance \(0.25+0.25=0.50\), requiring \(R\geq0.50/0.0004=1250\) independent datasets per method for the same standard-error target. The matched design uses the observed within-dataset relationship; its advantage should be verified rather than assumed.
A Practical Planning Process
Specify the scenario, performance summary, and whether the target is a proportion, mean, or paired method difference. Define how failed or undefined analyses enter the summary rather than silently dropping them.
Choose an acceptable MCSE or Monte Carlo interval half-width in meaningful units. A 0.01 target means one percentage point for a proportion, but 0.01 score units for a score.
Use a pilot that reflects the planned scenario and analysis. For proportions, consider the conservative \(p=0.5\) bound if the pilot is uninformative. For comparisons, estimate the standard deviation of the replicate-level differences.
Apply the appropriate formula, round upward, and check whether the resulting total across scenarios and methods fits the computational budget. If it does not, revise the precision target or budget explicitly rather than reporting an unjustified small count.
After running the planned replications, calculate MCSEs from the replicate-level outcomes and compare them with the targets. Large discrepancies may indicate that pilot variability was not representative or that assumptions behind the approximation are weak.
Scenarios, Diagnostics, and Limitations
A single replication count for every scenario is simple to implement, but can be wasteful when some scenarios have much less variability than others. Conversely, planning solely from an easy scenario can leave difficult or boundary scenarios imprecise. For a fixed per-scenario precision requirement, calculate counts for each scenario and metric, then use the largest relevant requirement when a common count is needed. If counts differ by scenario, state the rule used to allocate them. A total budget is the sum of the planned counts across scenarios, multiplied by the computational cost per replicate where costs differ.
Diagnostics should focus on the replicate-level quantities behind the estimate. Check the observed sample standard deviation against the pilot value, inspect whether a few extreme outcomes dominate a mean, and verify that each scenario actually received its planned number of valid replications. For a proportion near zero or one, the plug-in MCSE can be small even when the event is rare and its frequency is unstable; the normal approximation may also be poor when expected event counts are small. In that setting, use a binomial-appropriate interval or a conservative planning calculation rather than relying blindly on a normal approximation.
The formulas require independent replications within the scenario. If simulation outputs are dependent, for example because the data-generating process deliberately links replicate outputs, \(s_G/\sqrt{R}\) need not estimate the MCSE correctly. Likewise, an unstable algorithm that produces occasional very large values can make a small pilot’s standard deviation misleading. A short pilot is for estimating variability, not a guarantee that a final count will achieve the target. It is prudent to allow a margin when pilot estimates are uncertain and to report achieved MCSEs alongside estimates.
A pilot-based count should ordinarily be fixed before the main run, with the pilot excluded from the final estimate if its role or settings differ. Repeatedly checking the running estimate and stopping the instant it looks favorable can create stopping-related distortions. If a sequential stopping rule is used, it should be defined in advance and its statistical properties considered. Planning many scenarios also does not make a weak per-scenario precision target adequate: each key estimate needs a defensible target, and comparisons intended for interpretation need their own difference-scale target.
The central trade-off is between precision and computational cost, not between a universally “correct” count and an incorrect one. Replication planning makes that trade-off visible and connects it to the outputs a simulation is meant to estimate. For method comparisons, matched replicate-level contrasts are often the most direct target; for each scenario, the count should be justified by the variability and precision needs of its important summaries.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A proportion is expected to be 0.5. How many replications are needed for an estimated MCSE no greater than 0.02?
- A continuous summary has estimated standard deviation 0.30. What count gives an approximate 95% Monte Carlo interval with half-width at most 0.03?
- Why can matched datasets reduce the MCSE of a method difference, and what replicate-level quantity should be used?
- For a scenario with \(\hat{p}=0.10\) and \(R=900\), calculate the estimated MCSE of the proportion.
Solutions. 1. \(R\geq0.5(1-0.5)/0.02^2=0.25/0.0004=625\) replications. 2. \(R\geq(1.96(0.30)/0.03)^2=19.6^2=384.16\), so use 385. 3. Matched datasets can induce positive covariance between method outcomes, reducing the variance of their difference. Calculate \(D_r=G_{A,r}-G_{B,r}\) on each shared dataset and use \(s_D/\sqrt{R}\). 4. \(\operatorname{MCSE}=\sqrt{0.10(0.90)/900}=\sqrt{0.0001}=0.0100\), or one percentage point.