Selecting The Number Of Simulation Replicates

Why the Replicate Count Is a Design Choice

A simulated operating characteristic is an estimate of a probability defined by a trial procedure and a data-generating scenario. For example, one may estimate the probability that a Bayesian decision rule declares success under a specified treatment effect. Running more simulated trials usually makes that estimate more precise, but also costs more computation. The replicate count should therefore be chosen to meet a stated precision goal, not simply because a particular number is conventional.

Here, \(R\) denotes the number of independent simulated trial replicates for one scenario. In each replicate, record \(Y_r=1\) if the event of interest occurs and \(Y_r=0\) otherwise. The estimated operating characteristic is \(\hat{\theta}=\sum_{r=1}^{R}Y_r/R\). The target \(\theta\) is the event probability under the specified simulation mechanism. Monte Carlo uncertainty describes the variability of \(\hat{\theta}\) from using a finite number of simulated replicates; it is not uncertainty about the clinical effect in an individual trial, nor uncertainty about whether the simulation mechanism is appropriate.

Planning From the Bernoulli Variance

For a fixed scenario, suppose the simulated trial replicates are independent and identically generated, and the event indicator in each replicate has probability \(\theta\). Then \(Y_r\) is Bernoulli with variance \(\theta(1-\theta)\), and the variance of the replicate average is \(\theta(1-\theta)/R\). Thus, its Monte Carlo standard error is estimated by substituting \(\hat{\theta}\) for \(\theta\). This is the standard error due to finite simulation alone.

This relationship gives a direct planning formula. If the goal is a Monte Carlo standard error no greater than \(s_0\), use an anticipated probability \(\theta_0\) and choose \(R\) so that \(\sqrt{\theta_0(1-\theta_0)/R}\leq s_0\). If the goal is an approximate two-sided confidence-interval half-width no greater than \(h\) at confidence level \(1-\alpha\), use the normal-approximation multiplier \(z_{1-\alpha/2}\) and require \(z_{1-\alpha/2}\sqrt{\theta_0(1-\theta_0)/R}\leq h\). In either case, solve for \(R\) and round up to an integer.

The half-width formula follows by multiplying the Monte Carlo standard error by the normal critical value. For a 95% interval, \(z_{1-\alpha/2}\) is approximately 1.96. The formula is a planning approximation, not a guarantee that a realized interval will have exactly the requested width. In particular, the realized estimate can differ from the anticipated probability.

When the Probability Is Not Known

The planning formula depends on \(\theta(1-\theta)\), which is largest at \(\theta=0.5\). This gives a conservative choice when no credible anticipated probability is available and the event is binary. The reason is that \(\theta(1-\theta)=1/4-(\theta-1/2)^2\), which cannot exceed \(1/4\). Consequently, a replicate count planned at \(\theta_0=0.5\) meets the approximate target for every probability between 0 and 1, under the same assumptions and approximation.

Planning Across a Scenario Matrix

Calibration often evaluates the same decision rule under multiple scenarios. A single replicate count can be used for all scenarios, but its adequacy should be assessed against the most demanding scenario. For an absolute half-width target \(h\), calculate the required \(R\) for each anticipated probability and use the largest result. If no reliable probability forecasts exist, use the worst-case variance for each binary operating characteristic. If simulation costs differ materially across scenarios, report that difference and make the budget trade-off explicit rather than treating a common count as automatically optimal.

Cost, Diagnostics, and Alternatives

The choice of target is a scientific and computational decision. A small change in an estimated operating characteristic may be immaterial for the decision being calibrated, while a few percentage points may matter greatly near a prespecified performance threshold. Decide what differences would alter interpretation, then set the precision target accordingly. Because the required count is proportional to \(1/h^2\), halving the desired half-width requires approximately four times as many replicates. For example, changing a target half-width from 0.02 to 0.01 multiplies the planned count by four, all else equal.

A pilot run can provide an estimate of the probability or variance for planning. It is useful when simulation cost is substantial and prior information is weak, but a small pilot can underestimate variance by chance. A conservative alternative is to plan using a range of plausible probabilities or the worst-case variance. For a non-binary replicate-level quantity, replace the Bernoulli variance with the variance of that quantity and plan using an estimated or conservatively bounded variance; the Bernoulli formulas do not apply unchanged.

After running the simulations, compare the observed Monte Carlo standard error or interval width with the target. Also check that the number of completed replicates matches the plan, that event indicators are coded correctly, and that the simulated outcomes are consistent with the specified data-generating mechanism. A discrepancy may arise because the observed probability differs from its planning value, because a normal approximation is poor, or because the simulation implementation has a problem. More replicates reduce Monte Carlo variability, but they do not fix a misspecified model, a coding error, or biased simulation.

For rare events or estimates close to zero or one, a normal-approximation interval can behave poorly, and a planned approximate half-width should not be presented as an exact guarantee. A Wilson or exact binomial interval can be used to assess precision in such settings; planning may require evaluating candidate replicate counts with the chosen interval method. If the goal is a fixed relative precision rather than a fixed absolute half-width, the required count depends strongly on the event probability and can become very large for rare events. A fixed computational budget is another option, but it should be accompanied by the precision that the budget is expected to deliver.

A common mistake is to say, “We will run 1,000 replicates because that is usually enough.” That reasoning gives no precision criterion and ignores whether the target probability is near 0.5, whether the event is rare, and how the estimate will be used. The correction is to state an absolute standard-error or interval-width target, identify the relevant probability or variance range, calculate the count, and round upward. Remember that the resulting precision concerns simulation error only; it does not measure uncertainty in the clinical assumptions or trial model.