Choosing Monte Carlo Replication Counts

Why Replication Counts Need a Precision Target

A simulation study estimates design characteristics by generating trial paths under specified assumptions and applying the planned stopping rule to each path. Even if the code and assumptions are correct, the estimated type I error, power, or mean sample size varies from one set of simulated paths to another. The replication count should therefore be chosen to make this Monte Carlo error small enough for the intended decision or report.

There is no universally sufficient number of paths. A few hundred may estimate a common event probability adequately for a rough check, yet be too few for a narrow interval or a rare event. Conversely, a very large run can give a highly precise estimate of a design characteristic while leaving uncertainty about the assumptions themselves untouched. The calculation begins with a target precision, not with a customary replication count.

Definition: A replication is one independently generated simulated trial path, analyzed according to the specified design and stopping rule. If \(R\) independent paths are generated and \(Y_r\) is a quantity recorded from path \(r\), the Monte Carlo estimate of its mean is \(\bar{Y}_R=R^{-1}\sum_{r=1}^{R}Y_r\). Its Monte Carlo standard error (MCSE) describes the run-to-run standard deviation of that estimate under repeated simulations with the same setup.

The path is the unit of replication. Measurements from several interim looks within one path are not additional independent replications: they belong to the same trial. The calculations below assume paths are independently generated under the same scenario and that the simulation code correctly implements the specified design. If paths are dependent, or if the simulation procedure changes across paths, the ordinary MCSE formulas may not apply.

Replication Counts for Event Probabilities

A simulated type I error or power estimate is an event proportion. For example, define \(I_r=1\) if path \(r\) rejects \(H_0\), and \(I_r=0\) otherwise. If the probability of rejection under the scenario being simulated is \(q\), then \(I_r\) is Bernoulli with variance \(q(1-q)\), and the estimated probability is \(\hat{q}_R=R^{-1}\sum_{r=1}^{R}I_r\). Independence across paths gives

\operatorname{Var}(\hat{q}_R)=\frac{q(1-q)}{R}, \qquad \operatorname{MCSE}(\hat{q}_R)=\sqrt{\frac{q(1-q)}{R}}.

Because \(q\) is usually not known before running the simulation, use a plausible value from prior planning or a separate pilot run, or use the worst-case value \(q(1-q)\leq 0.25\). The inequality holds because \(q(1-q)=0.25-(q-0.5)^2\). For a target absolute MCSE \(e\), the planning inequality is

R\geq \frac{q(1-q)}{e^2}.

Since \(R\) must be an integer, round the calculated requirement up. With no reliable value for \(q\), using \(0.25\) gives \(R\geq 0.25/e^2\). This guarantees the requested standard-error bound for any event probability, under the independent Bernoulli model.

Worked Example: Target MCSE for a Rejection Probability

Suppose a hypothetical design simulation is expected to have a rejection probability near \(q=0.10\), and the target MCSE is at most 0.01. Assume independently generated paths and a correctly implemented rejection indicator.

Using the Bernoulli MCSE formula, the planned count must satisfy

R\geq\frac{0.10(1-0.10)}{0.01^2} =\frac{0.09}{0.0001}=900.

Thus, at least 900 independent paths are required on this planning assumption. The achieved MCSE uses the simulated proportion in place of \(q\); if the observed proportion differs from 0.10, its estimated MCSE will differ too. This count targets absolute precision of about one percentage point in standard-error units, not a guarantee that the estimate is within one percentage point of the true probability.

Planning From a Confidence-Interval Width

A confidence-interval requirement can be translated into a replication count using an approximate normal interval. For a two-sided confidence level of 95%, the approximate half-width is \(1.96\) times the MCSE. If the desired total width is \(W\), set the target half-width to \(d=W/2\). Using a planning value \(q\), the approximate requirement is

1.96\sqrt{\frac{q(1-q)}{R}}\leq d \quad\Longrightarrow\quad R\geq\frac{1.96^2q(1-q)}{d^2}.

If \(q\) is uncertain, replace \(q(1-q)\) with 0.25 for a conservative normal-approximation calculation. The approximation is most trustworthy when the simulated event and non-event counts are both not small. For rare events or small counts, the normal interval can be inaccurate; choose the count using an appropriate binomial interval calculation rather than assuming this approximation will deliver the desired actual interval width.

Worked Example: A 95% Interval for an Event Probability

For a hypothetical operating-characteristic estimate, suppose a 95% interval should have an approximate total width no greater than 0.04. With no reliable prior value of \(q\), use the worst-case variance \(0.25\). The target half-width is \(d=0.04/2=0.02\). The approximate count is

R\geq\frac{1.96^2(0.25)}{0.02^2} =\frac{3.8416(0.25)}{0.0004} =\frac{0.9604}{0.0004}=2401.

Plan for at least 2,401 independent paths for this approximate width target. If a well-supported planning value were \(q=0.20\), then the corresponding calculation would be \(R\geq 3.8416(0.16)/0.0004=1{,}536.64\), requiring 1,537 paths. That smaller count depends on the assumed probability being suitable; it is not the worst-case guarantee.

Precision for Expected Sample Size

The mean sample size is not an event proportion. Let \(N_r\) be the final sample size or information amount recorded for path \(r\), using the same units for every path. If the path-level values are independent with finite variance \(\sigma_N^2\), the estimated mean \(\bar{N}_R\) has MCSE \(\sigma_N/\sqrt{R}\). In practice, estimate the variability with the sample standard deviation \(s_N\), giving estimated MCSE \(s_N/\sqrt{R}\). A planning standard deviation or pilot estimate \(\sigma_N\), together with target half-width \(d\), yields the approximate 95% requirement \(R\geq(1.96\sigma_N/d)^2\).

If the final sample size is known to lie in an interval from \(a\) to \(b\), its variance is at most \((b-a)^2/4\). This bound follows because the variance of any variable confined to an interval of width \(b-a\) cannot exceed one quarter of the squared width. It can provide a conservative planning value when no credible pilot estimate of variability is available, although it may require many more paths than a realistic variance estimate.

Worked Example: Precision for Mean Sample Size

In a hypothetical simulation, final sample size is recorded in participants, and a separate pilot suggests a path-to-path standard deviation of 100 participants. The aim is an approximate 95% confidence-interval half-width of at most 5 participants for the mean.

Using the normal multiplier 1.96, the required count is

R\geq\left(\frac{1.96(100)}{5}\right)^2 =(39.2)^2=1{,}536.64.

Rounding up gives a requirement of 1,537 independent paths. The units cancel in the ratio: 1.96 times 100 participants divided by 5 participants. This is a planning calculation based on the pilot standard deviation; after simulation, report the estimated mean in participants and its MCSE, calculated from the variability across the production paths.

Putting the Calculation Into Practice

A simulation may report several characteristics from the same paths. Each characteristic has its own variability and precision requirement. For example, the rejection indicator and final sample size from a path are generally related, but the probability estimate uses the Bernoulli variance formula while the mean sample-size estimate uses the variability of \(N_r\). Choose \(R\) large enough for the most demanding primary precision target, then calculate and report the MCSE for each result.

Name the target.
Specify the scenario and the quantity to estimate: an event probability, mean sample size, or another path-level mean. State the units and whether the target is an MCSE or interval width.

Specify the variability assumption.
For an event probability, use a planning \(q\) or the conservative variance 0.25. For a mean, use a justified standard deviation estimate or a valid bound.

Solve and round up.
Apply the appropriate formula, check the arithmetic, and round the result to the next integer replication count.

Check achieved precision.
After the run, calculate the estimated MCSE from the simulated data and compare it with the stated target. A planning target is not itself evidence that the achieved precision met the target.

A pilot run can help estimate variability when it is hard to specify in advance. Its role is to inform the count, not to settle whether the design assumptions are correct. If the pilot estimate is unstable, use a conservative value or increase the count. If estimating several scenario-specific quantities, plan separately for each scenario; a count sufficient for a common event probability may be insufficient for a rare one.

Diagnostics, Alternatives, and Limitations

Review the achieved MCSEs alongside the estimates. For an event probability, the plug-in estimate is \(\sqrt{\hat{q}_R(1-\hat{q}_R)/R}\); for a mean sample size, it is \(s_N/\sqrt{R}\). Compare these with the pre-specified targets. A zero observed event count does not establish that the event probability is zero: the plug-in MCSE is then zero, but that does not represent the uncertainty well. Rare events require a method that accounts for small binomial counts and may need substantially more paths.

When available, deterministic calculations or a validated analytical method can sometimes estimate a design characteristic without Monte Carlo error. They may be preferable for a quantity with a tractable exact calculation. Simulation remains useful when the design and stopping rules make such calculations impractical. Increasing \(R\) reduces Monte Carlo error at the rate \(1/\sqrt{R}\): quadrupling the count approximately halves the MCSE. It does not remove bias from incorrect code, an unsuitable model, or an incomplete set of scenarios.

A common mistake is to use a fixed “large” number of replications for every result and assume that it implies adequate precision. The correction is to state a precision target and calculate the required count for the relevant quantity and its variability. Another is to confuse a narrow simulation interval with strong evidence that the planned design is appropriate. The interval describes simulation variability under the assumed scenario; it does not quantify uncertainty about whether that scenario represents the trial.

Key takeaway: Choose replication counts from the uncertainty of the path-level quantity and a stated precision goal. Use the Bernoulli formula for event probabilities, the sample-mean formula for expected sample size, and verify achieved MCSEs after the run.

Check Your Understanding

For each problem, identify the quantity and use the stated assumptions. Solutions follow the questions.

An event probability is expected to be 0.25. How many independent paths are needed for an approximate MCSE no greater than 0.005?
Using the worst-case event-probability variance, find the approximate count for a 95% interval with total width 0.02.
A pilot suggests that final sample size has standard deviation 60 participants. What count gives an approximate 95% half-width of 3 participants for the mean?
Why is a zero plug-in MCSE after observing no events not proof of exact precision?

Solutions. (1) \(R\geq0.25(0.75)/0.005^2=0.1875/0.000025=7{,}500\) paths. (2) The half-width is 0.01, so \(R\geq1.96^2(0.25)/0.01^2=0.9604/0.0001=9{,}604\) paths. This is a normal-approximation planning count; rare-count interval behavior should also be checked. (3) \(R\geq(1.96(60)/3)^2=(39.2)^2=1{,}536.64\), so plan for 1,537 paths. (4) With no events, the plug-in proportion is zero and its plug-in variance is also zero, but the true probability may be positive. Small-count uncertainty is not represented adequately by that zero estimate, so use an appropriate binomial interval and consider increasing the count.