Choosing Replicate Counts With Monte Carlo Error

Why Replicate Counts Matter

A simulation study estimates design performance by repeatedly generating data under a specified scenario and applying the planned analysis. Even when the scenario and analysis are held fixed, the estimated performance varies across runs because each run uses a finite number of random replicates. That variation is Monte Carlo error. Choosing the number of replicates is therefore a precision decision: the aim is to make simulation noise small enough for the performance estimate to answer the design question.

Here, a replicate is one complete simulated data set followed by the specified analysis and calculation of the performance measure. Let \(R\) be the number of replicates. The precision calculations below assume that, conditional on the chosen scenario and simulation procedure, replicates are generated independently from the same distribution. If replicates are dependent, the formulas based on \(R\) independent observations do not generally apply.

Definition: The Monte Carlo standard error (MCSE) is the standard deviation, across repeated runs of the simulation with the same \(R\), of the estimator of a performance quantity. It measures simulation noise in that estimate; it is not the standard error of a clinical parameter estimate within one simulated data set.

MCSE addresses finite simulation size, not whether the data-generating model is appropriate, whether a chosen performance measure is clinically relevant, or how uncertain the model inputs are. Those questions remain important, but increasing \(R\) does not resolve them.

Estimating a Performance Probability

Suppose the performance measure is the probability that an analysis meets a specified criterion, such as producing a successful fit according to a prespecified rule. For replicate \(r\), define \(I_r=1\) if the criterion is met and \(I_r=0\) otherwise. Let \(p=P(I_r=1)\) be the performance probability under the scenario, and let \(K=\sum_{r=1}^{R} I_r\) be the number of successful replicates. The estimate is \(\hat p=K/R\).

Result: If the \(I_r\) are independent Bernoulli random variables with common probability \(p\), then \(SE(\hat p)=\sqrt{p(1-p)/R}\). Since \(p\) is unknown, estimate the MCSE by \(\widehat{MCSE}(\hat p)=\sqrt{\hat p(1-\hat p)/R}\).

The result follows from the variance of an average. Each indicator has variance \(p(1-p)\). Independence makes the variance of their sum equal to \(R p(1-p)\), so dividing the sum by \(R\) gives the variance of \(\hat p\):

\operatorname{Var}(\hat p) =\operatorname{Var}\left(\frac{1}{R}\sum_{r=1}^{R} I_r\right) =\frac{1}{R^2}\sum_{r=1}^{R}\operatorname{Var}(I_r) =\frac{p(1-p)}{R}.

Taking the square root gives the standard error. Replacing the unknown \(p\) with the observed \(\hat p\) gives the estimated MCSE. This plug-in estimate is useful for planning and reporting, but it is itself estimated from the simulation results.

Worked Example: MCSE for an Estimated Success Probability

In a hypothetical simulation of a design, \(R=1{,}000\) independent replicates are analyzed. A prespecified criterion is met in \(K=820\) replicates. Estimate the probability of meeting the criterion and its MCSE.

The estimated probability is \(\hat p=820/1{,}000=0.820\). The estimated MCSE, in probability units, is

\widehat{MCSE}(\hat p) =\sqrt{\frac{0.820(1-0.820)}{1{,}000}} =\sqrt{0.0001476} =0.012149\ldots

Thus the estimated MCSE is \(0.01215\) in probability units, or \(1.215\) percentage points. The simulation estimates an 82.0% success probability, with a Monte Carlo standard error of about 1.215 percentage points. This is a measure of simulation precision, not a statement that the true probability is within 1.215 percentage points with a particular confidence level.

Choosing Replicates for a Target Precision

For a probability estimate, specify a desired maximum MCSE in probability units, say \(\delta\). Rearranging the formula gives a planning requirement. If a planning value \(p_0\) is available, choose \(R\) large enough that the estimated standard error under that value is no greater than \(\delta\). Because \(R\) must be an integer, round the calculated requirement up.

\sqrt{\frac{p_0(1-p_0)}{R}}\leq \delta \quad\Longleftrightarrow\quad R\geq \frac{p_0(1-p_0)}{\delta^2}.

The target must be expressed in the same units as the probability: a target of 0.5 percentage points is \(\delta=0.005\), not \(0.5\). A pilot estimate can supply \(p_0\), but the planning result depends on whether that estimate is representative. If no credible value is available, use the fact that \(p(1-p)\leq 0.25\) for \(0\leq p\leq1\). This yields the conservative bound \(R\geq0.25/\delta^2\), which protects against any probability value under the assumed model.

Worked Example: Planning From a Pilot Estimate

Use the hypothetical pilot estimate \(\hat p=0.820\) to plan a new simulation whose estimated MCSE is at most 0.5 percentage points. Assume the pilot probability is a reasonable planning value.

Convert the target to probability units: \(\delta=0.5/100=0.005\). Substituting \(p_0=0.820\) gives

R\geq\frac{0.820(1-0.820)}{0.005^2} =\frac{0.1476}{0.000025} =5{,}904.

Therefore, at least 5,904 replicates are required under this plug-in planning assumption. If 6,000 replicates are run, the anticipated MCSE is \(\sqrt{0.1476/6{,}000}=0.004960\), or \(0.4960\) percentage points. The pilot's 1,000 replicates had an estimated MCSE of 1.215 percentage points, so that run did not meet the new target.

Worked Example: Planning Conservatively Without a Pilot

Suppose a design team wants an MCSE no larger than 1.0 percentage point for a success probability, but has no reliable planning value for \(p\). Find a replicate count that guarantees the plug-in standard-error formula is at most the target for any \(p\).

Here \(\delta=0.010\). Since \(p(1-p)\leq0.25\), require

R\geq\frac{0.25}{0.010^2} =\frac{0.25}{0.0001} =2{,}500.

A run of 2,500 independent replicates is the conservative choice. For comparison, if the true probability were \(p=0.90\), the standard error at that replicate count would be \(\sqrt{0.90(0.10)/2{,}500}=0.006000\), or 0.6000 percentage points. The worst case occurs at \(p=0.50\), where it is exactly 1.000 percentage point. The conservative choice can use more computation than a credible pilot-based plan, but it avoids relying on an uncertain value of \(p\).

Continuous Performance Measures

Not every performance measure is a probability. Suppose \(Y_r\) is a numerical result from replicate \(r\), such as the estimated dose needed to reach a target or the number of observations required by a prespecified rule. If the goal is the mean performance \( \psi=E(Y_r) \), estimate it by \(\bar{Y}\), the average over \(R\) replicates. Under independent, identically distributed replicates with finite variance, the standard error is \(\sigma_Y/\sqrt{R}\), where \(\sigma_Y\) is the standard deviation of replicate-level results. In practice, estimate \(\sigma_Y\) by their sample standard deviation \(s_Y\), so the estimated MCSE is \(s_Y/\sqrt{R}\).

Worked Example: Replicates for a Mean Performance Measure

In a hypothetical pilot run, the mean simulated time to reach a target concentration is 14.2 hours, and the sample standard deviation of the replicate-level times is \(s_Y=3.60\) hours from \(R=400\) replicates. Find the estimated MCSE and plan the replicate count for an MCSE no greater than 0.10 hours, assuming the pilot standard deviation is suitable for planning.

The pilot MCSE is

\widehat{MCSE}(\bar{Y}) =\frac{s_Y}{\sqrt{R}} =\frac{3.60\ \text{hours}}{\sqrt{400}} =0.180\ \text{hours}.

For the target \(\delta=0.10\) hours, require \(s_Y/\sqrt{R}\leq0.10\). Squaring both sides and rearranging gives

R\geq\left(\frac{s_Y}{\delta}\right)^2 =\left(\frac{3.60}{0.10}\right)^2 =1{,}296.

Thus 1,296 replicates meet the planning target if the replicate-level standard deviation remains about 3.60 hours. At that count, the estimated MCSE is \(3.60/\sqrt{1{,}296}=3.60/36=0.1000\) hours. The mean of 14.2 hours describes the estimated performance; the 0.180-hour MCSE describes the finite-simulation precision of that mean.

Diagnostics, Alternatives, and Limitations

A replicate count should be chosen for the performance measure and scenario that need adequate precision. A count adequate for a frequent binary outcome may be inadequate for a rare event, a highly variable continuous outcome, or a stringent precision target. When several scenarios or metrics are central to the decision, plan for the least precise important estimate rather than assuming one replicate count works equally well for all.

Check the units. Convert percentage points to probability units before using the planning equation, and retain outcome units for continuous measures.
Check the variability estimate. For a mean, \(s_Y\) from a small pilot can be unstable. A larger pilot or a defensible upper bound on variability may be needed.
Do not treat a zero estimate as zero uncertainty. If no successes or all successes are observed, \(\hat p(1-\hat p)/R=0\). This does not prove that the underlying probability is exactly 0 or 1. Use a conservative bound, a justified pilot value, or an appropriate binomial interval to assess the uncertainty.
Inspect replicate independence. If simulations reuse random quantities in a dependent way, the independent-replicate formulas can understate or misstate MCSE. The variance calculation must reflect the actual simulation procedure.
Compare the MCSE with the decision scale. An MCSE can be numerically small yet too large to distinguish performance values that would lead to different design choices.

An alternative to reporting an MCSE is to report an interval for a simulated probability, such as a binomial confidence interval. Such intervals communicate uncertainty using a confidence-interval procedure and can be preferable when a normal approximation is unreliable, particularly for small counts or probabilities near 0 or 1. The MCSE is a compact measure of standard simulation noise; it is not itself a confidence interval. For continuous outcomes, replicate-level variability and an interval for the mean can also be reported, but the basic planning relationship still depends on the standard deviation and \(R\).

A common mistake is to claim that multiplying the number of replicates by four makes the estimate four times more precise. The standard error decreases as \(1/\sqrt{R}\), so quadrupling \(R\) halves the MCSE; it does not reduce it to one quarter. Another mistake is to increase \(R\) and assume that this corrects model misspecification or uncertainty in scenario inputs. It only reduces Monte Carlo noise conditional on the simulation setup.

Define the target.
State the performance quantity, scenario, and acceptable MCSE in explicit units.

Choose a planning variability.
Use a credible pilot estimate or, for a probability, the conservative bound \(p(1-p)\leq0.25\).

Calculate and round up.
Use \(R\geq p_0(1-p_0)/\delta^2\) for a probability or \(R\geq(s_Y/\delta)^2\) for a mean.

Verify the achieved precision.
After simulation, calculate the MCSE from the observed results and compare it with the prespecified target.

Key takeaway: For independent simulation replicates, Monte Carlo precision improves at the rate \(1/\sqrt{R}\). State the target precision first, plan \(R\) using the relevant replicate-level variability, and then verify the achieved MCSE rather than relying on replicate count alone.

Check Your Understanding

Answer each question, then check the concise solutions.

A hypothetical simulation estimates a probability as \(0.60\) using 900 independent replicates. What is its estimated MCSE in probability units and percentage points?
Using \(p_0=0.60\), how many replicates are needed for an estimated MCSE no greater than 0.2 percentage points?
A continuous replicate-level outcome has sample standard deviation 5 units. How many replicates are needed for an estimated MCSE no greater than 0.25 units?
Why is an estimated MCSE of zero after observing no successes not evidence that the true success probability is exactly zero?

Solutions. (1) \(\sqrt{0.60(0.40)/900}=\sqrt{0.0002667}=0.01633\), or 1.633 percentage points. (2) The target is \(\delta=0.002\), so \(R\geq0.60(0.40)/(0.002)^2=0.24/0.000004=60{,}000\). (3) \(R\geq(5/0.25)^2=20^2=400\). (4) The plug-in formula substitutes the observed proportion, which is zero, but a finite run can observe no successes even when the underlying probability is positive; a conservative planning bound or an appropriate interval is needed.