Replication Counts for Target Precision

Why Choose the Number of Replicates in Advance?

A simulation estimate is itself subject to Monte Carlo error: repeating the simulation with new random draws would generally produce a different estimate. In design assurance, that estimate may be a probability, such as the chance that an adaptive trial selects the correct treatment, or a mean, such as average trial duration. The number of simulated trials, denoted by \(R\), determines how precisely these quantities are estimated. Planning \(R\) around a precision target helps avoid both an unnecessarily large run and a run too small to support a useful estimate.

The earlier material on Monte Carlo standard errors for simulated rates gives the standard error of a simulated proportion. Here the goal is to invert that relationship: specify a desired standard error or approximate confidence-interval width, then find the smallest positive integer \(R\) that meets the target under stated planning assumptions.

Replication Counts for a Simulated Probability

Definition: Suppose each independently simulated trial produces an indicator \(Y_r\), equal to 1 when a specified event occurs and 0 otherwise, for \(r=1,\ldots,R\). If the event probability is \(p\), then the estimated probability is \(\hat{p}=\bar{Y}\), and its Monte Carlo standard error is \(\sqrt{p(1-p)/R}\). A target Monte Carlo standard error \(\delta>0\) means planning for this standard error to be no greater than \(\delta\).

This formula assumes that, under the fixed simulation scenario and algorithm, the \(Y_r\) are independent and have the same event probability \(p\). Since an indicator has variance \(p(1-p)\), the variance of the average of \(R\) independent indicators is \(p(1-p)/R\). Taking the square root gives the stated standard error.

Solving \(\sqrt{p(1-p)/R}\leq\delta\) for \(R\) gives the required replication count. Because \(R\) must be a positive integer, the planned count is

R_{\min}=\max\left(1,\left\lceil\frac{p(1-p)}{\delta^2}\right\rceil\right).

The ceiling rounds up, not to the nearest integer: rounding down could miss the target. The maximum with 1 ensures a positive replication count, including at \(p=0\) or \(p=1\), where the variance formula itself is zero. In practice, \(p\) is usually unknown, so this expression needs a planning value for \(p\). That value may come from a scientifically plausible scenario or a preliminary simulation; it is not a guarantee that the final Monte Carlo standard error will meet the target if the actual probability differs.

Worked Example: Target Monte Carlo Error for a Simulated Rate

Suppose a hypothetical simulation estimates the probability that a trial selects the correct treatment. For planning, use \(p=0.24\) and require a Monte Carlo standard error no greater than \(0.01\).

For an indicator, the planned variance is \(p(1-p)=0.24(0.76)=0.1824\). Thus,

R_{\min}=\max\left(1,\left\lceil\frac{0.1824}{0.01^2}\right\rceil\right) =\left\lceil 1824\right\rceil=1824.

The plan calls for 1,824 independently simulated trials. At the planning value, the resulting Monte Carlo standard error is \(\sqrt{0.1824/1824}=\sqrt{0.0001}=0.01\). This is a statement about simulation precision at \(p=0.24\), not about the uncertainty in the trial design itself or about whether the simulated scenario is realistic.

Planning an Approximate Confidence-Interval Width

A precision target can instead be expressed as the desired width of an approximate confidence interval. For a two-sided interval with nominal confidence level \(1-\alpha\), let \(z_{1-\alpha/2}\) be the corresponding standard normal quantile. If the interval is planned as estimate plus or minus \(z_{1-\alpha/2}\) times its standard error, its approximate full width is \(2z_{1-\alpha/2}\) times that standard error. If the target full width is \(w>0\), the corresponding target standard error is \(w/(2z_{1-\alpha/2})\).

For a simulated probability, substituting this standard-error target into the rate formula gives

R_{\min}=\max\left(1,\left\lceil \frac{z_{1-\alpha/2}^{\,2}p(1-p)}{(w/2)^2} \right\rceil\right).

This is a planning approximation, not a promise that the interval calculated after simulation will have exactly width \(w\). The achieved width depends on the estimated rate and the interval procedure used. The normal approximation is most dependable when the simulated event and non-event counts are both adequate; with very small expected counts, a normal-based width calculation may be a poor guide. The interval construction itself requires its own assumptions and method.

Worked Example: Planning a Rate Estimate by Interval Width

In a hypothetical design-assurance scenario, suppose the planning probability of a safety event is \(p=0.30\). The goal is an approximate two-sided 95% interval with full width no greater than \(0.06\). Use \(z_{0.975}=1.96\).

The target half-width is \(w/2=0.03\). The rate variance at the planning value is \(0.30(0.70)=0.21\), so

R_{\min}= \max\left(1,\left\lceil \frac{1.96^2(0.21)}{0.03^2} \right\rceil\right) = \left\lceil\frac{0.806736}{0.0009}\right\rceil = \lceil 896.373\ldots\rceil =897.

At \(R=897\), the planned standard error is \(\sqrt{0.21/897}\approx0.01530\), giving approximate half-width \(1.96(0.01530)\approx0.02999\) and full width about \(0.05998\). The rounded count meets the planned width. A smaller count may not: the unrounded requirement is about 896.373, so rounding down to 896 would not satisfy the inequality under this calculation.

Replication Counts for a Simulated Mean

The same planning logic applies when each simulated trial yields a numerical outcome \(Y_r\), such as duration in weeks or sample size in participants. Suppose the \(Y_r\) are independent and identically distributed with finite mean \(\mu_Y\) and variance \(\sigma_Y^2\). Their average estimates \(\mu_Y\), and the Monte Carlo standard error of that average is \(\sigma_Y/\sqrt{R}\). A target standard error \(\delta>0\) therefore leads to

R_{\min}=\max\left(1,\left\lceil\frac{\sigma_Y^2}{\delta^2}\right\rceil\right).

The result follows directly: \(\sigma_Y/\sqrt{R}\leq\delta\), with both sides nonnegative, implies \(\sigma_Y^2/R\leq\delta^2\), and hence \(R\geq\sigma_Y^2/\delta^2\). Taking the ceiling ensures the inequality holds for an integer replication count. As with the rate formula, the maximum with 1 prevents a zero count if \(\sigma_Y^2=0\). If the outcome is truly constant, one replicate suffices to reveal that value; the formula must not be interpreted as recommending zero simulations.

In applications, \(\sigma_Y\) is generally unknown. A planning value may come from prior simulations or a pilot run. If a pilot standard deviation \(s\) is used, replacing \(\sigma_Y\) by \(s\) is a planning approximation: it does not account for uncertainty in \(s\), and the final standard error can differ. A larger planning variance is more cautious because the required count increases in direct proportion to the variance.

Worked Example: Target Precision for Mean Trial Duration

Suppose a hypothetical simulation records total trial duration in weeks. A pilot suggests a replicate-level standard deviation of 12 weeks. Plan both for a Monte Carlo standard error no greater than 0.5 weeks and, separately, for an approximate 95% interval with full width no greater than 4 weeks. Treat 12 weeks as the planning standard deviation.

For the standard-error target, the planned variance is \(12^2=144\) weeks squared. Therefore,

R_{\min}=\max\left(1,\left\lceil\frac{144}{0.5^2}\right\rceil\right) =\left\lceil\frac{144}{0.25}\right\rceil =576.

For the approximate 95% interval-width target, the target half-width is 2 weeks. Using \(z_{0.975}=1.96\), the required count is

R_{\min}= \max\left(1,\left\lceil \frac{1.96^2(144)}{2^2} \right\rceil\right) = \left\lceil 138.2976\right\rceil =139.

The two answers differ because the precision targets differ: standard error at most 0.5 weeks implies a nominal normal half-width of \(1.96(0.5)=0.98\) weeks, whereas the separate interval target allows a 2-week half-width. These counts are conditional on the planning standard deviation and the independent-replicate assumption. The 139-run plan does not guarantee that an interval based on an estimated standard deviation will have width at most 4 weeks.

Choosing Planning Values and Checking Assumptions

When \(p\) is unknown, the variance \(p(1-p)\) is at most \(0.25\), because \(p(1-p)=0.25-(p-0.5)^2\). Thus, planning with \(p=0.5\) gives a conservative count for an absolute standard-error target across all Bernoulli probabilities:

R_{\min}=\max\left(1,\left\lceil\frac{0.25}{\delta^2}\right\rceil\right).

This guarantee concerns the variance-based standard error, not the realism of the simulated scenario. It may require many more replications than a defensible scenario-specific planning value when the event is uncommon. Conversely, using a small pilot estimate without caution can understate the needed count.

Before relying on a calculated replication count, check that:

Each replicate corresponds to the same specified scenario and executable design rules, so the target probability or mean is unchanged across replicates.
Replicates are independent for the variance formulas used here. Shared random draws or other dependence can change the Monte Carlo standard error.
The event indicator is defined consistently, or the numerical outcome is measured on the same scale and in the same units in every replicate.
The planning probability or variance is plausible for the scenario, and the chosen precision target is scientifically meaningful.
Any normal-based interval-width calculation is treated as approximate rather than as a guaranteed realized width.

A useful diagnostic is to compare planning assumptions with pilot output. If the observed event rate or outcome variability is much larger than anticipated, the planned count may not deliver the intended precision. A pilot with no observed events does not show that the event probability is zero; it supplies little information for estimating a rare probability. Likewise, a pilot with identical outcomes does not establish zero variability. Revisit the planning values, and report their role clearly rather than presenting the planned precision as a certainty.

Common Pitfalls and Practical Comparisons

Monte Carlo standard error and confidence-interval width are related but distinct planning targets. A standard-error target specifies the estimated variability of the Monte Carlo estimate. A width target additionally specifies a confidence level and uses a multiplier, such as 1.96 for a two-sided 95% normal approximation. For the same underlying variability, a narrower requested width requires more replicates; halving the target standard error or half-width multiplies the required count by four.

The formulas are convenient for planning, but they are not substitutes for checking the simulation. Counts based on a planning probability or pilot variance are only as relevant as those inputs. Also, the formulas presume independent, identically distributed replicates. When the simulation design induces dependence across replicates, the ordinary \(1/\sqrt{R}\) calculation need not apply. The appropriate response is not to insert a larger \(R\) into an invalid formula, but to use a precision calculation that reflects the actual dependence structure.

Wrong reasoning: “The pilot had no events, so \(p=0\), and the formula says no replicates are needed.”
Correction: A zero-event pilot does not establish \(p=0\), and a replication count must be positive. Use a defensible nonzero planning probability or the conservative Bernoulli variance bound when appropriate.
Wrong reasoning: “The pilot outcomes were all identical, so the required count for a mean is zero.”
Correction: A zero variance estimate from a limited pilot may be misleading; even if the true variance is zero, the planned count is at least one. Choose a justified planning variance and assess its uncertainty.

Key takeaway: Set a precision target, supply a justified planning probability or variance, solve the standard-error inequality for \(R\), and round up to a positive integer. State the assumptions and units, and treat the resulting count as a plan rather than a guarantee.

Check Your Understanding

Work through each question, then compare your reasoning with the solutions.

A simulated event has planning probability \(p=0.40\). How many independent replicates are required for a Monte Carlo standard error no greater than \(0.02\)?
For an unknown event probability, what conservative replication count gives an approximate 95% interval full width of at most \(0.04\), using the Bernoulli variance bound?
A simulated mean has planning standard deviation 8 participants. Find the replication count for a Monte Carlo standard error no greater than 0.25 participants.
A pilot simulation observes zero events. Explain why setting \(p=0\) is not a sound basis for concluding that no replicates are needed. Give a conservative count for standard error at most \(0.01\).

Solutions. (1) The variance is \(0.40(0.60)=0.24\), so \(R_{\min}=\max(1,\lceil0.24/0.02^2\rceil)=\lceil600\rceil=600\). (2) The half-width is \(0.02\), and the bound is \(p(1-p)\leq0.25\). Thus \(R_{\min}=\lceil1.96^2(0.25)/0.02^2\rceil=\lceil2401\rceil=2401\). (3) The variance is \(8^2=64\) participants squared, so \(R_{\min}=\lceil64/0.25^2\rceil=\lceil1024\rceil=1024\). (4) Zero observed events do not prove the event probability is zero; the pilot may simply be too small to observe a rare event. Using the conservative variance bound gives \(R_{\min}=\max(1,\lceil0.25/0.01^2\rceil)=2500\) independent replicates.