Choosing Replicate Counts for Precision

Why Replicate Count Is a Design Choice

A simulation estimate is itself subject to Monte Carlo variation. If too few simulated trials are run, an estimated familywise error rate, power, expected sample size, or trial duration may be too noisy to support a design decision. More replicates reduce that noise, but cost computation time. Choosing the replicate count is therefore a precision decision: decide how much simulation error is tolerable, then plan enough replicates to meet that target.

The earlier material on Monte Carlo standard errors for operating characteristics establishes how to quantify this variation. Here the direction is reversed: start with a desired standard error or approximate interval width, and solve for the number of replicates needed. The calculation is a planning approximation, not a guarantee that the realized estimate will have exactly the anticipated precision.

Replicate Counts for an Estimated Probability

Suppose each simulated trial independently either has a specified event or does not. The event might be at least one false rejection under a null scenario, or success for a particular treatment arm under a specified alternative. Let \(p\) be the probability of that event and let \(R\) be the number of simulated trials. For replicate \(r\), define \(X_r=1\) if the event occurs and \(X_r=0\) otherwise. The estimated operating characteristic is the event proportion, \(\hat{p}=\sum_{r=1}^{R}X_r/R\).

Definition: If the \(R\) simulated trials are independent and generated under the same scenario, then \(X_r\) are independent Bernoulli random variables with probability \(p\). The Monte Carlo standard error of \(\hat{p}\) is \(\sqrt{p(1-p)/R}\). For planning, if \(p\) is not known, use a plausible planning value or a conservative bound.

The formula follows from the variance of an average. Each indicator has variance \(p(1-p)\). Independence makes the variance of their sum \(R p(1-p)\), so dividing the sum by \(R\) gives

\operatorname{Var}(\hat{p})=\operatorname{Var}\left(\frac{1}{R}\sum_{r=1}^{R}X_r\right) =\frac{1}{R^2}\sum_{r=1}^{R}\operatorname{Var}(X_r) =\frac{p(1-p)}{R}.

Taking the square root gives the stated standard error. To plan for a target Monte Carlo standard error \(e\), require \(\sqrt{p(1-p)/R}\leq e\). Squaring both nonnegative sides and rearranging yields the minimum replicate count. Since \(R\) must be a whole number, round up.

R\geq\frac{p(1-p)}{e^2}.

Worked Example: Target Standard Error for Power

A hypothetical MAMS design is being evaluated under one alternative configuration. Suppose the planning value for the probability of declaring the target treatment effective is \(p=0.80\), and the desired Monte Carlo standard error is at most \(0.01\), or one percentage point.

Using the Bernoulli planning formula,

R\geq\frac{p(1-p)}{e^2} =\frac{(0.80)(0.20)}{(0.01)^2} =\frac{0.16}{0.0001}=1600.

Plan at least 1,600 independent simulated trials for this scenario. At the planning value, the standard error is \(\sqrt{0.16/1600}=0.01\). This is Monte Carlo uncertainty in the estimated power, not uncertainty in the power of a real clinical trial. If the simulated estimate is near 0.80, its estimated Monte Carlo standard error will be close to the planned value.

Planning From Approximate Interval Width

A precision target is often stated as a desired width for an approximate two-sided 95% interval for an event probability. Using the normal approximation, the half-width is approximately \(1.96\sqrt{p(1-p)/R}\); the full width is twice that amount. If the target full width is \(w\), solving for \(R\) gives

2(1.96)\sqrt{\frac{p(1-p)}{R}}\leq w \quad\Longrightarrow\quad R\geq\frac{4(1.96)^2p(1-p)}{w^2}.

This is a planning formula for an approximate interval, not a substitute for checking the interval procedure and its assumptions in a particular application. Its normal approximation is most dependable when the simulated event count and non-event count are both sufficiently large. The width target is in probability units: for instance, \(w=0.03\) means three percentage points from lower to upper endpoint, not a half-width of three percentage points.

Worked Example: Target Width for a Familywise Error Estimate

Suppose a hypothetical simulation estimates familywise error under a specified null configuration. Use planning value \(p=0.80\), a two-sided 95% normal-approximation interval, and target full width \(w=0.03\).

Substitute into the full-width formula, retaining the factor of 4 because \(w\) is the full width:

R\geq\frac{4(1.96)^2(0.80)(0.20)}{(0.03)^2} =\frac{2.458624}{0.0009} \approx2731.804.

Round up to 2,732 replicates. At that count, the approximate full width is \(2(1.96)\sqrt{0.16/2732}\), approximately 0.0300, or 3.00 percentage points. The estimated error rate describes the probability of the specified false-rejection event under the simulated scenario; it is not the probability that the null hypothesis is true.

If no reliable planning value for \(p\) is available, use \(p(1-p)\leq0.25\), which holds for every probability between zero and one. The conservative calculation is

R\geq\frac{4(1.96)^2(0.25)}{(0.03)^2} =\frac{3.8416}{0.0009} \approx4268.444.

Thus, at least 4,269 replicates are required under this conservative bound. This count protects against the largest Bernoulli variance, which occurs at \(p=0.50\); it can be larger than necessary when the true event probability is far from one-half.

Other Operating Characteristics and Several Targets

Not every operating characteristic is an event proportion. For example, a simulation may record the total sample size or trial duration in each replicate and estimate its mean. If those replicate-level values are independent with standard deviation \(\sigma_Y\), the Monte Carlo standard error of their mean is \(\sigma_Y/\sqrt{R}\). A target standard error \(e\) therefore implies \(R\geq(\sigma_Y/e)^2\). In practice, \(\sigma_Y\) is usually estimated from pilot simulations or a reasonable prior calculation, so allow for uncertainty in that estimate.

Worked Example: Precision for Expected Sample Size

In a hypothetical pilot simulation, total sample size has an estimated standard deviation of 100 participants across simulated trials. The design team wants the Monte Carlo standard error of the estimated expected sample size to be no more than 2 participants.

Using the pilot standard deviation as the planning value,

R\geq\left(\frac{\sigma_Y}{e}\right)^2 =\left(\frac{100\ \text{participants}}{2\ \text{participants}}\right)^2 =50^2=2500.

Plan at least 2,500 independent replicates. The units cancel in the ratio, as they should for a replicate count. Because the standard deviation came from a pilot, it may be imprecise; a safety margin or a second precision check after an initial batch can be sensible.

A MAMS simulation commonly reports several quantities for each scenario: one or more powers, an error rate, expected sample size, and perhaps expected duration. Set a precision target for every quantity that matters, calculate the corresponding replicate count, and use the largest count as the common plan if all estimates will use the same replicates. For example, a power target might require 1,600 replicates while a sample-size target requires 2,500; using 2,500 meets both planning targets, subject to their assumptions.

The estimates from a given simulated trial can be related—for example, a trial that stops early may have both a smaller sample size and a different chance of success. This dependence does not change the separate standard-error formulas for each marginal estimate, provided replicates are independent. It does matter if the goal is a joint guarantee that several estimates all meet their precision targets; that is a stronger requirement than meeting each approximate target separately.

Assumptions, Checks, and Alternatives

The calculations rely on the simulated replicates being independent draws from the same scenario and on the chosen variance or probability being a reasonable planning value. A replicate must represent one complete simulated trial under the specified design and data-generating configuration. If the simulation changes scenarios partway through, or if replicate outcomes are dependent, the simple formulas do not directly describe the Monte Carlo standard error.

After a run, compare the achieved precision with the target. For a simulated event rate, use the observed proportion to estimate its standard error as \(\sqrt{\hat{p}(1-\hat{p})/R}\); for an estimated mean, use the replicate standard deviation divided by \(\sqrt{R}\). Also check the event count and non-event count when relying on a normal approximation. An estimate based on very few event replicates can have poor normal-approximation behavior even if a formula using a guessed \(p\) suggested a modest \(R\). Rare-event planning needs special care rather than blind reliance on this approximation.

A fixed replicate count is simple and makes computational cost predictable, but a conservative count can waste computation. A pilot-based count can be more efficient, but a poor pilot estimate can understate the needed number. An alternative is to run batches and reassess estimated precision between batches. Such monitoring is useful operationally, but stopping based on the evolving results can affect the properties of the final estimate or interval; it should not be treated as identical to fixing \(R\) in advance.

The main limitation is that no replicate-count formula repairs a misspecified simulation model or incorrect decision code. Precision concerns how accurately the simulation estimates the behavior of the specified model; it does not establish that the model represents the intended trial. Nor does a narrow Monte Carlo interval eliminate statistical uncertainty about outcomes in an eventual clinical trial.

Common pitfall: When \(w\) denotes a full interval width, using \(1.96\sqrt{p(1-p)/R}\leq w\) omits the factor of 2 for the two half-widths. The correct full-width condition is \(2(1.96)\sqrt{p(1-p)/R}\leq w\), so the required count contains a factor of 4 after squaring. State clearly whether a target is a half-width, full width, or standard error before calculating.

Name the quantity and scenario.
Specify the operating characteristic and the MAMS configuration whose simulation precision matters.

Set the precision target.
Choose a maximum Monte Carlo standard error or an approximate full interval width, with units or probability scale stated.

Choose planning variability.
Use a plausible event probability or pilot variability; use \(p(1-p)=0.25\) when a conservative Bernoulli bound is appropriate.

Calculate, round up, and verify.
Compute the required count for each key quantity, select a count meeting all targets, then check achieved precision and approximation conditions.

Check Your Understanding

Try each problem first. Worked solutions follow so that the planning arithmetic and interpretation can be checked.

A simulated binary operating characteristic has planning value \(p=0.60\). How many independent replicates are needed for a Monte Carlo standard error no greater than 0.015?
For a probability estimate planned at \(p=0.50\), find the approximate number of replicates for a 95% normal-approximation interval with full width no greater than 0.04.
A replicate-level continuous outcome has pilot standard deviation 30 units. What replicate count targets a Monte Carlo standard error of 1.5 units for its mean?
Why is a count based on \(p=0.50\) conservative for a Bernoulli operating characteristic?

Solutions. (1) \(R\geq(0.60)(0.40)/(0.015)^2=0.24/0.000225=1066.67\), so plan 1,067 replicates. (2) \(R\geq4(1.96)^2(0.50)(0.50)/(0.04)^2=3.8416/0.0016=2401\), so plan 2,401 replicates. (3) \(R\geq(30/1.5)^2=20^2=400\), so plan 400 replicates. (4) The Bernoulli variance is \(p(1-p)\), which is at most 0.25 and reaches 0.25 at \(p=0.50\). Using that maximum plans for the largest possible standard error over event probabilities, although it may require more replicates than a well-supported probability-specific value.