Monte Carlo Precision for Re-Estimation Evaluation

Why Monte Carlo Precision Matters

A simulation estimate is itself variable: if the same design and scenario are simulated again with different random draws, the estimated power and average sample size will usually change. The simulation size should therefore be chosen to make this numerical variation small enough for the decisions the evaluation must support. A precise estimate does not guarantee that the assumed model, treatment effect, or re-estimation rule is appropriate; it means only that repeated simulation under those assumptions would give similar estimates.

In Evaluating Operating Characteristics by Simulation, each replicate applies the complete design and analysis to one generated trial. Here the focus is on the Monte Carlo precision of the resulting estimates. Suppose there are \(M\) independent simulated trials under one fixed scenario. For replicate \(m\), let \(R_m=1\) if the trial rejects \(H_0\), and \(R_m=0\) otherwise. Let \(N_m\) be the total sample size used in that replicate, and let \(C_m=1\) if the design reaches its sample-size cap and \(C_m=0\) otherwise.

Definition: The Monte Carlo standard error (MCSE) is the standard deviation, under repeated simulation, of an estimator calculated from \(M\) simulated replicates. It describes simulation error, not the sampling standard error of an estimate from a real trial.

The formulas below assume that simulated replicates are independent and generated under the same specified scenario, including the same data-generating model, re-estimation rule, and analysis procedure. The replicate must represent the full trial-level process: simulating only the final test statistic, for example, would not estimate sample-size behavior unless it also reproduces the rule determining \(N_m\).

Precision for Power and Cap Probability

Let \(\pi=P(R_m=1)\) be the design's power under the specified alternative and let \(\rho=P(C_m=1)\) be the probability of reaching the cap under that scenario. The corresponding simulation estimates are proportions of simulated trials:

\hat{\pi}=\frac{1}{M}\sum_{m=1}^{M}R_m,\qquad \hat{\rho}=\frac{1}{M}\sum_{m=1}^{M}C_m.

Each indicator is Bernoulli. The variance of the average of \(M\) independent Bernoulli variables with success probability \(q\) is \(q(1-q)/M\). Since the true probability is unknown, estimate the MCSE by substituting the simulated proportion:

\widehat{\operatorname{MCSE}}(\hat{\pi}) =\sqrt{\frac{\hat{\pi}(1-\hat{\pi})}{M}},\qquad \widehat{\operatorname{MCSE}}(\hat{\rho}) =\sqrt{\frac{\hat{\rho}(1-\hat{\rho})}{M}}.

For a sufficiently large \(M\), a rough 95% Monte Carlo interval is estimate \(\pm 1.96\) times its MCSE. This summarizes uncertainty due to the finite number of simulation replicates. It is not a confidence interval for a treatment effect, and it does not account for uncertainty in the assumed scenario.

Worked Example: Monte Carlo Error in Estimated Power

Suppose \(M=2{,}500\) independent hypothetical trials are simulated under a specified alternative, and 2,030 reject \(H_0\). The estimated power is

\hat{\pi}=\frac{2{,}030}{2{,}500}=0.812.

Its estimated MCSE is

\sqrt{\frac{0.812(1-0.812)}{2{,}500}} =\sqrt{\frac{0.152656}{2{,}500}} =\sqrt{0.0000610624} \approx 0.007814.

The approximate 95% Monte Carlo interval is \(0.812\pm1.96(0.007814)\), or approximately \((0.7967,0.8273)\). Thus, under the simulation assumptions, the estimated power is 81.2%, with an approximate Monte Carlo half-width of 1.53 percentage points. This interval reflects finite simulation size; it does not mean that the assumed power scenario itself has been validated.

Worked Example: Precision for Reaching the Sample-Size Cap

In 5,000 independent hypothetical simulated trials, suppose the design reaches its cap in 1,000 trials. Then \(\hat{\rho}=1{,}000/5{,}000=0.200\), and

\widehat{\operatorname{MCSE}}(\hat{\rho}) =\sqrt{\frac{0.200(0.800)}{5{,}000}} =\sqrt{0.000032} \approx 0.005657.

The approximate 95% Monte Carlo half-width is \(1.96(0.005657)\approx0.01109\), or about 1.11 percentage points. The simulation estimates that the cap is reached in 20.0% of trials; its Monte Carlo precision is substantially finer than, for example, a half-width of 3 percentage points.

Precision for Expected Sample Size

The expected total sample size under a scenario is \(\mu_N=E(N_m)\). Estimate it by the sample mean across simulated trials, and calculate the sample standard deviation of the simulated sample sizes using the usual sample standard deviation \(s\):

\bar{N}=\frac{1}{M}\sum_{m=1}^{M}N_m,\qquad s_N=\sqrt{\frac{\sum_{m=1}^{M}(N_m-\bar{N})^2}{M-1}},\qquad \widehat{\operatorname{MCSE}}(\bar{N})=\frac{s_N}{\sqrt{M}}.

This formula follows because independent replicate sample sizes have variance \(\sigma_N^2\), so their mean has variance \(\sigma_N^2/M\). Replacing the unknown \(\sigma_N\) with \(s_N\) estimates the standard error of \(\bar{N}\). Units matter: if \(N_m\) counts participants, both \(s_N\) and the MCSE are in participants.

Worked Example: Monte Carlo Error in Expected Sample Size

Suppose 1,000 hypothetical trials produce a mean total sample size of 286 participants and a sample standard deviation of 42 participants. The MCSE of the estimated mean is

\widehat{\operatorname{MCSE}}(\bar{N}) =\frac{42}{\sqrt{1{,}000}} =\frac{42}{31.6228} \approx1.328\text{ participants}.

An approximate 95% Monte Carlo interval is \(286\pm1.96(1.328)\), or approximately 283.4 to 288.6 participants. It describes the finite-simulation precision of the estimated expected sample size under this scenario. The interval does not assert that any individual trial will use a sample size in that range; individual trial sizes can vary much more, as the 42-participant standard deviation indicates.

Choosing the Number of Simulations

Choose \(M\) from the required precision, not from a universal rule of thumb. For an estimated probability, the approximate 95% Monte Carlo half-width is \(1.96\sqrt{q(1-q)/M}\). If the planning value is \(q\) and the desired half-width is \(d\), solving for \(M\) gives:

M\ \geq\ \frac{1.96^2q(1-q)}{d^2}.

This calculation targets a 95% half-width of \(d\), not an MCSE of \(d\): the latter omits the factor 1.96. If \(q\) is unknown, \(q(1-q)\leq0.25\) for every probability, so using 0.25 gives a conservative planning calculation for this approximate half-width.

Worked Example: Selecting a Replication Count for Power

Suppose an initial planning value for power is 0.812, and the goal is an approximate 95% Monte Carlo half-width no larger than 0.002. The required replication count is

M\ \geq\ \frac{1.96^2(0.812)(0.188)}{(0.002)^2} =\frac{0.5864432896}{0.000004} =146{,}610.8224.

Because \(M\) must be a whole number and must meet or exceed the bound, use at least 146,611 independent simulated trials. This is a demanding target: reducing the half-width by a factor of two requires approximately four times as many replicates, since the half-width is proportional to \(1/\sqrt{M}\). If no reliable planning value is available, the conservative calculation is \(M\geq1.96^2(0.25)/(0.002)^2=240{,}100\).

For expected sample size, if a planning standard deviation is \(s_N\) and the desired approximate 95% half-width is \(d_N\) participants, plan using \(M\geq(1.96s_N/d_N)^2\). For instance, \(s_N=42\) participants and \(d_N=2\) participants give \(M\geq(1.96(42)/2)^2=1{,}694.1456\), so at least 1,695 replicates are indicated. This is a planning approximation; a pilot simulation can provide a more relevant estimate of variability.

Checking Simulation Precision and Failure Modes

A useful report gives \(M\), the estimate, and its MCSE for every key operating characteristic, separately for each scenario. For example, report power as 0.812 (MCSE 0.0078), expected sample size as 286 participants (MCSE 1.33 participants), and cap probability as 0.200 (MCSE 0.0057). These values cannot be collapsed into one generic statement that the simulation was “large.”

Zero or nearly zero events. If no simulated trial reaches the cap, the plug-in formula gives an estimated MCSE of zero. That does not establish that the true cap probability is zero: the simulation may simply be too small to observe a rare event. Increase \(M\), report the event count, and consider a conservative planning probability or an appropriate rare-event interval rather than treating zero as certainty.
Replicates are not independent. The formulas above rely on independent trials. Reusing the same random outcome in a way that induces dependence, or computing an MCSE as if correlated replicates were independent, can understate Monte Carlo error. Use independent random draws for replicates or account for the dependence in the error calculation.
Precision differs across scenarios and outcomes. A simulation size adequate for expected sample size may not be adequate for a small cap probability or a closely estimated power difference. Set precision goals for the quantities that will inform the design decision.
Monte Carlo precision is not model certainty. A small MCSE cannot correct misspecified outcome distributions, an incorrect treatment effect, or a coding error in the re-estimation rule. Check implementation, inspect simulated sample-size distributions and event counts, and evaluate substantively plausible scenarios.

The main alternative to simulation is an analytical calculation. When a design and outcome model yield tractable formulas, those calculations can be faster and avoid Monte Carlo error. Re-estimation rules that make sample size depend on simulated interim data can make direct analytic evaluation difficult, which is why simulation is useful; its cost is finite-replicate error and dependence on correctly implemented assumptions. Increasing \(M\) reduces Monte Carlo error at the square-root rate, but does not remove those assumptions or other sources of uncertainty.

Practical Workflow

Define each target.
Specify the scenario and whether the target is power, expected total sample size, cap probability, or another operating characteristic.

Set a precision goal.
Choose an acceptable MCSE or approximate interval half-width in meaningful units, such as percentage points or participants.

Plan \(M\).
Use a plausible planning value or pilot estimate, and use a conservative probability variance when the event probability is uncertain.

Report and diagnose.
Give the replicate count, estimate, MCSE, and relevant event counts; investigate rare events, instability, and implementation errors.

Key takeaway: Monte Carlo precision is outcome-specific. For a probability estimate, its MCSE is approximately \(\sqrt{\hat{q}(1-\hat{q})/M}\); for an expected sample size, it is \(s_N/\sqrt{M}\). Select the simulation size to meet a stated precision goal, and never interpret small Monte Carlo error as evidence that the design assumptions are correct.

Check Your Understanding

Work through each item, then compare with the solutions below.

In 4,000 independent simulated trials, 2,000 reject \(H_0\). Find estimated power and its MCSE.
A simulation estimates a cap probability as 0.10 from 10,000 replicates. Find its approximate MCSE and 95% Monte Carlo half-width.
For a sample-size estimate with \(s_N=30\) participants, how many replicates are needed for an approximate 95% half-width of 1 participant?
Why is a plug-in MCSE of zero after observing no cap events not evidence that the cap probability is zero?

Solutions. (1) \(\hat{\pi}=2{,}000/4{,}000=0.500\), so MCSE \(=\sqrt{0.5(0.5)/4{,}000}=\sqrt{0.0000625}=0.007906\). (2) MCSE \(=\sqrt{0.10(0.90)/10{,}000}=0.003000\); the approximate half-width is \(1.96(0.003000)=0.005880\), or 0.588 percentage points. (3) \(M\geq(1.96(30)/1)^2=3{,}457.44\), so use at least 3,458 replicates. (4) The plug-in expression uses the observed proportion, which is zero when no events occur. A finite simulation can miss a rare event, so more simulation or a method appropriate for rare-event uncertainty is needed.