Monte Carlo Error for Means and Quantiles

Why Simulation Error Matters for Means and Quantiles

A simulation study approximates a design property by generating and analyzing many trial replicates. Even when the design and data-generating scenario are fixed, the estimated mean sample size or the 90th percentile of trial duration can change if the simulation is rerun with a different random seed. That variation is Monte Carlo error: uncertainty caused by using a finite number of simulated replicates to estimate a quantity defined by the simulation scenario.

Monte Carlo error is distinct from variation among trials. For example, trial durations may vary widely from one simulated trial to another, while the estimated mean duration can still be precise if enough replicates are run. Conversely, a tail quantile may be imprecise even when the simulation produces thousands of replicates, because relatively few replicates determine that tail.

The earlier material on Confidence Intervals for Operating Characteristics develops intervals for simulated rates. Here the target is a mean or a quantile of a continuous outcome, such as final sample size, trial duration, or an estimated treatment effect.

Monte Carlo Error for a Mean

This standard error estimates the standard deviation of the estimated mean across repeated simulation runs of size \(R\). Its units are the same as those of \(X\). For example, if \(X\) is final sample size, the Monte Carlo standard error is measured in participants. The formula assumes the replicate outcomes are independent and identically distributed under the scenario and that their variance is finite. A normal approximation for the sampling distribution of \(\bar{X}\) is generally reasonable for sufficiently large \(R\), though strong skewness or influential extreme outcomes can make the approximation poor.

When the normal approximation is suitable, an approximate 95% Monte Carlo interval for \(\mu_X\) is \(\bar{X}\pm1.96SE_{\mathrm{MC}}(\bar{X})\). It describes uncertainty in the simulated estimate of the scenario mean; it is not an interval containing 95% of individual trial outcomes.

Monte Carlo Error for a Quantile

Let \(F(x)=P(X\leq x)\) be the distribution function of a continuous simulated outcome. Its \(q\) quantile, written \(\xi_q\), is the value satisfying \(F(\xi_q)=q\), for \(0<q<1\), when \(F\) is continuous and the quantile is uniquely defined. The empirical estimate \(\hat{\xi}_q\) is the corresponding quantile of the \(R\) simulated outcomes.

Unlike a mean, a quantile's Monte Carlo error depends not only on the number of replicates but also on how concentrated the outcome distribution is near the target quantile. If the density \(f(\xi_q)\) is positive and sufficiently smooth near \(\xi_q\), then for large \(R\),

The density has units of inverse outcome units, so this standard error has the outcome's units. A small density near the quantile means that a modest change in cumulative probability corresponds to a larger change in outcome value, making the estimated quantile less precise.

Why the Quantile Formula Holds

The justification follows from the sampling variation of the empirical distribution function and a local approximation to \(F\). At the true quantile, the number of simulated outcomes at or below \(\xi_q\) has a binomial distribution with parameters \(R\) and \(q\). Therefore the empirical cumulative proportion at \(\xi_q\) has mean \(q\) and variance \(q(1-q)/R\). For large \(R\), its error is approximately normal.

Near \(\xi_q\), differentiability of \(F\) with positive density gives the local approximation \(F(\xi_q+\delta)\approx q+f(\xi_q)\delta\). Thus, to first order, an error in cumulative proportion is converted into an error in quantile by dividing by \(f(\xi_q)\). The resulting variance is approximately

which yields the stated standard error after taking the square root. This approximation requires independent replicates, a continuous distribution near the quantile, a positive and reasonably smooth density there, and enough simulations for the large-sample approximation to be useful. In practice, \(f(\xi_q)\) is unknown and must be estimated from the simulated outcomes.

Planning Replications for a Mean

The standard error formula can be rearranged to plan the number of simulations needed for a chosen precision. If the replicate standard deviation is expected to be about \(s_X\), a target approximate 95% margin of error \(h\) for a mean requires

The value of \(s_X\) is usually estimated from a pilot run or prior simulation experience, so the calculated replication count is a planning value rather than a guarantee. Rounding up is necessary, and a final precision assessment should use the standard deviation observed in the completed simulation.

Estimating and Checking Quantile Precision

A quantile standard error requires an estimate of the local density, not just the empirical quantile. One approach is to estimate the slope of the empirical distribution function near \(\hat{\xi}_q\). A local estimate based on a window of width \(w\), for example, is the fraction of simulated outcomes in that window divided by \(w\). More flexible density estimates can use nearby order statistics or a smoothed density curve. The choice of window or smoothing level matters: a narrow window can be noisy, while an overly wide one can conceal changes in density near the quantile.

For a normal-approximation interval, use the estimated density \(\hat f(\hat{\xi}_q)\) in the standard-error formula and calculate \(\hat{\xi}_q\pm1.96SE_{\mathrm{MC}}(\hat{\xi}_q)\). Interpret the result as an approximate interval for the scenario quantile, with the assumptions above in view. A density estimate close to zero, a tail quantile supported by very few observations, or a visibly irregular density near the target are warnings that this interval may be unreliable.

An alternative is an order-statistic interval. For a continuous distribution, the number of simulated outcomes below the true \(q\) quantile follows a \(\operatorname{Binomial}(R,q)\) distribution. This fact can be used to select lower and upper order-statistic indices with a desired coverage level, avoiding an explicit density estimate. Such intervals can be conservative because their coverage changes in discrete steps, and they can be wide when the quantile is in a tail. The density-based standard error is often a compact summary for large simulations; order-statistic intervals are a useful check when density estimation is questionable.

Diagnostics, Limitations, and a Common Pitfall

Before interpreting a Monte Carlo standard error, check that the simulated replicates are independent and generated under the same intended scenario. If outcomes are dependent, \(s_X/\sqrt{R}\) need not estimate the standard error correctly. Also inspect the outcome distribution: a mean can be sensitive to a long tail, while a quantile's estimated precision can be sensitive to local gaps, clustering, or an unstable density estimate. For a quantile, compare plausible density bandwidths or local windows; materially different standard errors indicate that the approximation needs caution.

The formulas quantify random simulation error, not errors in the protocol algorithm, the data-generating assumptions, or the chosen scenario. A very small Monte Carlo standard error cannot establish that the simulated model represents the clinical setting well. Nor does adding replicates remove bias from a coding error or an unsuitable model. Replication should be increased when the estimate is too noisy for its intended use, but first confirm that the simulation is producing the intended outcomes.

A second frequent error is to report the standard deviation of replicate outcomes as the Monte Carlo standard error of their mean. The standard deviation describes trial-to-trial spread; the estimated mean's Monte Carlo standard error is that standard deviation divided by \(\sqrt{R}\). For quantiles, reporting only the percentile and replicate count is not enough to assess precision: the local density also matters.