Monte Carlo Precision for Expected Sample Size

Why Monte Carlo Precision Matters

A simulation estimates expected enrollment by repeatedly generating the trial outcomes and applying the planned MAMS stopping and selection rules. The resulting estimate is itself random: a new set of simulation runs would usually give a somewhat different average. Monte Carlo precision describes this run-to-run uncertainty in the simulated estimate. It does not describe how much enrollment varies from trial to trial, and it does not account for uncertainty in the assumed effects, variances, or other design inputs.

The earlier material on estimating expected enrollment by simulation defines the enrollment outcome for each simulated trial. Here, treat each completed simulation as one draw from the enrollment distribution under a fixed design and fixed scenario. The question is how accurately the average of a finite number of these draws estimates the scenario’s true expected enrollment.

The replicate outputs should be independent and generated under the same design and scenario. In practice, this means using independent simulation random streams and applying the same enrollment rules and inputs on every run. Enrollment is typically bounded by the design’s maximum enrollment, so its variance is finite. The outputs may be discrete or skewed, however, and those features matter when assessing the quality of a confidence interval.

Monte Carlo Standard Error and Confidence Interval

Let \(s_Y\) be the sample standard deviation of the \(M\) simulated enrollment values, calculated using denominator \(M-1\). The estimated Monte Carlo standard error of their mean is \(s_Y/\sqrt{M}\). It decreases in proportion to \(1/\sqrt{M}\): multiplying the number of simulations by four approximately halves the standard error, rather than reducing it by a factor of four.

A conventional two-sided \(100(1-\alpha)\%\) Monte Carlo confidence interval is

where \(t_{1-\alpha/2,\,M-1}\) is the corresponding quantile of the \(t\) distribution with \(M-1\) degrees of freedom. If the replicate outputs are normally distributed, this interval has exact coverage under the independent, identically distributed simulation model. For sufficiently large \(M\), it is commonly used as an approximate interval for the mean even when the enrollment distribution is not normal, by the central limit theorem. The interval quantifies Monte Carlo uncertainty about \(\theta\), not a probability that the fixed value \(\theta\) lies in this particular interval.

Choosing a Simulation Size for a Precision Goal

A precision goal is often stated as a maximum confidence-interval half-width \(h\), in participants. If a pilot simulation suggests that the enrollment standard deviation is about \(s_{\text{pilot}}\), a first calculation replaces the \(t\) critical value with 1.96 for a 95% interval and solves for \(M\). Since the standard error is approximately \(s_{\text{pilot}}/\sqrt{M}\), the normal-approximation requirement is

Round up to a whole replicate. This is a planning approximation, not a guarantee: the pilot standard deviation is estimated, and the finite-\(M\) \(t\) critical value is slightly larger than 1.96. A careful plan checks the achieved half-width using the \(t\) critical value at the proposed \(M\), and the completed simulation reports its actual \(s_Y\) and interval.

Diagnostics, Alternatives, and Limitations

Before relying on a Monte Carlo interval, check both the simulation process and the replicate-output distribution. Confirm that each run follows the intended enrollment rules, that no runs were accidentally duplicated, and that the design inputs are unchanged across replicates. Inspect a histogram or other summary of the simulated enrollments for strong skewness, unusual spikes, or a large mass at the maximum enrollment. Because enrollment is bounded and often discrete, nonnormality is not surprising; the key issue is whether the sampling distribution of the mean is adequately approximated at the chosen \(M\).

With substantial skewness or a small number of runs, the \(t\)-interval may be a poor approximation, despite the finite variance. Increasing \(M\) can improve the approximation, but it does not fix a coding error, dependence between runs, or a change in scenario inputs. A bootstrap interval computed by resampling the simulated outputs is one possible alternative for assessing uncertainty in their mean, although it too depends on the simulated sample being representative and may be unreliable when the effective number of distinct outputs is small. An analytic calculation, when available and applicable, avoids Monte Carlo error for that calculation; it does not remove uncertainty in the underlying design assumptions.

Avoid truncating a confidence interval merely because an endpoint falls outside the logically possible enrollment range. Such clipping can alter its coverage properties. Instead, investigate whether the approximation is appropriate and, if needed, use more simulations or a justified alternative method. Also distinguish precision planning from a guarantee: a chosen \(M\) based on pilot variability may miss the target if the final sample standard deviation is higher than expected.

The main alternatives answer different questions. Reporting only \(\bar{Y}\) gives a point estimate but hides its Monte Carlo precision. Reporting \(\bar{Y}\), \(s_Y\), \(M\), and a confidence interval makes both the estimated expected enrollment and its simulation uncertainty visible. Increasing \(M\) is the direct way to reduce Monte Carlo error for one fixed scenario. Comparing scenarios requires care about how their simulations are generated; that comparison is a separate issue from estimating the precision of one scenario’s mean.