Monte Carlo Error Versus Design Uncertainty

Two Different Sources of Uncertainty

A simulation can estimate an operating characteristic very precisely and still leave an important design question unresolved. The precision of the estimate concerns how much it would vary if the same simulation were repeated with a new set of random numbers. The unresolved question may instead be whether the simulation’s assumed trial conditions are close to the conditions that will actually prevail. These are different uncertainties, and increasing the number of simulated trials addresses only the first.

For example, a simulated probability of trial success depends on assumptions about the treatment effect, outcome variability, recruitment, and trial conduct. Holding those inputs fixed defines one simulated setting. Monte Carlo error describes the finite-run uncertainty in estimating performance in that setting; uncertainty about the inputs concerns whether that setting represents the trial of interest. The introductory material on Monte Carlo standard errors gives the basic precision calculations; the emphasis here is on interpreting their scope.

Definition: Let \(\theta\) denote the set of conditions specified for a simulated trial, and let \(\psi(\theta)\) be an operating characteristic under those conditions, such as the probability that the design declares success. Monte Carlo error is the random error from estimating \(\psi(\theta)\) with a finite number of simulation replicates, conditional on \(\theta\). Design uncertainty is uncertainty about whether the specified \(\theta\), or the resulting \(\psi(\theta)\), adequately represents the trial conditions relevant to the design decision.

The distinction is conditional. A simulation answers a question of the form, “What is the operating characteristic if these specified conditions hold?” It does not, solely by running more replicates, answer, “How likely are these conditions to hold?” Nor does a narrow Monte Carlo interval establish that the assumed model or inputs are realistic.

Monte Carlo Error Conditional on Fixed Conditions

Suppose each replicate simulates one complete trial under the same fixed conditions \(\theta\). Define \(X_i=1\) if replicate \(i\) meets a specified success criterion and \(X_i=0\) otherwise. If the simulated trials are independent and follow the same protocol and probability model, then \(X_i\) are independent Bernoulli random variables with \(P(X_i=1\mid\theta)=\psi(\theta)\). For \(M\) replicates, the estimated success probability is \(\hat{\psi}_M=M^{-1}\sum_{i=1}^{M}X_i\).

E(\hat{\psi}_M\mid\theta)=\psi(\theta), \qquad \operatorname{Var}(\hat{\psi}_M\mid\theta) =\frac{\psi(\theta)\{1-\psi(\theta)\}}{M}.

Because \(\psi(\theta)\) is not usually known, the Monte Carlo standard error is estimated by substituting \(\hat{\psi}_M\):

\widehat{SE}_{MC} = \sqrt{\frac{\hat{\psi}_M(1-\hat{\psi}_M)}{M}}.

This estimate measures finite-run variability for the specified conditions. For a sufficiently large \(M\), a normal-approximation interval is \(\hat{\psi}_M\pm1.96\widehat{SE}_{MC}\). It is an interval for the conditional operating characteristic under the simulation model, with repeated simulation runs as the source of variation. It does not cover uncertainty about whether that model or its inputs describe the intended trial.

Worked Example: Precision of a Conditional Success-Rate Estimate

In a hypothetical simulation, 9,240 of \(M=12{,}000\) independently generated trials meet the prespecified success criterion under one fixed set of trial assumptions. Estimate the conditional success probability and its Monte Carlo precision.

The simulated rate is \(\hat{\psi}_M=9{,}240/12{,}000=0.7700\). Using the estimated rate in the standard-error formula gives

\widehat{SE}_{MC} = \sqrt{\frac{0.7700(1-0.7700)}{12{,}000}} = \sqrt{\frac{0.1771}{12{,}000}} \approx 0.003842.

The approximate 95% Monte Carlo interval is \(0.7700\pm1.96(0.003842)=0.7700\pm0.007530\), or approximately \(0.7625\) to \(0.7775\). Under repeated independent simulations using these same assumptions, this interval procedure has approximately 95% coverage for the conditional success probability. The result does not establish that the actual trial’s success probability lies in this interval: that broader claim would require accounting for uncertainty about the conditions themselves.

Why More Replicates Do Not Resolve Input Uncertainty

The formula for Monte Carlo standard error makes its scope explicit: increasing \(M\) reduces the conditional standard error at the rate \(1/\sqrt{M}\), provided the simulation model and inputs stay fixed. For example, multiplying \(M\) by 100 divides the Monte Carlo standard error by 10. But the same operation does not change \(\theta\), validate it, or establish that it represents future trial conditions. The simulation may become more certain about the answer to its conditional question while the relevance of that answer remains uncertain.

A useful formal distinction is to treat the specified conditions \(\Theta\) as uncertain from the design team’s perspective. The trial conditions are fixed in reality, but the analyst may be uncertain about their values and may describe that uncertainty with a probability distribution or a set of plausible values. For a given \(\Theta\), \(\hat{\psi}_M\) varies because of finite simulation. Across possible values of \(\Theta\), the conditional performance \(\psi(\Theta)\) can also vary. The law of total variance separates these sources:

\operatorname{Var}(\hat{\psi}_M) = E\!\left\{\operatorname{Var}(\hat{\psi}_M\mid\Theta)\right\} + \operatorname{Var}\!\left\{E(\hat{\psi}_M\mid\Theta)\right\} = E\!\left\{\frac{\psi(\Theta)\{1-\psi(\Theta)\}}{M}\right\} + \operatorname{Var}\{\psi(\Theta)\}.

The first term is average conditional Monte Carlo variance; it decreases as \(M\) increases. The second term is variation in the conditional operating characteristic across uncertain conditions. More replicates do not reduce that term. If no defensible probability distribution for \(\Theta\) is available, the same distinction can be made without assigning probabilities: report how the operating characteristic differs across specified plausible assumptions rather than pretending that a single Monte Carlo interval captures that uncertainty.

Fix the simulation conditions.
Specify the trial assumptions and the operating characteristic being estimated. A conditional estimate has meaning only relative to these choices.

Quantify finite-run error.
Use the Monte Carlo standard error or an appropriate interval to describe uncertainty due to the finite number of replicates.

Assess relevance separately.
Ask whether the assumed conditions are credible for the intended trial and whether plausible alternatives could change the design conclusion.

Worked Comparisons Across Assumptions

Worked Example: A Large Difference Between Assumptions

Suppose two hypothetical simulations each use 12,000 independent replicates. Under assumption set A, 9,240 trials succeed, giving \(0.7700\). Under a different, plausibly relevant assumption set B, 8,160 succeed, giving \(0.6800\). Compare the estimates and their Monte Carlo errors.

For set A, the estimated standard error is \(\sqrt{0.7700(0.2300)/12{,}000}\approx0.003842\). For set B, it is

\sqrt{\frac{0.6800(0.3200)}{12{,}000}} = \sqrt{\frac{0.2176}{12{,}000}} \approx 0.004258.

The estimated conditional success rates differ by \(0.7700-0.6800=0.0900\), or 9.00 percentage points. Each estimate has a Monte Carlo standard error of about 0.4 percentage points, so the difference is much larger than the finite-run noise in either estimate. If the two simulation runs are independent, the estimated standard error of their difference is \(\sqrt{0.003842^2+0.004258^2}\approx0.005735\). The important finding is not that one estimate is imprecise; both are reasonably precise conditional estimates. Rather, performance changes materially with the assumptions. Increasing the replicates would clarify each conditional rate, but would not determine which assumptions better represent the intended trial.

Worked Example: Separating the Two Components of Uncertainty

For illustration, suppose an analyst assigns equal probabilities to two uncertain trial conditions. The conditional success probability is \(0.70\) under condition 1 and \(0.82\) under condition 2. Assume that, under either condition, \(M=10{,}000\) independent replicates are run. Calculate the mean conditional success probability, the between-condition variation, and the average finite-run variance.

The probability-weighted mean conditional success probability is \((0.70+0.82)/2=0.76\). The variance of the conditional probabilities is

\operatorname{Var}\{\psi(\Theta)\} = \frac{(0.70-0.76)^2+(0.82-0.76)^2}{2} = \frac{0.0036+0.0036}{2} = 0.0036.

The average conditional Monte Carlo variance is

E\!\left\{\frac{\psi(\Theta)\{1-\psi(\Theta)\}}{M}\right\} = \frac{1}{2} \left\{ \frac{0.70(0.30)}{10{,}000} + \frac{0.82(0.18)}{10{,}000} \right\} = 0.00001788.

Thus, the variance of the estimate when uncertainty about the condition is also represented is \(0.0036+0.00001788=0.00361788\), with standard deviation approximately \(0.06015\). The between-condition standard deviation alone is \(0.06000\), much larger than the average conditional Monte Carlo standard error. If \(M\) were increased to 1,000,000 in each condition, the average Monte Carlo variance would fall to \(0.0000001788\), while the between-condition variance would remain \(0.0036\). The equal probabilities here are an explicit illustrative assumption, not a general rule for weighting conditions; without a justified weighting distribution, the conditional results should be presented separately.

Diagnostics, Alternatives, and Limitations

A Monte Carlo diagnostic asks whether finite-run noise is small enough for the intended numerical decision. It may involve comparing the standard error with a prespecified precision target or checking whether uncertainty from simulation could change a threshold-based conclusion. Design uncertainty calls for a different diagnostic: examine the source and credibility of the inputs, identify which conclusions depend on them, and compare performance under relevant alternative assumptions. A stable estimate from repeated batches of simulation addresses numerical stability, not the credibility of the conditions supplied to the code.

These uncertainties should also be distinguished from the sampling variability of the actual clinical trial. That trial-to-trial variability is part of the operating characteristic being estimated: each simulated replicate represents a possible trial outcome under the specified model. Monte Carlo error concerns how accurately a finite set of those simulated outcomes estimates the operating characteristic. Design uncertainty concerns the conditions under which the operating characteristic is being evaluated.

Question	Source of uncertainty	What can address it?
Would another finite simulation run give a different estimated rate?	Monte Carlo error	More replicates, when needed for the required precision
Would performance differ if the assumed trial conditions were wrong?	Design uncertainty	Evidence about inputs and evaluation under relevant alternatives
Would a future trial produce a different outcome under fixed conditions?	Trial-to-trial variability	Represented in the simulated trial outcomes; it is not removed by more simulation replicates

There are limits to both kinds of assessment. A Monte Carlo interval relies on a suitable simulation procedure and, for the simple formula above, independent Bernoulli replicates with a common success probability conditional on fixed conditions. Dependence between replicates, changes in the simulated procedure across runs, or an incorrectly calculated success indicator can make the standard error misleading. Likewise, evaluating a few alternative inputs cannot rule out uncertainty from assumptions that were never considered. Neither a small Monte Carlo error nor a collection of conditional results proves that the simulation model is correct.

Common Mistake

Wrong reasoning: “The Monte Carlo interval is narrow, so the design’s success probability is known precisely.” This confuses precision conditional on the assumed inputs with confidence that those inputs describe the intended trial. A narrow interval can coexist with substantial variation in performance across plausible assumptions. The correction is to state the conditions attached to each estimate, report its Monte Carlo precision, and separately assess how much the design conclusion depends on uncertain conditions. Increasing replicates is appropriate for excessive Monte Carlo error; it is not a remedy for uncertain or poorly supported inputs.

Key takeaway: More simulation replicates reduce finite-run Monte Carlo error at a fixed set of assumptions. They do not reduce uncertainty about those assumptions, validate the simulation model, or establish that a conditional operating characteristic applies to the intended trial.

Check Your Understanding

Answer each question, then compare with the brief solutions.

A simulated success rate is \(0.50\) from 10,000 independent replicates under fixed conditions. What is its estimated Monte Carlo standard error?
If the replicate count increases fourfold while conditions stay fixed, by what factor does the approximate Monte Carlo standard error change?
A team repeats a simulation many times and obtains stable estimates, but the treatment effect input has weak supporting evidence. Which uncertainty has been assessed more directly, and which remains?
Why is trial-to-trial variability not the same as Monte Carlo error?

Solutions. 1. \(\sqrt{0.50(0.50)/10{,}000}=\sqrt{0.000025}=0.005000\), or 0.50 percentage points. 2. The standard error is proportional to \(1/\sqrt{M}\), so multiplying \(M\) by four halves the standard error. 3. The repeated runs assess Monte Carlo stability; uncertainty about the treatment effect input remains. 4. Trial-to-trial variability is represented by variation among simulated trial outcomes under fixed conditions and contributes to the operating characteristic. Monte Carlo error is uncertainty in estimating that characteristic from a finite number of those outcomes.