Monte Carlo Error for Assurance Estimates

Why Quantify Monte Carlo Error?

Assurance is the prior-averaged probability that a trial will satisfy a specified success rule. When it is estimated by simulation, the estimate varies from one set of simulation draws to another, even when the design, priors, outcome model, and decision rule are held fixed. Monte Carlo error measures this variation. It answers a practical question: how precisely has the simulation calculated assurance for the specified setup?

This error is distinct from uncertainty about whether the model or design assumptions are appropriate. A very large simulation can estimate assurance precisely under the specified assumptions while leaving uncertainty about those assumptions unchanged. The confidence intervals in this tutorial quantify simulation error only.

Assurance as a Mean of Success Indicators

For each simulation replication, draw a parameter value from the design prior, generate trial data from the trial model, and apply the specified Bayesian decision rule. Record whether that simulated trial meets the success criterion. Let \(Y_i=1\) if replication \(i\) succeeds and \(Y_i=0\) otherwise. If the assurance for the fixed design and decision rule is \(A\), then \(P(Y_i=1)=A\).

Definition: With \(M\) independent, identically distributed trial simulations, the Monte Carlo estimate of assurance is \(\hat{A}=M^{-1}\sum_{i=1}^{M}Y_i\). Under the stated simulation setup, \(Y_i\sim\text{Bernoulli}(A)\), so \(E(\hat{A})=A\) and \(\operatorname{Var}(\hat{A})=A(1-A)/M\).

The assumptions behind this calculation are important: each replication must represent the same design and decision rule, and the replication outcomes must be independent draws from the specified simulation procedure. In particular, \(M\) counts independently generated trial replications, not the number of patients in one trial. If the simulation procedure changes across replications, this simple binomial calculation need not apply.

Deriving the Monte Carlo Standard Error

The sample proportion of successful replications estimates the unknown success probability \(A\). Because a Bernoulli variable has variance \(A(1-A)\), independence gives the variance of the average directly:

\operatorname{Var}(\hat{A})=\operatorname{Var}\left(\frac{1}{M}\sum_{i=1}^{M}Y_i\right) =\frac{1}{M^2}\sum_{i=1}^{M}\operatorname{Var}(Y_i) =\frac{A(1-A)}{M}.

The Monte Carlo standard error (MCSE) is the standard deviation of the estimator’s simulation-to-simulation variation. Since \(A\) is unknown, estimate it by \(\hat{A}\):

\widehat{\operatorname{MCSE}}(\hat{A})= \sqrt{\frac{\hat{A}(1-\hat{A})}{M}}.

This is a standard error in probability units. For example, an MCSE of 0.01 corresponds to about one percentage point. It is not the standard deviation of patient outcomes, nor does it describe uncertainty in an individual patient’s treatment response.

Worked Example: MCSE and a 95% Interval

Suppose a hypothetical assurance simulation uses \(M=2{,}000\) independent trial replications, of which 1,240 satisfy the prespecified success rule. Calculate the estimated assurance, its MCSE, and an approximate 95% Monte Carlo confidence interval.

The estimated assurance is \(\hat{A}=1{,}240/2{,}000=0.6200\). Substituting into the estimated MCSE gives

\widehat{\operatorname{MCSE}}(\hat{A}) =\sqrt{\frac{0.6200(1-0.6200)}{2{,}000}} =\sqrt{0.0001178} =0.01085.

Using the normal approximation, a 95% interval is \(\hat{A}\pm1.96\,\widehat{\operatorname{MCSE}}(\hat{A})\). Thus the margin is \(1.96(0.01085)=0.02127\), giving

0.6200\pm0.02127=(0.5987,\ 0.6413).

For the specified simulation setup, the estimated assurance is 62.0%, with an approximate 95% Monte Carlo confidence interval from 59.9% to 64.1%. The interval expresses repeated-simulation uncertainty in estimating \(A\); it is not a 95% probability statement about the fixed assurance value.

Confidence Intervals and Their Conditions

A normal-approximation, often called Wald, interval is \(\hat{A}\pm z_{1-\alpha/2}\widehat{\operatorname{MCSE}}(\hat{A})\), where \(z_{1-\alpha/2}\) is the standard normal quantile. Its approximation is most reliable when the simulation includes enough successes and failures for the distribution of the sample proportion to be reasonably close to normal. A common diagnostic is to check that both \(M\hat{A}\) and \(M(1-\hat{A})\) are not small; this is a practical guide, not a guarantee of coverage. The Wald interval can perform poorly near 0 or 1 and can even extend beyond the possible range \([0,1]\).

A Wilson interval is often preferable when the estimate is near a boundary or the number of replications is modest. For a two-sided confidence level \(1-\alpha\), let \(z=z_{1-\alpha/2}\). The Wilson interval has center and half-width

\frac{\hat{A}+z^2/(2M)}{1+z^2/M} \quad\text{and}\quad \frac{z}{1+z^2/M} \sqrt{\frac{\hat{A}(1-\hat{A})}{M}+\frac{z^2}{4M^2}},

respectively. The interval is the center plus or minus the half-width. An exact binomial interval is another option, especially for very few successes or failures. These methods address the same binomial Monte Carlo sampling problem but can behave differently in small or boundary cases. For routine simulations with ample successes and failures, the Wald interval is often an adequate summary; near a boundary, report a Wilson or exact interval instead.

Worked Example: Planning Replications for a Target Precision

A design team expects assurance to be about 0.80 and wants an approximate 95% Monte Carlo confidence interval with a half-width no greater than 0.01. How many independent trial simulations should it plan?

For a target half-width \(d\), the normal-approximation formula is \(d\approx z_{0.975}\sqrt{A(1-A)/M}\). Solving for \(M\), and substituting the expected assurance and \(z_{0.975}=1.96\), gives

M\geq\frac{1.96^2(0.80)(0.20)}{0.01^2} =\frac{3.8416(0.16)}{0.0001} =6{,}146.56.

The team should round up and plan at least 6,147 replications. This is a planning approximation: the achieved interval width depends on the observed estimate, and the normal approximation should be checked after simulation. If the assurance is not known well enough to use 0.80 for planning, the variance \(A(1-A)\) is largest at \(A=0.50\). Using that conservative value gives \(M\geq3.8416(0.25)/0.0001=9{,}604\) replications for the same approximate half-width.

Boundary Cases and Simulation Diagnostics

If no simulated trials succeed, the plug-in MCSE is zero because \(\hat{A}=0\). That does not establish that the true assurance is zero: it means only that no success was seen in the finite set of replications. Similarly, observing success in every replication does not establish assurance of exactly one. In either case, an interval that accounts for the binomial boundary is needed.

Worked Example: No Successes in 100 Replications

In a hypothetical simulation of \(M=100\) independent trial replications, suppose none meets the success criterion. Then \(\hat{A}=0\). The plug-in MCSE is \(\sqrt{0(1-0)/100}=0\), which would give a degenerate Wald interval. That interval is misleading because the simulation has limited information about a small, nonzero assurance.

For \(x=0\) successes, the upper endpoint of a two-sided 95% exact binomial interval is \(1-(0.025)^{1/M}\). Here it is

1-(0.025)^{1/100}=1-\exp\left(\frac{\ln(0.025)}{100}\right) =1-\exp(-0.03689) \approx0.03622.

The exact two-sided interval is approximately \((0,\ 0.0362)\). Thus these 100 replications do not rule out assurance as high as about 3.62% at this confidence level. The result calls for more simulation if a precise upper bound matters; it does not justify reporting zero assurance.

Several diagnostics help identify trouble. Check the number of successes and failures, examine whether the interval width meets the planned precision, and confirm that the recorded number of replications matches the intended \(M\). Replication outcomes that are unintentionally dependent can make the binomial MCSE too small. Also verify that all replications use the same model, design prior, analysis procedure, and decision threshold. Repeating the simulation with a different random-number seed can reveal substantial simulation-to-simulation variation, but apparent stability across a few runs is not a substitute for a calculated MCSE or interval.

Repeatedly checking an ordinary fixed-sample confidence interval and stopping as soon as it is narrow enough can alter its coverage. If simulation is monitored with an unplanned stopping rule, the usual fixed-\(M\) interval may not retain its nominal coverage. A simple solution is to set \(M\) in advance using a precision target and then report the interval at that replication count.

What Monte Carlo Precision Does—and Does Not—Show

An MCSE or confidence interval quantifies the randomness introduced by approximating assurance with a finite number of simulated trials. It does not measure uncertainty from the design prior, the assumed data-generating model, choices about the success criterion, or implementation errors. Those are separate considerations. A narrow interval can therefore accompany an assurance estimate that is sensitive to assumptions.

When an exact calculation or deterministic numerical integration is available for the specified model, it can avoid this particular simulation error. For models where simulation is used, increasing \(M\) improves precision at a square-root rate: multiplying the replication count by four approximately halves the MCSE, provided the simulation remains independent and otherwise unchanged. It does not make the model more realistic or eliminate other sources of uncertainty.

Key takeaway: For independent simulated trial outcomes, the assurance estimate is a sample proportion and its estimated MCSE is \(\sqrt{\hat{A}(1-\hat{A})/M}\). Plan \(M\) around a stated precision goal, choose an interval suitable for the observed number of successes and failures, and interpret the result as simulation error under the specified assumptions.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A simulation has 500 successful replications out of 1,000. What are \(\hat{A}\) and its estimated MCSE?
Approximately how many replications are needed for a 95% interval with half-width 0.02 if assurance is expected to be 0.50?
Why is a zero plug-in MCSE inappropriate when no successes are observed?
Does a narrow Monte Carlo confidence interval show that the design prior is appropriate? Explain.

Solutions. (1) \(\hat{A}=500/1{,}000=0.50\), and the MCSE is \(\sqrt{0.50(0.50)/1{,}000}=\sqrt{0.00025}=0.01581\). (2) Using \(M\geq1.96^2(0.50)(0.50)/0.02^2\) gives \(M\geq2{,}401\). (3) The plug-in formula substitutes \(\hat{A}=0\), but observing no successes in finitely many trials does not establish that the underlying assurance is zero; use an exact or other boundary-appropriate interval. (4) No. The interval measures finite-simulation error conditional on the specified assumptions, not whether the design prior or other assumptions are appropriate.