Monte Carlo Precision for Simulated Power

Why Simulated Power Needs a Precision Assessment

Simulation-based power is estimated by repeatedly generating a hypothetical study under a specified design and model, then recording whether each simulated study meets the prespecified success criterion. The proportion of successful simulations estimates the probability of success under those assumptions. But that proportion is itself random: a new set of simulation replicates will generally produce a different estimate.

Monte Carlo precision describes the variability caused by using a finite number of simulation replicates. It is distinct from uncertainty about model inputs, such as the assumed within-subject variability, dropout rate, or treatment effect. Increasing the number of replicates reduces Monte Carlo error; it does not resolve uncertainty or misspecification in those inputs.

Definition: Let \(M\) be the number of independently generated simulation replicates. For replicate \(j\), define \(I_j=1\) if the simulated study meets the specified success criterion and \(I_j=0\) otherwise. If each replicate has the same success probability \(p_{\mathrm{pow}}\), then the estimated simulated power is \(\hat p_{\mathrm{pow}}=K/M\), where \(K=\sum_{j=1}^{M} I_j\) is the number of successful replicates.

The success criterion must be fixed before counting successes. For example, a replicate might count as successful if the planned bioequivalence decision rule declares equivalence. The simulation procedure must apply that rule to each generated study in the same way. The Monte Carlo calculations below concern the probability of this defined event, not the correctness of the model or the appropriateness of the decision rule.

Monte Carlo Standard Error

Under independent, identically distributed replicates, each \(I_j\) is Bernoulli with success probability \(p_{\mathrm{pow}}\). Thus \(K\) has a binomial distribution with \(M\) trials and success probability \(p_{\mathrm{pow}}\). The expected value of \(\hat p_{\mathrm{pow}}\) is \(p_{\mathrm{pow}}\), and its variance is \(p_{\mathrm{pow}}(1-p_{\mathrm{pow}})/M\). Replacing the unknown probability by its estimate gives the estimated Monte Carlo standard error.

\operatorname{Var}(\hat p_{\mathrm{pow}})=\frac{p_{\mathrm{pow}}(1-p_{\mathrm{pow}})}{M}, \qquad SE_{\mathrm{MC}}(\hat p_{\mathrm{pow}}) = \sqrt{\frac{\hat p_{\mathrm{pow}}(1-\hat p_{\mathrm{pow}})}{M}}.

This standard error is on the probability scale. Multiplying a power estimate and its standard error by 100 expresses them in percentage points. It is not the standard error of a treatment-effect estimate within one simulated study.

Justification: Since the indicators are independent, \(\operatorname{Var}(K)=\sum_{j=1}^{M}\operatorname{Var}(I_j)=M p_{\mathrm{pow}}(1-p_{\mathrm{pow}})\). Because \(\hat p_{\mathrm{pow}}=K/M\), dividing \(K\) by \(M\) divides its variance by \(M^2\), giving \(\operatorname{Var}(\hat p_{\mathrm{pow}})=p_{\mathrm{pow}}(1-p_{\mathrm{pow}})/M\). The reported Monte Carlo standard error uses \(\hat p_{\mathrm{pow}}\) in place of the unknown \(p_{\mathrm{pow}}\).

Confidence Intervals for Simulated Power

A confidence interval adds a range of plausible values for the underlying success probability, conditional on the simulation setup. The Wilson interval is often preferable to the simple normal, or Wald, interval, especially when the estimated power is close to zero or one. For a two-sided confidence level \(1-\alpha\), let \(z_{1-\alpha/2}\) be the corresponding standard normal quantile. The Wilson interval is:

\frac{ \hat p_{\mathrm{pow}}+\dfrac{z_{1-\alpha/2}^{\,2}}{2M} }{ 1+\dfrac{z_{1-\alpha/2}^{\,2}}{M} } \ \mathbin{\pm}\ \frac{z_{1-\alpha/2}}{1+\dfrac{z_{1-\alpha/2}^{\,2}}{M}} \sqrt{ \frac{\hat p_{\mathrm{pow}}(1-\hat p_{\mathrm{pow}})}{M} +\frac{z_{1-\alpha/2}^{\,2}}{4M^2} }.

The interval follows by inverting the score-test inequality for a binomial probability: values of \(p_{\mathrm{pow}}\) satisfying \(\left|\hat p_{\mathrm{pow}}-p_{\mathrm{pow}}\right| \leq z_{1-\alpha/2}\sqrt{p_{\mathrm{pow}}(1-p_{\mathrm{pow}})/M}\) are retained. Squaring both sides gives a quadratic inequality in \(p_{\mathrm{pow}}\); its two roots are the Wilson endpoints. This construction avoids the impossible negative lower bound or above-one upper bound that a Wald interval can produce.

In repeated simulation runs under the same conditions, a 95% confidence procedure has approximately 95% coverage of the fixed \(p_{\mathrm{pow}}\). It does not mean there is a 95% probability that the parameter lies in this particular interval. An exact binomial interval is another standard option; it can be conservative, while the Wilson interval generally gives useful coverage without the Wald interval’s boundary problems.

Worked Example: Estimating Power and Its Monte Carlo Interval

In a hypothetical simulation, \(M=1{,}000\) independent studies are generated. The prespecified bioequivalence success criterion is met in \(K=720\) studies. Calculate the estimated power, its Monte Carlo standard error, and a 95% Wilson interval.

The estimate is \(\hat p_{\mathrm{pow}}=720/1{,}000=0.720\), or 72.0%. Its estimated Monte Carlo standard error is:

SE_{\mathrm{MC}} = \sqrt{\frac{0.720(1-0.720)}{1{,}000}} = \sqrt{0.0002016} = 0.01420.

Thus the standard error is about 1.42 percentage points. For a 95% interval, use \(z_{0.975}=1.96\). The Wilson center and half-width are:

\text{center} = \frac{0.720+3.8416/(2{,}000)}{1+3.8416/1{,}000} = 0.7192, \qquad \text{half-width} = \frac{1.96}{1.003842} \sqrt{0.0002016+\frac{3.8416}{4{,}000{,}000}} = 0.02779.

The interval is \(0.7192\pm0.02779\), or approximately \((0.6914,0.7469)\). Under the simulation assumptions, the underlying probability of meeting the criterion is estimated as 72.0%, with a 95% Wilson interval from about 69.1% to 74.7%. This range describes Monte Carlo uncertainty, not uncertainty about whether the assumptions represent the intended study.

Choosing the Number of Simulation Replicates

A simulation should use enough replicates for its estimate to answer the planning question. For a target approximate 95% margin of error \(h\), the normal approximation gives:

M \approx \frac{z_{0.975}^{\,2}p_{\mathrm{pow}}(1-p_{\mathrm{pow}})}{h^2}.

Because the true power is not known in advance, a planning value is needed. The factor \(p_{\mathrm{pow}}(1-p_{\mathrm{pow}})\) is largest at \(p_{\mathrm{pow}}=0.5\), where it equals 0.25. Using 0.25 gives a conservative approximate replicate count for a specified absolute margin of error. If a defensible planning value is available, it may give a smaller count, but precision will depend on that value being reasonable.

Worked Example: Planning for a One-Percentage-Point Margin

Suppose the goal is an approximate 95% margin of error no greater than \(h=0.01\), or one percentage point. Using the conservative value \(p_{\mathrm{pow}}=0.5\):

M \approx \frac{1.96^2(0.5)(0.5)}{0.01^2} = \frac{3.8416(0.25)}{0.0001} = 9{,}604.

Plan at least 9,604 independent replicates under this approximation. If instead a planning value of \(p_{\mathrm{pow}}=0.80\) is justified, the calculation is:

M \approx \frac{1.96^2(0.80)(0.20)}{0.01^2} = 6{,}146.56.

Rounding up gives 6,147 replicates. This smaller count relies on the planning value: if the actual power is nearer 0.5, the achieved margin may be wider than one percentage point. A final interval based on the observed successes should therefore be reported, rather than treating the planned margin as guaranteed.

Boundary Results, Diagnostics, and Failure Modes

The plug-in standard error has a notable boundary limitation. If no replicate succeeds, \(\hat p_{\mathrm{pow}}=0\), and the plug-in formula returns zero; if every replicate succeeds, it also returns zero. Neither result establishes that the underlying probability is exactly zero or one. A Wilson interval or exact binomial interval better reflects the remaining uncertainty.

Worked Example: No Successful Replicates

In a hypothetical run of \(M=200\) independent replicates, suppose \(K=0\). The estimate is \(\hat p_{\mathrm{pow}}=0/200=0\), and the plug-in Monte Carlo standard error is \(\sqrt{0(1-0)/200}=0\). That zero is a boundary artifact, not evidence of perfect precision.

For a 95% Wilson interval with \(z=1.96\), the center and half-width are:

\text{center} = \frac{3.8416/400}{1+3.8416/200} = 0.009423, \qquad \text{half-width} = \frac{1.96}{1.019208} \sqrt{\frac{3.8416}{160{,}000}} = 0.009423.

The interval is approximately \((0,0.01885)\). The simulation observed no successes, but values of the underlying success probability up to about 1.9% remain compatible with this interval. If such small probabilities matter, more replicates may be needed to estimate them informatively.

The binomial standard error and interval require independent replicates generated under the same simulation conditions. If replicates share random inputs or are otherwise dependent, the binomial variance formula may not apply. Also check that the implementation counts the intended event exactly: an incorrect inequality at an equivalence boundary, an inconsistent missing-data rule, or a coding error in the success indicator can produce a precise estimate of the wrong quantity.

A fixed number of replicates avoids another subtle problem. Repeatedly checking an ordinary 95% interval and stopping the first time it looks sufficiently narrow does not, in general, preserve the nominal coverage. If simulation must stop according to interim results, use a method designed for sequential monitoring. Otherwise, set \(M\) in advance and report the resulting estimate and interval.

How to Use Precision Results

The standard error is convenient for a quick precision check, while a confidence interval communicates the range of values compatible with the simulation result. The Wald interval, \(\hat p_{\mathrm{pow}}\pm zSE_{\mathrm{MC}}\), is simple but can have poor coverage near zero or one and can extend outside the probability range. Wilson or exact binomial intervals are preferable when boundary estimates or small replicate counts are possible.

Precision should be judged in the context of the decision the estimate will inform. A difference of one percentage point may be material when comparing designs near a target power, whereas a wider interval might suffice for a rough feasibility assessment. The Monte Carlo interval does not account for uncertainty in assumed variability, treatment effects, dropout, or analysis choices. Those are separate sources of uncertainty and should not be made to appear resolved merely by increasing \(M\).

Key takeaway: Treat each independently simulated study as one Bernoulli trial for the prespecified success event. Report \(\hat p_{\mathrm{pow}}\), assess its Monte Carlo uncertainty with a standard error and preferably a suitable binomial interval, and choose \(M\) to match the precision needed for the intended decision.

Check Your Understanding

Answer each question, then compare with the solutions.

In 400 independent replicates, 300 meet the success criterion. Calculate the estimated power and Monte Carlo standard error.
Using the conservative planning value \(p_{\mathrm{pow}}=0.5\), approximately how many replicates are needed for a 95% margin of error of 0.02?
Why is a plug-in Monte Carlo standard error of zero after 200 successes in 200 replicates not proof that simulated power equals 1?
Name two assumptions needed for the binomial Monte Carlo calculations, and give one reason that Monte Carlo precision does not establish that the simulation model is appropriate.

Solutions. (1) \(\hat p_{\mathrm{pow}}=300/400=0.750\). The standard error is \(\sqrt{0.750(0.250)/400}=\sqrt{0.00046875}=0.02165\), or 2.17 percentage points. (2) \(M\approx1.96^2(0.25)/(0.02)^2=2{,}401\). (3) The estimate is at the boundary, so substituting it into the plug-in formula gives zero; finite simulation has not ruled out a success probability below 1. A Wilson or exact interval retains uncertainty. (4) Replicates must be independent and generated under the same success probability, with the success event defined and applied consistently. Even a very narrow Monte Carlo interval only quantifies simulation error under the specified model; it cannot show that assumed inputs or the model represent the intended study.