Monte Carlo Precision Checks for Sensitivity Results

Why Check Monte Carlo Precision?

Probabilistic sensitivity analysis propagates uncertainty in decision inputs through a decision rule. The reported probability of an event—such as meeting a development target or choosing go—depends both on the uncertainty represented by the input distributions and on the finite number of simulation draws used to estimate that probability. This tutorial focuses on the second contribution: Monte Carlo error, the numerical error caused by estimating a probability with a finite simulation.

Monte Carlo error is not the same as uncertainty about a drug’s true effect, nor is it the full uncertainty in a development decision. It is the variation we would expect in the estimated probability if we repeated the same simulation procedure with new random draws from the same specified uncertainty distribution. A probability estimate can therefore be precise as a simulation result while still reflecting substantial uncertainty about the decision itself.

The event must be stated clearly. For example, \(A\) might mean that predicted efficacy exceeds a prespecified target and safety remains acceptable. Each draw is classified as satisfying or not satisfying that event; the resulting indicators are what determine the precision of the estimated probability.

Standard Error and an Approximate Precision Interval

If the simulation draws are independent and each is generated from the same specified uncertainty distribution, then each \(I_m\) is a Bernoulli random variable with probability \(p_A\). Its expected value is \(p_A\), and its variance is \(p_A(1-p_A)\). The estimated probability is the mean of \(M\) such indicators.

The first expression gives the variance across repeated simulations; the second replaces the unknown \(p_A\) with its estimate. The resulting standard error is measured in probability units. It describes simulation precision, not the spread of outcomes across patients or the uncertainty distribution of a clinical parameter.

Batch Comparisons as a Diagnostic

A useful diagnostic is to divide a long run into \(B\) disjoint batches of equal size and calculate the event proportion in each batch. If draws are independent and the simulation is implemented correctly, batch estimates should fluctuate around a common value. Their variability should be broadly compatible with the standard error expected for a batch of that size.

Batch comparisons can reveal an implementation problem, an accidental change in assumptions, or an unstable simulation process. They are not a proof of accuracy: a small number of batches gives a noisy picture, and agreement between batches cannot detect a shared coding error or a wrong uncertainty distribution. Use them alongside the formula-based standard error and checks of the simulation code and inputs.

Choosing a Sufficient Number of Draws

A draw count should be tied to a precision requirement, such as an approximate 95% Monte Carlo margin of error no larger than \(h\) probability units. For a planning value \(p_A\), rearranging the normal-approximation margin \(1.96\sqrt{p_A(1-p_A)/M}\) gives the required number of draws. Since \(p_A(1-p_A)\leq0.25\) for any probability, the maximum occurs at \(p_A=0.5\). This provides a conservative calculation when there is no reliable pilot estimate.

Failure Modes and Alternatives

The standard error formula relies on independent, identically distributed simulation draws from the target uncertainty distribution. If draws are correlated across iterations, the nominal count \(M\) overstates the amount of independent information and the formula can understate simulation error. Likewise, if batch proportions show a persistent trend or unexpectedly large differences, check the random-number implementation, code, input handling, and whether each batch actually used the intended simulation procedure. Do not treat a visually stable running estimate alone as evidence that the simulation is correct.

A particularly important boundary case occurs when the event is rare. If no successes are observed, the plug-in formula gives a standard error of zero because \(\hat{p}_A=0\). That does not establish that \(p_A=0\). For zero successes in \(M\) independent trials, an exact one-sided 95% binomial upper bound is \(1-0.05^{1/M}\), approximately \(3/M\) when \(M\) is large. Use a binomial interval appropriate for rare events rather than interpreting a zero plug-in standard error literally.

For example, with zero events in 5,000 independent hypothetical draws, the approximate upper bound is \(3/5000=0.000600\), or 0.0600%. The simulation has not shown that the event is impossible; it has only failed to observe it at this draw count. If rare-event probabilities are decision-critical, increase the simulation effort or use a method appropriate to rare-event estimation, and report the method and its precision.

When an event probability can be calculated analytically, that calculation avoids Monte Carlo error and may be preferable. For ordinary independent Bernoulli draws, binomial intervals can improve on the normal approximation, especially near 0 or 1. Repeated batches diagnose instability but do not replace a valid error calculation. Designs that deliberately alter how draws are generated, such as stratified or quasi-Monte Carlo methods, may have different error properties; the independent-draw formula should not be applied automatically.

A practical report should identify the event being estimated, the number of draws, the event count, the estimated probability, and its Monte Carlo precision. State whether the draws satisfy the independence assumption behind the calculation. If repeated looks at the simulation are used to decide when to stop, treat a conventional fixed-sample interval as an approximation; a prespecified draw count and final precision check make the reported precision easier to interpret.