Quantifying Monte Carlo Error and Choosing a Run Size

Why a Simulation Estimate Varies

A simulation estimates a probability by repeating a chance experiment and recording how often an event occurs. Even when every run follows exactly the same model, a new set of random trials will usually produce a different event proportion. This variation is Monte Carlo error: the random error in an estimate caused by using a finite number of simulated trials.

The size of a simulation estimate alone does not tell us how much Monte Carlo error to expect. An estimate based on 100 trials is generally less stable than one based on 10,000 trials, but the event probability matters too. The formulas in this tutorial quantify that variability and let us plan a run size for a desired level of approximate precision.

From Event Outcomes to Standard Error

Suppose a simulation consists of $n$ independent trials, and each trial uses the same probability model. Let the event of interest have probability $p$ on any one trial. Code a success as 1 when the event occurs and 0 when it does not. The simulated event proportion, $\hat{p}$, is the number of successes divided by the number of trials.

Definition: The Monte Carlo standard error of a simulated event proportion is its standard deviation across repeated runs of the same size under the simulation model. For $n$ independent trials with event probability $p$, it is $$ SE(\hat{p})=\sqrt{\frac{p(1-p)}{n}}. $$ Because $p$ is usually unknown, an estimated Monte Carlo standard error uses the simulated proportion: $$ \widehat{SE}(\hat{p})=\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}. $$

The standard error is measured in proportion units. For instance, a standard error of 0.01 means that the simulated proportion typically varies by about 0.01, or one percentage point, from the model probability across repeated runs of that size. It does not measure whether the probability model itself is a good description of the real situation.

Why the Formula Works

For each trial $i$, define $Y_i=1$ if the event occurs and $Y_i=0$ otherwise. Then $Y_i$ is a Bernoulli random variable: $P(Y_i=1)=p$ and $P(Y_i=0)=1-p$. Its expected value is $E(Y_i)=p$. Since $Y_i^2=Y_i$, its variance is

\operatorname{Var}(Y_i)=E(Y_i^2)-[E(Y_i)]^2 =p-p^2=p(1-p).

The simulated proportion is the average of these success indicators, $\hat{p}=(Y_1+\cdots+Y_n)/n$. Independence means the variances of the indicators add, so

\operatorname{Var}(\hat{p}) =\operatorname{Var}\left(\frac{Y_1+\cdots+Y_n}{n}\right) =\frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(Y_i) =\frac{np(1-p)}{n^2} =\frac{p(1-p)}{n}.

Taking the square root gives $SE(\hat{p})=\sqrt{p(1-p)/n}$. In practice, replacing $p$ by $\hat{p}$ gives an estimate of that standard error. This derivation depends on trials being independent and having the same event probability. If a simulation's trial design does not meet those conditions, this formula may not describe its variability.

Worked Example: Estimating Monte Carlo Error

Suppose a hypothetical card simulation repeatedly shuffles and draws one card from a standard deck, replacing and reshuffling before each new trial. The event is drawing a heart. In 1,000 independent trials, hearts appear 184 times. Estimate the event probability and its Monte Carlo standard error.

The simulated proportion is $$ \hat{p}=\frac{184}{1{,}000}=0.184. $$

Use this estimate in the standard error formula: $$ \widehat{SE}(\hat{p}) =\sqrt{\frac{0.184(1-0.184)}{1{,}000}} =\sqrt{\frac{0.184(0.816)}{1{,}000}} =\sqrt{0.000150144} \approx 0.01225. $$

The estimated Monte Carlo standard error is about 0.0123, or 1.23 percentage points. This describes the typical run-to-run variability of a 1,000-trial estimate under the specified simulation model; it is not a claim that the model is accurate for every real-world question involving cards.

Approximate Margins of Error

When the number of successes and failures is large enough for a normal approximation to be reasonable, the repeated-run variation of $\hat{p}$ is approximately normal around $p$. A useful approximate margin of error is a normal critical value multiplied by the standard error. For a 95% level, the critical value is about 1.96.

Approximate result: For $n$ independent, identically distributed event trials, an approximate margin of error at confidence level $C$ is $z^*SE(\hat{p})$, where $z^*$ is the standard normal critical value for that level. With an estimated standard error, the approximate margin is $$ m \approx z^*\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}. $$ This is a planning and variability approximation, not a guarantee that a particular run will be within $m$ of $p$.

For the card simulation, using the unrounded standard error gives an approximate 95% margin of $$ 1.96\sqrt{\frac{0.184(1-0.184)}{1{,}000}} =1.96\sqrt{0.000150144} \approx 0.0240. $$

Thus a rough description of the Monte Carlo precision is 0.184 with an approximate margin of 0.0240, or 2.40 percentage points. Across many hypothetical repetitions of this 1,000-trial simulation, the normal approximation says that about 95% of the resulting intervals formed by taking the estimate plus or minus this margin would cover the model probability. It does not mean there is a 95% probability that the fixed model probability lies in this particular interval.

Choosing a Run Size

A run-size question starts with a target margin $m$ and a desired approximate confidence level. Rearranging the margin formula gives a planning equation. If a preliminary estimate of $p$ is available, use it to estimate the required number of trials:

m=z^*\sqrt{\frac{p(1-p)}{n}} \quad\Longrightarrow\quad n=\frac{(z^*)^2p(1-p)}{m^2}.

In planning, the $p$ on the right is either a reasonable preliminary value or a conservative value chosen when no useful preliminary estimate is available. Since a trial count must be a whole number, round the calculated value up. Rounding down could make the run smaller than the target precision requires.

Worked Example: Planning From a Preliminary Estimate

A hypothetical simulation team expects an event proportion near 0.30 based on a small preliminary run. They want a 95% approximate margin of error no larger than 0.02. How many independent trials should they plan?

Use $z^*=1.96$, preliminary $p=0.30$, and target $m=0.02$: $$ n=\frac{(1.96)^2(0.30)(1-0.30)}{(0.02)^2} =\frac{3.8416(0.21)}{0.0004} =\frac{0.806736}{0.0004} =2{,}016.84. $$

Rounding up gives a planned run size of $n=2{,}017$ trials. The approximate standard error at the planning value is $$ \sqrt{\frac{0.30(0.70)}{2{,}017}} =\sqrt{\frac{0.21}{2{,}017}} \approx 0.01020, $$ so the corresponding approximate 95% margin is $1.96(0.01020)\approx 0.0200$. The planning value implies about 605 successes and 1,412 failures, so the normal approximation is reasonable if the actual event probability is indeed near 0.30. The achieved precision may differ if the actual probability is substantially different.

Planning Without a Preliminary Probability

The quantity $p(1-p)$ is largest at $p=0.5$, where it equals 0.25. This follows from

p(1-p)=\frac{1}{4}-\left(p-\frac{1}{2}\right)^2 \leq \frac{1}{4}.

Therefore, using $p=0.5$ gives a conservative run-size calculation for the standard error formula: it plans for the greatest variability any event proportion can have. This can require more trials than necessary when the probability is far from one-half, but it avoids relying on an uncertain guess of $p$.

Worked Example: A Conservative Run Size

Suppose a hypothetical simulation needs a 95% approximate margin of error no larger than 0.01, but no preliminary estimate of the event probability is available. Find a conservative run size.

Use $p=0.5$, $z^*=1.96$, and $m=0.01$: $$ n=\frac{(1.96)^2(0.5)(0.5)}{(0.01)^2} =\frac{3.8416(0.25)}{0.0001} =\frac{0.9604}{0.0001} =9{,}604. $$

Plan for 9,604 independent trials. At the conservative value $p=0.5$, the standard error is $$ \sqrt{\frac{0.25}{9{,}604}} \approx 0.005103, $$ and $1.96(0.005103)\approx 0.0100$. If the actual event probability is not near 0.5, its Monte Carlo standard error will be smaller at this run size, under the same independent-trial assumptions.

What More Trials Can—and Cannot—Fix

The standard error decreases in proportion to $1/\sqrt{n}$. Consequently, multiplying the number of trials by four cuts the Monte Carlo standard error approximately in half; multiplying it by 100 cuts it by about a factor of 10. Improving precision can therefore become computationally expensive. The run size should be chosen to meet a useful target, not increased without considering the scale of error that matters.

Monte Carlo error is only one source of uncertainty. More trials reduce the random variation of an estimate under the specified model, but they do not correct an incorrectly defined event, a coding error, or assumptions that make the simulation model inappropriate for the question. A very stable simulated proportion can still estimate the wrong probability if the model does not represent the intended chance process.

Common Mistake

Pitfall: “I ran the simulation ten times longer, so the Monte Carlo error must be ten times smaller.” The standard error scales as $1/\sqrt{n}$, not $1/n$. A tenfold increase in trials reduces standard error by a factor of $\sqrt{10}$, about 3.16, rather than by a factor of 10. To reduce standard error by a factor of 10, the run size must be multiplied by $10^2=100$.

Also distinguish a standard error from a margin of error: the margin is a chosen critical value times the standard error. For example, an estimated standard error of 0.01 corresponds to an approximate 95% margin of $1.96(0.01)=0.0196$, not 0.01. State which quantity is being reported and keep the scale in context.

Key takeaway: For independent simulation trials with the same event probability, the standard error of the simulated event proportion is approximately $\sqrt{\hat{p}(1-\hat{p})/n}$. Use a target margin and a suitable critical value to plan $n$, round up, and interpret the result as approximate Monte Carlo precision under the simulation model.

Check Your Understanding

Try these questions without looking back.

A simulation has 250 successes in 1,000 independent trials. Calculate the simulated proportion and its estimated Monte Carlo standard error.
Why does the standard error formula require independent trials with the same event probability?
A planned run uses a preliminary event probability of 0.40 and seeks a 95% approximate margin of 0.03. Write the run-size formula and explain why the result must be rounded up.
When no preliminary probability is available, why is $p=0.5$ a conservative planning value?
What happens to the Monte Carlo standard error when the run size is multiplied by four? Explain why more trials do not correct a faulty simulation model.