Checking Simulations Against Exact Probability Results

Why Compare a Simulation With an Exact Result?

A simulation estimates a probability by repeatedly generating outcomes from a model and recording how often an event occurs. Even when the model and simulation are both correct, the simulated proportion will usually differ slightly from the exact probability. The simulation uses a finite number of random trials; it does not generally reproduce the theoretical probability exactly.

For a small model, it may be possible to calculate the exact probability directly. That result gives a useful benchmark: it lets us check whether a simulated estimate is close enough to be plausible given ordinary simulation variability. The comparison is not a proof that the simulation code is correct. It is one diagnostic, and a mistake in the code or model can sometimes produce a plausible-looking estimate.

Definition: An exact probability is a probability calculated from the specified chance model, rather than estimated from a finite simulation. A Monte Carlo estimate of an event probability is the simulated proportion of trials on which the event occurs. The difference between the estimate and the exact probability is expected to vary from run to run because of randomness.

Calculate an Exact Probability in a Small Model

When the possible outcomes are equally likely and can be listed, an event’s exact probability is the number of outcomes in the event divided by the total number of possible outcomes. It matters that the outcomes counted are the outcomes of the model: for two dice, for example, the ordered pair records the first die and the second die. There are 36 equally likely ordered pairs, not 11 equally likely possible sums.

Worked Example: An Exact Probability for Two Dice

Suppose two fair six-sided dice are rolled. Find the exact probability that their sum is 5 or less.

Model and assumptions. Each die has six possible faces, each equally likely. The dice are independent, so each ordered pair of faces has probability $1/36$. There are $6\times6=36$ such pairs.

For sums of 2, 3, 4, and 5, the numbers of ordered pairs are respectively 1, 2, 3, and 4. For instance, sum 5 has the pairs $(1,4),(2,3),(3,2),(4,1)$. Thus the event contains

$$ 1+2+3+4=10 $$

ordered pairs. Therefore,

P(\text{sum}\leq 5)=\frac{10}{36}=\frac{5}{18}\approx 0.2778.

The exact probability is $5/18$, or about 27.78%. This value is the benchmark for a simulation of this same model and event.

Measure the Simulation’s Expected Variability

To compare an estimate with an exact probability, we need a measure of how much a simulated proportion typically varies. Suppose each simulated trial independently follows the same model, and the event has probability $p$ on every trial. Let $I_i=1$ if the event occurs on trial $i$, and $I_i=0$ otherwise. Then each $I_i$ is a Bernoulli random variable with mean $p$ and variance $p(1-p)$. The simulated proportion is the mean of these indicators:

\hat{p}=\frac{I_1+\cdots+I_n}{n}.

Because the trials are independent, the variance of their sum is the sum of their variances. Dividing that sum by $n^2$ gives the variance of the simulated proportion:

\operatorname{Var}(\hat{p}) =\frac{n p(1-p)}{n^2} =\frac{p(1-p)}{n}.

The square root of this variance is the Monte Carlo standard error (Monte Carlo $SE$) of the simulated proportion. It measures the standard deviation of $\hat{p}$ across repeated simulations of $n$ independent trials. When an exact $p$ is available, use it in this calculation to assess the simulated estimate.

Result: If the simulated trials are independent, each has the same event probability $p$, and $\hat{p}$ is the proportion of event outcomes in $n$ trials, then $$ SE(\hat{p})=\sqrt{\frac{p(1-p)}{n}}. $$ The difference $\hat{p}-p$, expressed in units of this standard error, is $$ \frac{\hat{p}-p}{\sqrt{p(1-p)/n}}. $$ This signed quantity describes the discrepancy on the scale of simulation variability; it is not, by itself, a probability or a formal test.

The assumptions matter. If a program accidentally makes outcomes dependent, changes the event probability from trial to trial, or does not generate the intended model, this standard-error formula need not describe its variability. The formula also assumes $0<p<1$; if $p=0$ or $p=1$, the event never varies under the model and the standard error is zero.

Worked Example: A Large Simulation Close to the Exact Value

For the two-dice event that the sum is 5 or less, a hypothetical simulation runs $n=36{,}000$ independent trials and records 9,990 event outcomes. Compare its estimate with the exact probability.

The simulated proportion is

\hat{p}=\frac{9{,}990}{36{,}000}=0.2775.

The exact value is $p=5/18\approx 0.2777778$. Under the stated model, the event has the same probability on each trial and trials are independent, so the binomial standard-error calculation applies. The Monte Carlo $SE$ is

SE(\hat{p}) =\sqrt{\frac{(5/18)(13/18)}{36{,}000}} =\sqrt{\frac{65}{11{,}664{,}000}} \approx 0.002361.

The signed difference is

\hat{p}-p =0.2775-\frac{5}{18} \approx -0.0002778,

which is about $-0.0002778/0.002361\approx -0.1177$ standard errors. The simulated estimate is therefore about 0.12 standard errors below the exact value, a small discrepancy relative to the variability expected from 36,000 trials. This is consistent with ordinary simulation variation.

Interpret the Difference in Context

A difference is easier to assess after placing it on the Monte Carlo standard-error scale. A raw difference of 0.01 might be substantial in a very large simulation but unremarkable in a small one. The standard error accounts for both the event probability and the number of simulated trials.

This comparison is descriptive. A discrepancy of one or two standard errors is on the scale of ordinary random fluctuation, but that statement alone does not give the exact probability of observing such a discrepancy. The binomial model specifies the full distribution of the event count if that more detailed probability is needed. For checking a simulation, the standard-error comparison is a useful first diagnostic, not a guarantee that every implementation detail is correct.

Worked Example: A Larger Difference in a Smaller Run

Another hypothetical simulation uses the same two-dice model and event, but runs only $n=400$ trials. It records 126 event outcomes. Compare its estimate with the exact probability.

The estimate is

\hat{p}=\frac{126}{400}=0.315.

The model and event are unchanged, so $p=5/18$. The trials are stipulated to be independent repetitions, and the event probability remains constant. The standard error is

SE(\hat{p}) =\sqrt{\frac{(5/18)(13/18)}{400}} =\sqrt{\frac{65}{129{,}600}} \approx 0.02240.

The estimate is above the exact probability by

0.315-\frac{5}{18} \approx 0.03722.

In standard-error units, this is $0.03722/0.02240\approx 1.662$. A difference of this size is larger than in the 36,000-trial example, but it is still on the scale of simulation variability for a 400-trial run. This comparison does not establish that the estimate is correct or incorrect; it shows how to interpret its distance from the exact benchmark relative to the run size.

Use the Benchmark Carefully

When an exact result is available, a sound check has several parts: verify the exact count or calculation, confirm that the simulated trial matches the chance model, calculate the simulated proportion, and compare the difference with its Monte Carlo standard error. In the dice examples, each trial must roll two fair dice and record whether their sum is at most 5. Simulating a different event or assigning unequal probabilities to the faces would not be a valid check against $5/18$.

A close match is reassuring, but not conclusive. For example, an incorrect program might still produce a proportion near the exact value by chance, especially if its error is small. Conversely, a correct program can produce a relatively distant estimate in a particular run. Rechecking the outcome mapping and event coding is sensible when a discrepancy is surprising; agreement alone cannot rule out those errors.

Key takeaway: An exact probability is a theoretical benchmark; a simulation proportion is a random estimate. Compare their difference with $\sqrt{p(1-p)/n}$, using the exact $p$ when available and checking that the simulation’s trials satisfy the model assumptions.

Common Mistakes

Expecting exact equality. “The simulated proportion is not exactly $p$, so the simulation must be wrong” ignores the random variation of finite runs. Check the difference relative to its Monte Carlo standard error.
Counting sums as equally likely dice outcomes. The possible sums are not equally likely. Count the equally likely ordered face pairs, or otherwise calculate their probabilities correctly.
Using the wrong standard error. The standard error of a simulated proportion is $\sqrt{p(1-p)/n}$, not $p(1-p)/n$. The latter is its variance.
Treating a close match as proof. “The estimate is close, so the code must be correct” overstates what the comparison establishes. A benchmark is a diagnostic, not proof of correct implementation.
Ignoring run size. The same raw discrepancy can have different meanings for different $n$, because the standard error depends on $n$.

Check Your Understanding

Answer each question using the model and reasoning in this tutorial.

For two fair dice, why are there 36 equally likely outcomes rather than 11 equally likely sums?
A simulated event occurs in 84 of 300 independent trials. What is the simulated proportion?
State the assumptions needed to use $\sqrt{p(1-p)/n}$ as the standard error of a simulated proportion.
If $p=0.25$ and $n=1{,}000$, write the expression for the Monte Carlo standard error. What happens to this standard error if $n$ increases while $p$ stays fixed?
Why does a simulated estimate that closely matches an exact probability not prove that the simulation code is correct?