Introduction
A simulation can produce a precise-looking estimate and still represent the wrong probability model. Two common causes are treating dependent events as independent and coding the link between random numbers and outcomes incorrectly. In either case, repeating the simulation more times does not repair the model: it gives a more stable estimate of the wrong quantity.
A useful audit asks two separate questions. First, does the simulation update the possible outcomes and probabilities correctly as a process unfolds? Second, does each generated random number lead to an outcome with the probability intended by the model? Tracing a few possible paths and checking exact probabilities can reveal errors that are hard to see from a long run alone.
Independence Must Match the Process
A sequence of independent random-number draws does not guarantee that the simulated outcomes are independent. For example, a program may generate a fresh uniform random number at every stage, but use a different set of available outcomes or a different probability rule after the first outcome. The random numbers can be independent while the outcomes are dependent.
Conversely, if the process really consists of independent repetitions with a fixed probability for each outcome, the simulation should preserve that probability at every repetition. The correct implementation depends on the model: a draw with replacement leaves the set of possible objects unchanged, while a draw without replacement removes the selected object. The word “random” does not decide which rule applies; the description of the process does.
Worked Example: Two Draws Without Replacement
Suppose a bag contains 4 red and 6 blue tokens. Two tokens are drawn uniformly at random, without replacement. Find the probability that both are red, and diagnose a simulation that resets the bag before the second draw.
Model and assumptions. There are 10 distinct tokens. The first token is equally likely to be any of the 10; after it is drawn, it is not returned, and the second token is equally likely to be any of the 9 remaining tokens. These rules describe dependence: if the first token is red, only 3 red tokens remain.
The probability of red on the first draw is \(4/10\). Given a red first draw, the probability of red on the second is \(3/9\). Therefore,
A faulty program might draw a token uniformly from all 10 tokens on each stage, restoring the original composition for the second draw. That simulates replacement. It gives \((4/10)(4/10)=0.16\) for two reds, not \(2/15\). Its second-stage probability is wrong because it ignores the first draw.
For a concrete path check, label the original tokens \(R_1,R_2,R_3,R_4,B_1,\ldots,B_6\). If the first draw selects \(R_2\), a valid second draw can select any of the nine remaining tokens, but cannot select \(R_2\) again. A program that permits \(R_2\) on the second draw has not implemented drawing without replacement. Checking a few such paths can expose the defect before running thousands of repetitions.
Mapping Uniform Random Numbers to Outcomes
Many simulations use a random number \(U\) that is uniformly distributed on \([0,1)\). To represent an outcome with probability \(p\), assign it an interval of length \(p\). For several outcomes, use non-overlapping intervals whose lengths equal their probabilities and that cover \([0,1)\). Endpoints must be handled consistently; with half-open intervals, a boundary value belongs to exactly one interval.
This gives a direct check on outcome coding: read the boundaries in the program, calculate each interval’s length, and compare that length with the intended probability. If all outcomes are meant to be equally likely, equal intervals are appropriate. If their probabilities differ, equal intervals are not appropriate.
Worked Example: Coding a Weighted Spinner
A spinner has outcomes A, B, and C with probabilities \(0.50\), \(0.30\), and \(0.20\), respectively. A program uses \(U\sim\text{Uniform}[0,1)\). Give the correct coding rule and identify the error in splitting the interval into three equal parts.
Model and assumptions. Each spin has the stated outcome probabilities. A generated \(U\) is uniform on \([0,1)\), and one spin is simulated by assigning each value of \(U\) to exactly one outcome. The interval for each outcome must therefore have length equal to that outcome’s probability.
One correct rule is:
- If \(0\leq U<0.50\), record A.
- If \(0.50\leq U<0.80\), record B.
- If \(0.80\leq U<1\), record C.
For instance, \(U=0.12\) gives A, \(U=0.67\) gives B, and \(U=0.91\) gives C. The interval lengths are \(0.50\), \(0.30\), and \(0.20\), so the rule represents the target probabilities.
The interval-length justification is direct. For a uniform \(U\) on \([0,1)\), the probability that \(U\) falls in an interval is the interval’s length. Thus \(P(0\leq U<0.50)=0.50\), \(P(0.50\leq U<0.80)=0.30\), and \(P(0.80\leq U<1)=0.20\). The assignments produce the intended probabilities.
If the program instead assigns the first third of the interval to A, the second third to B, and the last third to C, it simulates probabilities \(1/3,1/3,1/3\). For example, \(U=0.67\) would then be assigned to C rather than B. A long run of this faulty code would tend toward roughly one-third of each outcome, not the spinner’s specified distribution.
In a finite run, observed proportions will not usually equal the target probabilities exactly. That alone does not establish a coding error. The audit is whether the code’s intervals have the correct lengths; simulated frequencies are a useful additional diagnostic, not a requirement for exact matches in every run.
Conditional Stages Need Conditional Probabilities
In a multi-stage process, the probability used at a later stage may depend on what happened earlier. The simulation should preserve that dependence by applying the relevant conditional probability on each path. This does not require reusing a random number. Independent random numbers can drive successive stages while the outcome rule for a later number changes according to the earlier simulated state.
Worked Example: A Conditional Shutdown Stage
Suppose an alarm occurs with probability \(0.20\). A shutdown occurs with probability \(0.70\) if an alarm has occurred, and with probability \(0.10\) if there was no alarm. Find the probability of both an alarm and a shutdown, and show how to simulate the two stages.
Model and assumptions. The first stage produces an alarm with probability \(0.20\). The shutdown rule is conditional: its probability is \(0.70\) on alarm paths and \(0.10\) on no-alarm paths. Use independent uniform random numbers \(U_1\) and \(U_2\), each on \([0,1)\), with \(U_1\) determining the alarm and \(U_2\) determining shutdown under the threshold appropriate to the first-stage result.
Set alarm if \(U_1<0.20\). If there is an alarm, set shutdown if \(U_2<0.70\); otherwise set shutdown if \(U_2<0.10\). For example, \(U_1=0.12\) and \(U_2=0.50\) produce alarm and shutdown. But \(U_1=0.45\) and \(U_2=0.50\) produce no alarm and no shutdown.
The probability of both events is the probability of an alarm multiplied by the conditional probability of shutdown given alarm:
To see why using a single unconditional shutdown probability would be wrong, first calculate that marginal probability:
If a program incorrectly treated alarm and shutdown as independent, it would calculate their joint probability as \((0.20)(0.22)=0.044\), rather than \(0.14\). The error is that \(0.22\) averages shutdown probabilities across both kinds of paths; it is not the probability of shutdown specifically on alarm paths. The two uniform random numbers in the correct simulation are independent, but the resulting alarm and shutdown outcomes are dependent because the shutdown threshold changes after the alarm result.
A Practical Audit for Simulation Errors
A useful diagnosis proceeds from the intended process to the code, rather than from a surprising output back to a guess about the cause. State what can happen at each stage, what probabilities apply, and what changes after an outcome. Then trace the code against those statements.
Specify the possible outcomes, their probabilities, the order of stages, and whether outcomes are replaced, removed, or affected by earlier results.
Follow a few possible inputs through the code. Confirm that impossible paths stay impossible and that each later-stage rule uses the correct state.
For a uniform-number method, calculate every interval length. For draws from a changing set, check that each draw uses the current set of available outcomes.
Calculate the probability for a small event from the stated model and compare it with the simulation estimate. A mismatch may indicate a coding problem or ordinary simulation error, so also inspect the actual rules.
For example, an event estimated at \(0.14\) in a simulation is not automatically evidence that the code is correct just because \(0.14\) is plausible. The path rules and probability mapping still need to match the model. Similarly, a discrepancy between a simulation and an exact probability does not by itself prove a bug: finite-run estimates vary. The point of exact checks is to provide a reference value alongside a direct inspection of the implementation.
Common Mistakes
- Assuming each stage is independent because it uses a new random number. New independent random numbers can still generate dependent outcomes when later rules depend on the earlier state. Check the outcome probabilities conditional on the path so far.
- Resetting the set of outcomes without replacement. This restores objects that should remain removed and changes later probabilities. Update the available set after each draw.
- Giving unequal-probability outcomes equal-sized intervals. Equal intervals simulate equal probabilities. Use interval lengths that match the specified weights.
- Using an unconditional probability at a conditional stage. A marginal probability averages over different histories; it may not apply within a particular history. Use the probability conditional on that history.
- Expecting simulated frequencies to match exactly. Random variation remains in a finite run. Check whether the code implements the right probability model before interpreting differences in observed frequencies.
Check Your Understanding
Use the model rules, not just the simulated frequencies, to decide whether each implementation is sound.
- A bag contains 3 green and 5 yellow tokens. Two are drawn without replacement. Why should the probability of green on the second draw depend on the first draw?
- A uniform random number on \([0,1)\) is split into intervals of lengths \(0.25\) and \(0.75\). What probabilities do these intervals assign to the two outcomes?
- In the conditional shutdown example, what threshold should the simulation use for shutdown after a no-alarm result, and why?
- Can independent random-number draws produce dependent simulated outcomes? Explain using a rule that changes after an earlier outcome.
- A simulation produces observed proportions close to the target probabilities in one run. Why is this not enough, by itself, to establish that its outcome coding is correct?