Comparing Alternative Probability Models by Simulation

Why Compare Probability Models?

A simulation does not predict an outcome without assumptions. It applies a set of rules to uncertain inputs and records what happens. When two reasonable accounts of the same process are possible, their predictions may differ because they represent uncertainty differently: for example, one may treat events as independent while another allows them to cluster. Comparing the models can reveal whether that choice matters for the event of interest.

A useful comparison changes the model assumption under investigation while keeping the rest of the setup as similar as possible. Both models should describe the same process, use the same definition of the event, and be run for the same number of trials. Whenever possible, use matched inputs: on each replicate, give both models the same underlying random inputs and other fixed conditions. Then a difference in their results is easier to connect to the model rules rather than to the fact that they happened to receive different simulated conditions.

Definition: A simulation model is a specified set of rules for turning inputs, including uncertain inputs, into an outcome. For a defined event, the simulated event rate is the number of simulation replicates on which that event occurs divided by the total number of replicates. Comparing alternative probability models means running distinct, plausible rule sets for the same process and comparing their predicted event rates under a common setup.

Let model A and model B be run for \(R\) matched replicates. For replicate \(i\), define \(Y_{Ai}=1\) if the event occurs under model A and \(Y_{Ai}=0\) otherwise; define \(Y_{Bi}\) in the same way. Their simulated event rates are

\hat{r}_A=\frac{1}{R}\sum_{i=1}^{R}Y_{Ai}, \qquad \hat{r}_B=\frac{1}{R}\sum_{i=1}^{R}Y_{Bi}.

The estimated difference, taking model B minus model A, is

\hat{\Delta}=\hat{r}_B-\hat{r}_A =\frac{1}{R}\sum_{i=1}^{R}(Y_{Bi}-Y_{Ai}).

A positive difference means the event occurred more often in the simulated runs under model B; a negative difference means it occurred less often. The rate is a proportion of simulated replicates, not a guarantee about what will happen in a future case.

How to Make a Fair Comparison

Before comparing results, specify the assumptions of each model. State what is held fixed, what varies randomly, and how the random quantities are related. Also define the event precisely. For example, “a failure” may mean at least one component fails during a run, while “multiple failures” means at least two fail. These are different events and can produce different model comparisons.

Define the shared question.
State the process, the event being counted, and any fixed conditions, such as the number of components or the length of a time period.

Write down the competing assumptions.
Be specific about what differs between the models, such as whether risks are independent or affected by a shared condition. Do not silently change other rules.

Run matched replicates.
Use the same number of trials and, where practical, the same underlying random inputs and fixed conditions for the two models in each replicate.

Calculate and interpret the rates.
Divide each model’s event count by its number of replicates, then report the difference in the stated direction and in context.

Check the conclusion.
Ask whether the size and direction of the difference make sense under the changed assumption, and whether additional simulation runs might clarify a small or unstable difference.

Worked Comparisons

Worked Example: Independent or Clustered Component Failures?

A hypothetical system has three components. The event of interest is that at least two components fail during one operating period. Both candidate models give each component a 10% marginal failure risk. Model A assumes the three failures are independent. Model B assumes components share an operating condition: with probability 0.20 the period is adverse, and with probability 0.80 it is ordinary. Given an adverse period, each component has a 0.40 failure risk; given an ordinary period, each has a 0.025 failure risk. Within either condition, Model B treats component failures as independent.

First check that the models match on the marginal risk for one component. Under Model B, that risk is

$$ (0.20)(0.40)+(0.80)(0.025)=0.08+0.02=0.10. $$

So a difference in the event rate is not explained by one model simply assigning a larger average risk to each component. The changed assumption is whether a shared operating condition can raise the risks of several components at once.

Suppose a hypothetical simulation runs 200 matched operating periods for each model. The models use the same period-level inputs and the same component-level random inputs in corresponding replicates; Model A applies its 0.10 risk, while Model B applies the threshold set by that period’s condition. The simulated event counts are 6 periods under Model A and 15 under Model B. Therefore,

\hat{r}_A=\frac{6}{200}=0.030, \qquad \hat{r}_B=\frac{15}{200}=0.075, \qquad \hat{\Delta}=0.075-0.030=0.045.

In these simulation runs, the estimated chance of at least two failures is 3.0% under the independent model and 7.5% under the shared-condition model, a difference of 4.5 percentage points. The result illustrates why matching marginal risks is not enough to make models equivalent for a multi-component event: dependence can affect how often failures occur together. The simulated difference is an estimate, not an exact probability or proof that the shared-condition model is the true one.

Worked Example: Independent or Persistent Wet Days?

Consider a hypothetical five-day period, and define the event as “all five days are wet.” Model A assigns each day a 0.30 chance of being wet and assumes day-to-day independence. Model B uses a state-transition rule: the first day is wet with probability 0.30; after a wet day, the next day is wet with probability 0.60; after a dry day, the next day is wet with probability about 0.1714. This transition rule gives a long-run wet-day proportion of 0.30, matching Model A’s marginal rate, but permits wet days to persist.

To check the matching in Model B, if the current day is wet with probability 0.30, the next-day wet probability is

(0.30)(0.60)+(0.70)(0.1714) \approx 0.1800+0.1200=0.3000.

Suppose 10,000 matched five-day periods are simulated. The same sequence of uniform random inputs is supplied to each model; each model uses those inputs according to its own rules. In a hypothetical run, the all-wet event occurs 24 times under Model A and 384 times under Model B. The estimated event rates are

\hat{r}_A=\frac{24}{10{,}000}=0.0024, \qquad \hat{r}_B=\frac{384}{10{,}000}=0.0384, \qquad \hat{\Delta}=0.0384-0.0024=0.0360.

The simulated rate for five consecutive wet days is 0.24% under Model A and 3.84% under Model B, a difference of 3.60 percentage points. The models agree on the single-day wet rate, but differ on persistence, which directly affects the chance of a long uninterrupted wet spell. This is a model comparison, not evidence that the transition rule describes actual weather.

Worked Example: Drawing Two Red Items

A hypothetical container has 3 red items and 7 blue items. Two items are drawn, and the event is that both are red. Compare a model that replaces the first item before the second draw with one that does not. Assume each model starts with the same composition and follows its stated drawing rule exactly.

The replacement model has probability \(3/10\) of red on each draw, so its exact event probability is

\left(\frac{3}{10}\right)\left(\frac{3}{10}\right)=0.0900.

For the no-replacement model, after a red first draw there are 2 red items among 9 remaining items. Its exact event probability is

\left(\frac{3}{10}\right)\left(\frac{2}{9}\right)=\frac{1}{15}\approx 0.0667.

Now suppose 6,000 simulated two-draw trials are run under each model, using the same initial composition and trial count. A hypothetical simulation records both-red outcomes 541 times with replacement and 397 times without replacement. The simulated rates and their difference are

\hat{r}_{\text{replace}}=\frac{541}{6{,}000}\approx 0.0902, \qquad \hat{r}_{\text{no replace}}=\frac{397}{6{,}000}\approx 0.0662,

\hat{r}_{\text{no replace}}-\hat{r}_{\text{replace}} \approx 0.0662-0.0902=-0.0240.

The simulated event rate is about 2.40 percentage points lower without replacement. The rates are close to the exact probabilities calculated from each rule. In this example, the composition and event definition are held fixed; the replacement assumption is what changes the probability of a second red draw after a first red draw.

Why Matching Helps, and What It Cannot Do

Matching inputs makes the comparison more focused because it removes some irrelevant variation between runs. If both models face the same simulated adverse conditions in a replicate, their paired outcomes can be compared directly. The difference in rates is the average of the replicate-by-replicate differences:

\hat{\Delta} =\frac{1}{R}\sum_{i=1}^{R}(Y_{Bi}-Y_{Ai}).

This identity follows by distributing the sum across its two terms:

\frac{1}{R}\sum_{i=1}^{R}(Y_{Bi}-Y_{Ai}) = \frac{1}{R}\sum_{i=1}^{R}Y_{Bi} - \frac{1}{R}\sum_{i=1}^{R}Y_{Ai} = \hat{r}_B-\hat{r}_A.

When the matched inputs tend to make both models succeed or fail on the same replicates, the paired differences can be less variable than a comparison of two unrelated sets of runs. Matching does not remove the randomness in simulation, and it does not make a misspecified model correct. A small observed difference could reflect simulation variation; repeating the simulation with more replicates can make the estimated rates more stable. If the difference is important for a decision, report the event counts, the rates, the number of replicates, and the model assumptions rather than presenting a rate alone.

Simulation answers a conditional question: what event rate follows if these model rules and inputs are used? It does not decide which assumptions are realistic. That judgment requires knowledge of the process and, where available, comparison with relevant observed data. A model comparison is most informative when the alternatives are plausible and the results can be traced to a clearly stated difference in assumptions.

Common Mistakes

Changing several things at once. If one model has a different event definition, time period, and risk rule, the rate difference cannot be attributed to one assumption. Keep the setup fixed except for the assumption being examined.
Assuming equal marginal risks imply equal event rates. Components or days can have the same individual risk in two models but different dependence. Events involving several components or consecutive days can then have different rates.
Comparing raw event counts from unequal numbers of trials. Counts alone are not comparable when denominators differ. Calculate each event count divided by its own number of replicates.
Treating the simulated difference as certainty. A simulation estimate varies from run to run. Report it as an estimated rate difference under the specified models, not as proof that one model is true.

Key takeaway: A useful comparison holds the process, event, fixed conditions, and simulation effort constant while changing a clearly identified model assumption. Matched runs help isolate that assumption’s effect, but the resulting simulated rates remain estimates conditional on the models.

Check Your Understanding

Answer each question using the ideas in this tutorial.

Two models are run for 500 replicates. Model A has 32 event occurrences and Model B has 45. Calculate both simulated rates and the difference, taking B minus A.
Why might a shared-condition model produce a different rate for “at least two failures” even when each component has the same marginal failure risk as an independent model?
What does matching the random inputs across two models help control, and what does it not establish?
In a five-day weather simulation, explain why equal wet-day rates do not imply equal probabilities of five consecutive wet days.
Name two details, besides the event counts, that should be reported when presenting a comparison of simulated event rates.