Why Compare Rules by Simulation?
A game can be changed in several ways: its winning condition might be loosened, a round might involve more random outcomes, or one of the outcomes might be scored differently. To understand the consequences, it is useful to simulate the game under both rule sets and compare what happens. A comparison is informative only if both versions are evaluated on the same basis: each simulated trial must be defined clearly, the random inputs must follow the intended model, and the number of trials should be consistent.
The main summaries are often the win proportion and the distribution of other outcomes, such as the number of successful rounds in a game. A higher simulated win proportion does not by itself describe every consequence of a rule change. Two versions could have similar win rates but quite different frequencies of zero, one, or several successes. The distribution helps reveal those differences.
Define the Trial Before Comparing Rules
The rules may differ, but the description of the chance process should make clear what one trial contains. For example, if a game consists of three die rolls, then one trial should contain all three rolls, and the outcome should record what each rule would do with those same rolls. That lets the comparison address the effect of the rule change rather than a change in what counts as a trial.
When both versions use the same chance process, feeding the same random inputs into both rule sets is often useful. For instance, the same three simulated die rolls can be checked against two alternative winning conditions. This creates paired outcomes, making it easy to identify trials on which both rules win, neither wins, or only one wins. Shared inputs are appropriate only when they preserve the intended chance model for each rule. If changing a rule also changes how chance operates, the simulation must represent that change rather than force an unsuitable match.
Keep the number of trials the same for both rules when making a straightforward comparison. Equal run sizes make the summaries easier to compare and give the two estimated win proportions similar Monte Carlo precision. Do not compare raw win counts from runs of different sizes: the count depends on how many trials were run, not just on how often the game was won.
Worked Example: Comparing Two Winning Conditions
Consider a hypothetical game consisting of three rolls of a fair six-sided die. Under Rule A, a roll is a success if it is 5 or 6. Under Rule B, a roll is a success if it is 4, 5, or 6. Under either rule, the player wins the game by getting at least two successes among the three rolls. For a small demonstration, suppose the same eight simulated sets of rolls are used under both rules.
| Trial | Three rolls | Successes, A | Successes, B | Win, A? | Win, B? |
|---|---|---|---|---|---|
| 1 | 1, 2, 4 | 0 | 1 | No | No |
| 2 | 5, 1, 3 | 1 | 1 | No | No |
| 3 | 6, 4, 2 | 1 | 2 | No | Yes |
| 4 | 2, 5, 6 | 2 | 2 | Yes | Yes |
| 5 | 3, 3, 4 | 0 | 1 | No | No |
| 6 | 5, 5, 1 | 2 | 2 | Yes | Yes |
| 7 | 1, 6, 6 | 2 | 2 | Yes | Yes |
| 8 | 4, 2, 3 | 0 | 1 | No | No |
For example, in trial 3, Rule A counts only the 6 as a success, whereas Rule B counts both the 6 and the 4. Rule B wins that trial because its two successes meet the “at least two” condition.
Rule A produces three wins in eight trials; Rule B produces four. The simulated win proportions are
In this small run, Rule B's win proportion is higher by \(0.500-0.375=0.125\), or 12.5 percentage points. This is a simulation result for eight trials, not a guarantee that Rule B will win more often in every set of plays. A larger run would give a more stable estimate.
Compare the Outcome Distributions Too
In the example, a game outcome can also be recorded as the number of successes, from 0 to 3. Counting these outcomes gives more information than the win proportions alone. For Rule A, the eight trials produced three outcomes of 0 successes, two of 1, three of 2, and none of 3. For Rule B, they produced none of 0, four of 1, four of 2, and none of 3.
| Number of successes | Rule A frequency | Rule A proportion | Rule B frequency | Rule B proportion |
|---|---|---|---|---|
| 0 | 3 | 3/8 = 0.375 | 0 | 0/8 = 0 |
| 1 | 2 | 2/8 = 0.250 | 4 | 4/8 = 0.500 |
| 2 | 3 | 3/8 = 0.375 | 4 | 4/8 = 0.500 |
| 3 | 0 | 0 | 0 | 0 |
Each rule's outcome proportions sum to 1, because every simulated trial has exactly one recorded number of successes. The table shows that Rule B shifted outcomes away from zero successes in this run. It also shows that the eight simulated trials happened to include no games with three successes under either rule. That absence in a small simulation does not establish that three successes are impossible; it only reports what occurred in these trials.
Paired Differences and Their Variability
When the same chance inputs are used for both rules, define \(D_i\) for trial \(i\) as the win indicator under Rule B minus the win indicator under Rule A. A win indicator equals 1 for a win and 0 otherwise, so \(D_i\) can be 1, 0, or \(-1\). A value of 1 means only Rule B won that trial; a value of \(-1\) means only Rule A won; and 0 means both rules had the same win status.
The average paired difference is exactly the difference between the two win proportions when both rules are evaluated on the same \(n\) trials:
This follows because summing the trial-by-trial differences subtracts the number of Rule A wins from the number of Rule B wins, and dividing by \(n\) turns those counts into proportions. Pairing also shows which trials account for the observed difference.
There is a useful variance calculation, but its conditions matter. Suppose the simulated trial-pairs are independent and identically distributed, so each \(D_i\) has the same variance and different pairs are independent. Then:
The second equality uses independence, which makes the covariances between distinct \(D_i\)'s zero. The final equality also uses the identical distribution assumption, so every trial-pair has the same variance. Independence alone is not enough for that last step. If independent pairs have different variances, the correct expression is \(\operatorname{Var}(\bar{D})=\sum_{i=1}^{n}\operatorname{Var}(D_i)/n^2\).
Pairing can make a comparison more precise when the two rules tend to have similar results on the same random inputs, but it does not guarantee this. It is a way to organize the simulation and comparison, not a claim that one rule is better.
Worked Example: Estimating a Paired Win-Rate Difference
Suppose a hypothetical simulation uses 1,000 independent trial-pairs, with the same random inputs evaluated under both rules. The paired results are: both rules win on 300 trials, only Rule A wins on 52, only Rule B wins on 197, and neither wins on 451. These counts sum to 1,000.
Rule A wins \(300+52=352\) trials, so \(\hat{p}_A=352/1000=0.352\). Rule B wins \(300+197=497\), so \(\hat{p}_B=497/1000=0.497\). The estimated difference, with Rule B minus Rule A, is
There are 197 values of \(D_i=1\), 52 values of \(D_i=-1\), and 751 values of \(D_i=0\). Thus \(\sum D_i=197-52=145\) and \(\sum D_i^2=197+52=249\). The sample variance of the paired differences is
Under the independent, identically distributed trial-pair assumption, an estimated Monte Carlo standard error for the average difference is
This standard error describes the run-to-run simulation variability in estimating the model's paired win-rate difference. It does not account for whether the game model itself is a good description of a real game. The estimated difference of 0.145 means that, in this run, Rule B's simulated win proportion was 14.5 percentage points higher than Rule A's.
Run Size and a Fair Comparison
Using equal run sizes does not make two estimates identical in precision, but it avoids a simple imbalance and makes the comparison easier to interpret. When run sizes differ, comparing proportions is more appropriate than comparing counts, but the Monte Carlo uncertainty can differ. For a win indicator with estimated win proportion \(\hat{p}\) from \(n\) independent trials, the estimated Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/n}\). This formula assumes trials are generated independently from the same model for that rule.
Worked Example: Unequal Run Sizes
Suppose Rule A is simulated for 10,000 trials and produces 4,000 wins, while Rule B is simulated for 2,000 trials and produces 920 wins. The estimated proportions are
The estimated difference is \(0.460-0.400=0.060\), or 6 percentage points. Comparing the counts 4,000 and 920 would be misleading because Rule A had five times as many trials. The estimated Monte Carlo standard errors are
The smaller Rule B run has the larger estimated Monte Carlo standard error. If the goal is a direct comparison with similar simulation precision, a sensible next step is to simulate more Rule B trials, or to rerun both rules with the same larger run size. The proportions are still computable with unequal run sizes, but the precision difference should not be ignored.
A Practical Comparison Process
Specify all chance inputs and what outcomes will be recorded for each game. Keep this definition consistent across rules.
Check that the simulated random inputs have the correct probabilities and that each rule classifies the outcomes as intended.
Use the same \(n\) trials for each rule when practical, and decide whether common random inputs make a valid paired comparison.
Report the number and proportion of wins under each rule, the difference in proportions, and frequencies or proportions for other relevant outcomes.
State that the summaries are estimates from the specified simulation model and run, and note whether Monte Carlo variability could materially affect the comparison.
Common Mistakes
- Changing the meaning of a trial between rules. If one version is simulated for one roll and the other for a full three-roll game, their win proportions do not answer a direct comparison. Define the complete game trial first.
- Comparing raw counts from unequal runs. A larger count may simply reflect more trials. Compare proportions and account for the different run sizes when discussing precision.
- Reporting only the win proportion. A win rate omits how nonwinning outcomes are distributed. Include other outcome categories when they matter to the game question.
- Assuming paired inputs always improve precision. Pairing is useful when shared inputs correctly represent both rules, but it does not guarantee a smaller standard error. Check the model and describe the comparison accurately.
- Treating a simulated result as a certainty. A simulation estimates outcomes under its stated probability model. A single run does not establish that the observed difference will occur in every future set of plays.
Check Your Understanding
Answer these questions using the definitions and comparison methods in this tutorial.
- A game consists of four die rolls. What should be specified before simulating two different winning conditions?
- In a paired simulation, what does a difference \(D_i=-1\) mean when the difference is defined as Rule B's win indicator minus Rule A's?
- Why is comparing 500 wins from 1,000 trials with 600 wins from 2,000 trials misleading if the counts alone are reported?
- Give one reason to compare outcome distributions in addition to win proportions.
- What assumptions justify writing \(\operatorname{Var}(\bar{D})=\operatorname{Var}(D)/n\) for simulated paired differences?