Judging If a Result Is Surprising by Simulation

Compare an Observed Result with a Simulated Distribution

In Reading a Dotplot of Simulation Results, you learned to count simulated results in a specified tail. Now we use that idea to ask a bigger question: does an observed result seem surprising if a particular chance model is true?

Suppose someone flips a coin 10 times and gets 9 heads. That result is possible even for a fair coin, but is it common or rare? To answer, we can simulate many sets of 10 flips under a fair-coin model and compare the observed count with the simulated distribution of head counts.

Here, the statistic is the number of heads in 10 flips. The model says that each flip has probability 0.5 of heads and that the flips are independent. One complete trial is 10 flips, and one result from a trial is its head count. This follows the simulation planning ideas from Setting Up a Simulation Model and Describing a Simulation in Words.

A simulation could represent each flip with one random digit: digits 0–4 mean heads and 5–9 mean tails. Generate 10 digits for one trial, count the heads, and repeat the full trial many times. As explained in Common Errors in Designing Simulations, the complete 10-flip trial—not an individual flip—must be repeated and counted.

Choose the Relevant Tail

The tail is the part of the simulated distribution containing results at least as extreme as the observed result. Which tail to count depends on the question. If the question is whether the coin produced unusually many heads, count results equal to or greater than the observed number. If it asks about unusually few heads, count results equal to or less than the observed number.

Include results equal to the observation. For 9 heads, a result of 9 is at least as extreme as 9 in the “many heads” direction, and 10 is even more extreme. Counting only results greater than 9 would incorrectly leave out the observed result itself.

This estimate is not the exact probability. As discussed in Estimating a Probability from Simulation Results and How Many Trials Are Enough, simulation results vary from run to run. More trials generally make a relative-frequency estimate less variable, but do not guarantee any one run will be closer to the exact probability.

What “Unusual” Means in This Comparison

“Unusual” is a judgment about how much of the simulated distribution lies at least as far into the relevant tail as the observation. If only a small fraction of simulated trials reach or exceed the observed result, then the result would not commonly occur under the model. If a large fraction do, the result is not especially surprising under that model.

The simulation itself gives a relative frequency, not a universal boundary between unusual and ordinary. For example, the estimate 0.013 in the worked example describes the simulated tail proportion. It does not, by itself, say what action to take or establish that the model is wrong. Later work on significance will formalize how a small tail probability is used as evidence against a chance model.

Keep the comparison tied to the model being tested. Saying “9 heads is unusual” without qualification is incomplete. A careful statement says that 9 or more heads appears unusual if the coin is fair and the flips are independent, based on the simulated tail proportion. The simulation evaluates results under those assumptions; it does not independently verify them.

One Direction or Either Direction?

Sometimes the question predicts a direction in advance, such as whether a coin produces unusually many heads. Then the appropriate comparison is one-sided: count results in that direction. Other questions ask whether the result is surprising in either direction—unusually many or unusually few heads. For those questions, both tails may matter.

The fair-coin model distribution is centered at 5 heads, though the empirical simulated distribution need not be exactly centered there. If the observed result is 9 heads and the question is whether the result differs unusually from the center in either direction, results at least as far from 5 as 9 are 9 and 10 heads, along with 1 and 0 heads. This compares equal distances from the center: 9 is four above 5, while 1 is four below 5.

Common Mistakes and AP Exam Tips

• Counting the wrong tail. If the question asks about unusually many heads, count outcomes at or above the observation. If it asks about unusually few, count outcomes at or below it. If it asks about either direction, explain which outcomes in both tails qualify.
• Leaving out the observed value. “At least as extreme” includes the observed statistic. For 9 heads, the upper-tail count includes both 9 and 10.
• Dividing by the wrong total. Divide the qualifying simulated trials by all completed trials, not only by the trials in one part of the table. In the first example, the denominator is 1,000.
• Calling the estimate an exact probability. A simulation provides an estimate that can vary from run to run. Say “the estimated proportion in this simulation” or “about 1.3%,” rather than claiming the simulation establishes an exact probability.
• Claiming that a rare result proves the model false. A rare result may provide evidence against the model, but it remains possible under the model. State the conclusion conditionally and in context.
• Choosing a direction after seeing the result without explanation. The direction must match the question. If the question concerns either unusually many or unusually few, account for both tails rather than reporting only the tail that makes the result seem rarer.

A full-credit explanation identifies the model, names the observed statistic, justifies the tail or tails, shows the count and denominator, and interprets the simulated proportion in context. For example: “Assuming a fair coin and independent flips, 13 of 1,000 simulated sets had at least 9 heads, an estimated proportion of 0.013. Thus, 9 heads appears unusual under this model, although it does not prove the coin is unfair.”