Use a Simulated Distribution as a Comparison
In Using Data to Check a Probability Model, you compared observed counts with expected counts. That comparison is a useful first step, but a count’s distance from its expectation does not, by itself, tell us how often a difference that large might occur under the model. Simulation helps answer that question by repeatedly generating results as if the model were true.
Suppose a model describes a chance process with 20 trials and a success probability of 0.5. A simulation can generate one possible count of successes, repeat that process thousands of times, and display the resulting counts. The collection of simulated counts is a simulated distribution. It shows the variation in results the model can produce over repeated sets of trials.
If the event is “the count is at least as large as the observed count,” the simulation proportion estimates the chance, according to the model, of getting a result at least that high. A small proportion indicates that results like the observed one are uncommon under the model. A larger proportion indicates that the observed result is not especially unusual by this comparison.
A simulation proportion is an estimate, not an exact probability. It depends on the random results generated in that particular set of repetitions. More repetitions generally make the estimate less sensitive to the luck of one simulation run, although they do not fix a model that fails to represent the real process.
Plan the Comparison Before Counting
To interpret a simulated distribution, first decide what result would count as “at least as unusual” as the observation. If the question concerns an unusually high count, count simulated values equal to or greater than the observed count. If it concerns an unusually low count, count values equal to or less than the observation.
Sometimes either direction would be a concern. For example, if a model predicts a count near 18, an observed count far below 18 or far above 18 could both call the model into question. In that case, define a measure of distance from 18 and count simulated results at least as far from 18 as the observed result. The direction and measure of extremeness should match the question; do not choose a tail only after seeing which one makes the result look more surprising.
For a binomial model, as covered in Choosing Between Binomial and Other Models and Independence Assumptions in Real Settings, each simulated repetition must use the fixed number of trials, the same success probability on every trial, and independent trials. If those assumptions do not fit the real process, the simulation may be precise about the wrong model.
Define the count, give the model’s relevant probabilities, and report the observed value.
Use the same number of trials and the same chance assumptions in each repetition.
Use the relevant upper tail, lower tail, or a clearly stated distance measure.
Divide the number of marked results by the number of repetitions, then explain what that estimated proportion says about the observation under the model.
Worked Example: A High Number of Heads
Worked Example: A High Number of Heads
In a fictional classroom activity, a student flips a coin 20 times and gets 15 heads. Consider whether this high count is unusual under a model that treats the coin as fair and the flips as independent.
State. Let \(X\) be the number of heads in 20 flips. Under the fair-coin model, \(X\sim B(20,0.5)\). The observed count is 15 heads.
Plan. Because the observation is high, count simulated repetitions with \(X\geq15\). Each repetition must consist of 20 independent flips, with probability 0.5 of heads on every flip. We will use the proportion from 10,000 repetitions as an estimate of the model probability of getting at least 15 heads.
Do. A simulation of 10,000 repetitions produces 218 counts of 15 or more heads. Thus the simulation proportion is:
The simulation estimates that about 2.18% of sets of 20 flips would produce at least 15 heads under this model. The endpoint is included: repetitions with exactly 15 heads are counted along with those with more than 15.
Conclude. Getting 15 or more heads appears unusual under the fair-coin model, based on this simulation. This is evidence that the result is not typical under the model, but it does not prove the coin is unfair. The simulation estimates a model-based chance; it does not establish whether the coin actually meets the model’s assumptions.
Worked Example: A Count That Is Not Especially Unusual
Worked Example: A Count That Is Not Especially Unusual
Imagine a fictional snack company’s planning exercise assumes that 20% of customers choose a particular fruit flavor. In a group of 50 customers, 13 choose it. Use a simulation to judge whether 13 is unusually high under the claimed model.
State. Let \(X\) be the number of customers, out of 50, who choose the fruit flavor. The model is \(X\sim B(50,0.20)\), and the observed count is 13.
Plan. The question is about an unusually high count, so use the event \(X\geq13\). Simulate 50 independent customer choices per repetition, assigning probability 0.20 to the flavor on each choice. Suppose 10,000 repetitions are run.
Do. In the simulation, 1,094 repetitions have 13 or more customers choosing the flavor. The estimated proportion is:
The model’s expected count is \(50(0.20)=10\), so the observed count is 3 above the expectation. More importantly for this comparison, about 10.94% of the simulated groups have a count at least as high as 13. That is not a very small proportion.
Conclude. A count of 13 does not appear especially unusual under the stated model: simulated counts of 13 or more occur in about 10.94% of repetitions. These results are reasonably consistent with the model in this respect. They do not prove that exactly 20% of all customers choose the flavor.
Worked Example: A Low Count Compared with Either Direction
Worked Example: A Low Count Compared with Either Direction
A fictional seed supplier’s model says that each seed has probability 0.60 of sprouting. A gardener tests 30 seeds and observes 12 sprouts. The gardener wants to know whether a count this far from the model’s expectation, in either direction, would be unusual.
State. Let \(X\) be the number of seeds that sprout. The claimed model is \(X\sim B(30,0.60)\), and the observed count is 12. The expected count is \(30(0.60)=18\), making the observed count 6 below the expectation.
Plan. Since a count either far below or far above 18 could be concerning, define “at least as extreme” as a count at least 6 away from 18. That means counting simulated results with \(X\leq12\) or \(X\geq24\). Simulate 30 independent seeds per repetition, each with sprouting probability 0.60.
Do. Suppose 10,000 repetitions are simulated and 402 have counts in either of those two tails. The estimated proportion is:
Thus, about 4.02% of simulated counts are at least 6 away from 18. The observed count of 12 is included in the lower tail because \(12\leq12\); counts of 24 or more are included in the upper tail because they are at least 6 above 18.
Conclude. Under the stated model, results at least this far from the expected count in either direction appear unusual, with an estimated simulation proportion of 0.0402. This comparison gives a reason to examine the model and the seed-testing process. It does not identify which assumption, if any, is inaccurate.
Reading the Simulation Carefully
The simulation proportion describes the results of the repetitions that were run. For example, 0.0402 means that 402 out of these 10,000 simulated results met the stated extremeness rule. It is an estimate of the probability under the model, not a statement that the model assigns exactly that probability.
A simulation with more repetitions can give a more stable estimate. With 10,000 repetitions, the smallest positive proportion that can be reported is \(1/10{,}000=0.0001\). If no simulated result meets the event, the estimated proportion from that run is 0, but this does not show that the event is impossible under the model. It may be very rare, or the simulation may simply not have generated it.
Be precise about the comparison. “At least as extreme” must include the observed value when appropriate, and a two-sided comparison needs an explicit rule for what counts as equally far or farther from the model’s center. A simulation of only unusually high counts cannot answer whether the observation is unusual in either direction.
Common Mistakes and AP Exam Tips
- Using the wrong tail. If the question concerns a high count, count results equal to or above the observation. If it concerns a low count, count results equal to or below it. For either-direction concern, state a two-sided extremeness rule.
- Dividing by the wrong number. The denominator is the total number of simulation repetitions, not the number of trials in each repetition. If 218 of 10,000 repetitions qualify, the proportion is \(218/10{,}000\), not \(218/20\).
- Leaving out the observed value. “At least as high as 15” means 15 and above. Write the event explicitly so the endpoint is clear.
- Calling the estimate an exact probability. A simulation proportion estimates a probability under the model. Say “the simulation estimates” or “about this proportion of repetitions,” rather than reporting the result as an exact model probability.
- Claiming that a small proportion proves the model false. A small proportion indicates that the observation is unusual if the model and its assumptions are correct. It is evidence to investigate, not proof of a particular explanation.
- Simulating a different process. If the real setting has a fixed number of trials, the simulation should use that number. Its probabilities and dependence assumptions should also represent the claimed model.
Key Takeaway
An expected count gives a model’s center, while a simulated distribution shows the range of variation the model can produce. Counting simulated results at least as extreme as an observation turns that comparison into an estimated probability. The smaller the simulation proportion, the more unusual the observation appears under the stated model—but the conclusion remains conditional on the model and its assumptions.
Check Your Understanding
Use the simulated distribution and a clearly defined comparison event to interpret each result.
- A model predicts a success probability of 0.5 for 16 independent trials. An observed count is 12 successes. If the question concerns a high count, which simulated values should be counted as at least as extreme?
- In 5,000 repetitions, 175 simulated counts are at least as high as the observed count. Calculate the simulation proportion and interpret it in context.
- A model has expected count 18, and the observed count is 13. Give a rule for identifying results at least as far from 18 in either direction.
- Why does a simulation proportion of 0 from a limited number of repetitions not prove that an event is impossible?
- Name two details of a binomial chance process that each simulation repetition must reproduce.