A Model’s Answer Is Conditional
In Judging Whether a Normal Model Fits, you considered whether the shape of data supports using a normal model. Probability models have broader limitations, too: even when the calculations are correct, the model may not give a reliable answer if its inputs are uncertain, its data are biased, or its assumptions do not match the process.
A model is a simplified description of chance. Its probability is conditional on the outcomes, parameter values, and assumptions used to construct it. For example, a binomial calculation answers a question under a fixed number of trials, a constant success probability, and independent trials. It does not prove that those conditions describe a particular real situation.
Three sources of limitations deserve particular attention. A small sample may give an unstable estimate of a probability. A biased sample may systematically misrepresent the group being studied, even when it is large. And a model may rely on assumptions—such as independence or a constant chance of success—that do not hold in context.
These problems differ from ordinary chance variation. If a suitable model assigns a low probability to an outcome, that outcome can still occur. One unusual result does not, by itself, show that a model is wrong. In contrast, a biased sample or unrealistic assumption calls the model’s suitability into question before interpreting its calculated probabilities.
Small Samples Make Estimated Probabilities Uncertain
Many models use a probability \(p\) estimated from observed data. The observed relative frequency is a reasonable starting point, but a small sample may not represent the long-run probability well. If a pilot test includes only a few trials, one or two outcomes can change the estimate substantially.
The standard deviation of a sample proportion under a model with probability \(p\) is \(\sqrt{p(1-p)/n}\), as covered in Normal Models for Sample Means and Proportions. This quantity describes sample-to-sample variation when the model’s \(p\) is treated as known. When \(p\) itself was estimated from a small sample, the formula does not remove the uncertainty in that estimate. It can, however, help illustrate why small samples tend to produce less stable proportions.
A helpful technique is sensitivity analysis: calculate or compare the answer under more than one plausible input value. If a small change in the estimated probability leads to a large change in the answer, the model’s conclusion is sensitive to that input. Sensitivity analysis does not tell you which input is correct; it shows how much the conclusion depends on the choice.
Worked Example: A Small Pilot Test for Product Defects
A fictional manufacturer tests 20 items in a pilot run and finds 2 defective items. The estimated defect probability is \(2/20=0.10\). A manager asks for the probability of finding no defective items in the next 20. First calculate the answer under the estimated probability, then examine how sensitive it is to \(p\).
Let \(X\) be the number of defective items among the next 20. The question is \(P(X=0)\), using the pilot estimate \(p=0.10\).
Use a binomial model only if there are 20 fixed trials, each item has the same defect probability, and outcomes are independent. These assumptions may be reasonable for a stable process, but the estimate \(p=0.10\) comes from only 20 pilot items, so it may be imprecise.
Under the stated model, the probability of zero defects is \((1-p)^{20}\). Substitute the pilot estimate and calculate.
The calculated probability is conditional on \(p=0.10\) and the binomial assumptions. Compare alternative values to see whether the answer depends strongly on the estimated probability.
If the actual defect probability were \(0.02\), the same calculation would give \(0.98^{20}\approx0.6676\). If it were \(0.25\), it would give \(0.75^{20}\approx0.0032\). These are not claims that either alternative is correct; they show that the probability of a defect-free run changes greatly with \(p\).
Conclude. Using the pilot estimate, the model gives a probability of about \(0.1216\) for no defects in the next 20 items. But 2 defects in 20 items is a limited basis for treating \(0.10\) as a known, stable probability. The calculation is valid under its stated conditions; its real-world reliability depends on how well the pilot estimates the ongoing process and whether that process remains stable.
Bias Can Persist Even in a Large Sample
A sample is biased when its method of selection or response systematically favors some members or outcomes over others. A voluntary-response survey, for example, may attract people with strong opinions. A convenience sample may overrepresent people who are easiest to reach. In either case, the observed proportion can differ systematically from the population proportion.
Increasing the number of observations can reduce random variation in a properly selected sample, but a larger biased sample can still give a misleading picture. This is why sample size and sample quality answer different questions: size affects how much random variation to expect, while the selection method affects whether the sample represents the target group.
The 10% condition, discussed in The 10% Condition for Independence in Binomial Settings, concerns dependence when sampling without replacement. Meeting that condition does not establish that a sample is random or representative. A survey can meet a numerical sample-size condition and still suffer from selection bias or nonresponse bias.
Worked Example: A Voluntary School Survey
A fictional school has 1,200 students. An online survey asks whether students support a proposed change to the lunch schedule. The link is posted on a student activities page, and 120 students choose to respond; 102 support the change. An analyst wants to use the observed proportion to describe all students.
The respondent proportion is \(102/120=0.85\), or 85%. But the 120 students were not chosen at random: they found the survey link and decided to respond. Students who use the activities page or feel strongly about lunch scheduling may be more likely to participate. The respondent group may therefore differ from the full student population in ways related to their opinions.
For comparison, suppose someone proposes a binomial model with \(n=120\) and \(p=0.60\). Under that model, the expected number of supporters would be \(np=120(0.60)=72\), and the standard deviation would be \(\sqrt{120(0.60)(0.40)}=\sqrt{28.8}\approx5.367\) supporters. The observed 102 is 30 above that model’s mean. That comparison describes what the proposed model predicts, but it does not make the voluntary respondents a random sample or justify treating the model as a valid account of all 1,200 students.
The respondent count is exactly 10% of the school population, but that numerical fact does not fix the selection problem. The 10% condition is not a test for random selection. A probability calculation based on these responses cannot by itself establish the level of support among all students.
Conclusion. The survey found that 85% of its voluntary respondents supported the change. Because participation was self-selected, this result may not represent all students. A larger number of voluntary responses would not automatically remove that limitation; a more representative way to select students is needed for a stronger population estimate.
Unrealistic Assumptions Can Change the Distribution
A model’s assumptions determine not only its inputs but also how probabilities are combined. In a binomial model, independence means one trial’s outcome does not change the probability of another trial’s outcome. A fixed success probability means the model assigns the same chance to every trial. Shared conditions, changing procedures, or groups with different success rates can undermine one or both assumptions.
A particularly important pattern is clustering: outcomes in the same group tend to be alike because they share a condition. In that setting, the individual outcomes may have the same overall success probability while still being dependent. Matching the average probability is not enough to guarantee that the model’s distribution of counts is right.
Worked Example: Sensor Tests with a Shared Environment
A fictional lab tests 10 sensors in one batch. The proposed model says each sensor passes independently with probability \(0.90\). However, all 10 sensors share the same calibration environment. Suppose the environment is good with probability \(0.80\), in which case each sensor passes with probability \(0.98\); it is poor with probability \(0.20\), in which case each sensor passes with probability \(0.58\). Within either environment, assume sensor results are independent.
For one sensor, the overall pass probability is \(0.80(0.98)+0.20(0.58)=0.784+0.116=0.90\). Thus the proposed model has the correct marginal probability for an individual sensor. But the outcomes in the batch are not independent: learning that the environment is poor makes failures more likely for every sensor in that batch.
Under the proposed independent model, the probability that all 10 sensors pass is \(0.90^{10}\approx0.3487\). Under the shared-environment description, all 10 pass either in a good environment or in a poor one. The probability is therefore \(0.80(0.98^{10})+0.20(0.58^{10})\). Since \(0.98^{10}\approx0.8171\) and \(0.58^{10}\approx0.0043\), this is \(0.80(0.8171)+0.20(0.0043)\approx0.6545\).
The two answers differ substantially even though both descriptions give an individual sensor a 0.90 chance of passing. The independent model underestimates the chance that the entire batch passes in this example because it does not represent the shared environment. It also fails to reflect how results can cluster under a common condition.
Conclusion. A fixed marginal probability does not establish independence. Before using the binomial model for a group of sensors, consider whether shared conditions connect their outcomes. A model that omits a relevant shared influence may give unreliable probabilities for group events, even when its per-sensor probability looks appropriate.
How to Evaluate a Model’s Limits
When a probability model gives an answer that matters, separate three questions. First, are the model’s outcomes and parameter values credible for the situation? Second, does the process support the assumptions needed for the chosen model? Third, how sensitive is the answer to uncertain inputs or assumptions?
Use the context and the way the data were collected to answer these questions. Compare model predictions with observed results, as in Using Data to Check a Probability Model, but interpret a discrepancy cautiously. Random variation can create differences between observed and expected results. Conversely, results that happen to agree with a model do not prove that its assumptions are true.
- Input quality: Is the probability known from the process, or estimated from a small or unrepresentative sample?
- Selection: Were the observations selected in a way that represents the group or process of interest?
- Assumptions: Are outcomes mutually exclusive and exhaustive where required, and are claims such as equal likelihood, a fixed probability, or independence plausible in context?
- Sensitivity: Would a reasonable change in an uncertain input or assumption substantially change the answer?
- Scope: Does the model describe the specific group, time period, and process the question asks about?
These checks do not require you to reject every simplified model. Models are useful precisely because they simplify. The goal is to identify which simplifications matter for the question and to avoid presenting a conditional model result as certainty about reality.
Common Mistakes and AP Exam Tips
- Calling a small sample automatically biased. A small sample can be representative but give a noisy estimate; bias concerns systematic differences caused by selection or response. Explain which problem the sampling method raises.
- Assuming a large sample removes bias. More responses can reduce random variation, but they do not necessarily correct a systematic selection problem. State why the way people entered the sample matters.
- Treating a model probability as guaranteed. A model describes the chance of an event under stated conditions; it does not promise that the event will occur at that rate in every short run.
- Confusing one surprising result with proof that a model is wrong. A low-probability outcome can occur. Describe it as evidence to consider, not conclusive proof by itself.
- Checking only the value of \(p\). The correct individual success probability does not ensure a binomial model is appropriate if outcomes are dependent or probabilities change between trials.
- Reporting a calculation without its limitation. A full-credit response says what assumptions the result depends on and connects any concern to the real situation.
Key Takeaway
A probability model’s result depends on the quality of its inputs and the realism of its assumptions. Small samples can leave estimated probabilities uncertain, biased samples can misrepresent a population, and shared conditions can make outcomes dependent. Distinguish those limitations from ordinary chance variation, and explain what the model’s answer means conditionally.
Check Your Understanding
For each situation, identify the main limitation and explain what can or cannot be concluded.
- A company estimates a failure probability from 8 tested products and uses it to predict failures in the next 500. Why might the probability input be uncertain?
- A city asks for opinions by posting a survey link on a neighborhood discussion board. Why might a large number of responses still fail to represent all residents?
- In a binomial model, why is having the same probability of success on each trial not enough if outcomes are dependent?
- A model assigns a probability of 0.02 to an event, and the event occurs once. Does that single occurrence prove the model is wrong? Explain.
- What does sensitivity analysis reveal when plausible values of an estimated probability give very different answers?