What Margin of Error Does Not Account For

A Small Margin of Error Is Not a Guarantee of Accuracy

As explained in Margin of Error and Its Meaning, the margin of error for a one-proportion confidence interval is the distance from the sample proportion \(\hat p\) to either endpoint. It summarizes how much the estimate might vary from sample to sample under the method’s assumptions. That is useful information—but it is not a measure of every possible problem in a survey.

A survey can have a small calculated margin of error and still give a misleading estimate. The margin of error does not correct a sample that leaves out important groups, account for people who do not respond, or repair a question that leads people toward a particular answer. Those issues can systematically shift the results away from the population proportion \(p\).

Definition: In the usual one-proportion \(z\)-interval, the margin of error measures estimated sampling variability under the interval method’s conditions. It does not measure or automatically account for bias from how people were selected, who responded, or how the question was worded.

This distinction builds on Sampling Variability Versus Bias in Proportions. Sampling variability is the natural change in \(\hat p\) across random samples. Bias is a systematic tendency for a method to produce estimates above or below \(p\). A margin of error addresses the first, not the second.

What the Calculation Covers

For a one-proportion \(z\)-interval, the margin of error is calculated as the critical value times the estimated standard error:

ME=z^*\sqrt{\frac{\hat p(1-\hat p)}{n}}

The sample size \(n\) and observed proportion \(\hat p\) determine the estimated standard error; the confidence level determines \(z^*\). When the sampling and count conditions support the interval, this calculation describes the method’s sampling uncertainty. It does not put a protective boundary around all errors that might affect a survey result.

“Sampling error” in this setting does not mean that someone made a mistake. It means that a different appropriate random sample could produce a different \(\hat p\). By contrast, if some people have little or no chance of being sampled, or if people who respond differ from people who do not, the sample may be systematically unrepresentative. A larger sample can make the usual margin smaller without removing that systematic shift.

Key distinction: The margin of error is not a maximum total error, and it does not promise that the sample estimate is within that distance of \(p\). It summarizes sampling variability for an appropriate method; it does not measure every source of survey error.

Three Questions for Auditing a Survey

When you see a reported margin of error, look beyond the number. A useful audit asks who could enter the sample, who actually answered, and what the question measured. These questions help identify problems that the margin calculation cannot diagnose or fix.

Who could be selected?
Check whether the sampling frame covers the population of interest. If important groups were left out or were hard to reach, the sample may have undercoverage.

Who answered?
Check whether many selected people did not respond and whether respondents could differ from nonrespondents on the characteristic being studied.

What did the question measure?
Check whether the wording was clear and neutral and whether the response truly represents the characteristic the survey aims to estimate.

A survey might have more than one of these issues. For example, a web survey could miss people without reliable internet access and also receive answers mainly from people with strong opinions. A conventional margin of error does not quantify either problem.

Worked Examples

Worked Example: A Random Sample with a Modest Margin of Error

A fictional town randomly selects 625 adult residents from a complete list of adults. All selected residents answer a clear, neutral question about whether they support extending library hours; 325 say yes. Suppose the town has at least 6,250 adults. Find the conventional 95% margin of error and explain what it does—and does not—say.

The sample proportion is \(\hat p=325/625=0.52\). There are 325 yes responses and \(625-325=300\) no responses, so both observed counts are at least 10. The sample is random, and the 10% condition is met because \(625\leq0.10(6{,}250)=625\). Assume the list covers the target population, everyone responds, and the question measures the intended characteristic.

For 95% confidence, use \(z^*=1.96\). The estimated standard error is \(\sqrt{0.52(0.48)/625}=\sqrt{0.00039936}\approx0.01998\). Thus, \(ME=1.96(0.01998)\approx0.03917\), or about 3.9 percentage points after rounding.

The conventional interval is \(0.52\pm0.03917\), approximately \((0.4808,0.5592)\). Under the stated conditions, the margin summarizes sampling variability for the interval method. It does not guarantee that the interval captures \(p\), and it would not protect the estimate if, for instance, the list omitted a group of adults or the wording prompted answers that did not reflect residents’ genuine views.

Worked Example: A Large Opt-In Survey with a Small Nominal Margin

For illustration, a fictional organization posts an open online poll about a proposed transit plan. Of 1,000 people who choose to respond, 640 support the plan. The organization calculates a 95% margin of error. Explain the calculation and its limitation. For this constructed example, suppose the actual proportion of the target population that supports the plan is stipulated to be 50%.

The respondent proportion is \(\hat p=640/1{,}000=0.64\). The respondent counts are 640 supporters and \(1{,}000-640=360\) nonsupporters, both at least 10. If the target population has at least 10,000 people, the numerical 10% check for a sample of this size would be met. But the random condition is not met: people decide for themselves whether to enter the poll.

The usual calculation gives \(\sqrt{0.64(0.36)/1{,}000}=\sqrt{0.0002304}=0.01518\). Multiplying by 1.96 gives \(1.96(0.01518)\approx0.02975\), or about 3.0 percentage points. This is only a nominal margin from applying the formula; because the responses are opt-in rather than a random sample, it does not establish a valid conventional 95% margin of error for the target population.

In the stipulated scenario, the poll’s 64% result is 14 percentage points above the population’s 50% support proportion. People with strong opinions or a special interest in transit may be more likely to respond. The small nominal margin does not cover that selection bias. It shows why increasing the number of volunteers cannot, by itself, make a self-selected poll representative.

Worked Example: Nonresponse After Random Selection

A fictional community randomly selects 500 adults from a population of 20,000 to ask about a neighborhood safety program. Only 200 adults respond, and 140 of the respondents support the program. Calculate the conventional 95% margin using the respondents and explain what nonresponse changes.

Among respondents, \(\hat p=140/200=0.70\). The observed counts are 140 supporters and \(200-140=60\) nonsupporters, each at least 10. The original selection was random, and \(500\leq0.10(20{,}000)=2{,}000\), so the 10% condition for the original sample is satisfied. However, only 200 of the 500 selected adults responded: the response rate is \(200/500=0.40\), or 40%.

Using the respondent count \(n=200\), the nominal calculation is \(\sqrt{0.70(0.30)/200}=\sqrt{0.00105}\approx0.03240\). Then \(ME=1.96(0.03240)\approx0.06351\), or about 6.4 percentage points. The arithmetic describes sampling variability only under the assumption that the respondents provide an appropriate basis for inference. Random selection of the original 500 does not automatically make the 200 respondents representative.

For a hypothetical illustration, suppose 90 of the 300 nonrespondents would have supported the program. Then all 500 selected adults would include \(140+90=230\) supporters, a proportion of \(230/500=0.46\). The respondent proportion of 70% is 24 percentage points higher. We do not know the nonrespondents’ views in an actual survey, but if they differ from respondents, the respondent estimate can be biased. The 6.4-point calculation does not account for that difference.

Worked Example: Wording Can Change the Responses

A fictional random sample of 600 residents is split at random into two groups of 300. One group is asked, “Do you support adding a protected bike lane on Pine Street?” Of that group, 147 say yes. The other group is asked, “Do you support adding a protected bike lane on Pine Street to make cycling safer?” Of that group, 204 say yes. Find each group’s conventional 95% margin of error and discuss what the different wording means.

For the neutral wording, \(\hat p=147/300=0.49\), with 147 yes and \(300-147=153\) no responses. Its estimated standard error is \(\sqrt{0.49(0.51)/300}=\sqrt{0.000833}\approx0.02886\). The margin is \(1.96(0.02886)\approx0.05657\), or about 5.7 percentage points.

For the wording that mentions safety, \(\hat p=204/300=0.68\), with 204 yes and \(300-204=96\) no responses. Its estimated standard error is \(\sqrt{0.68(0.32)/300}=\sqrt{0.00072533}\approx0.02693\). The margin is \(1.96(0.02693)\approx0.05279\), or about 5.3 percentage points. In each group, both response counts are at least 10. If the initial sample was random from at least 6,000 residents, the 10% condition is met for the full sample; random assignment gives each wording group 300 residents.

The observed support proportions differ by \(0.68-0.49=0.19\), or 19 percentage points. The wording may have influenced the answers: mentioning safety could make a particular benefit more salient. The margins quantify sampling variability for each wording’s response proportion under the stated design; they do not adjust answers to what residents would have said to a different, more neutral question. The survey measures responses to the wording actually used.

Common Mistakes and AP Exam Tips

Calling the margin a measure of total error. A margin of 3 percentage points does not mean that the estimate is at most 3 points from \(p\). It summarizes sampling variability under appropriate conditions, not every possible source of error.
Assuming a large sample fixes bias. A larger \(n\) usually reduces the calculated margin, but it does not make a volunteer sample representative or make nonrespondents answer. A precise estimate can still be systematically off target.
Treating a random initial sample as proof that respondents are representative. Nonresponse can change who is represented in the data. Describe the response problem and avoid claiming that the usual margin covers it.
Using a conventional margin without checking the sampling method. If participants self-select, say that the random condition is not met. You may show what the formula produces as a nominal calculation, but do not present it as a justified margin of error for the population.
Ignoring the exact question. A confidence interval concerns the proportion answering a particular question in a particular way. If wording is leading or ambiguous, the interval cannot turn those answers into an unbiased measure of a different characteristic.

AP Exam Tip: State what the margin of error measures, then identify important limitations of the survey method. A strong response distinguishes random sampling variability from possible bias due to coverage, nonresponse, or wording. Do not claim that the margin measures or corrects those sources of bias.

Key Takeaway

A one-proportion margin of error summarizes how much \(\hat p\) may vary from sample to sample under the interval method’s assumptions. It is not a measure of total survey error. A survey can have a small margin and still be biased if its sampling frame misses people, its respondents differ from nonrespondents, or its wording affects the answers.

Key takeaway: Read a margin of error as a statement about sampling variability, not as a guarantee of accuracy. To judge a survey, also ask who could be selected, who responded, and what the question actually measured.

Check Your Understanding

For each situation, explain what the margin of error can address and identify any problem it does not address.

A random survey reports 52% support with a 4-percentage-point margin of error. Explain why this does not guarantee that the population proportion is between 48% and 56%.
An online poll has 8,000 voluntary responses and a very small calculated margin. What sampling concern remains?
A random sample of 400 people is contacted, but only 120 respond. Why might the margin calculated from the respondents fail to account for nonresponse bias?
A survey asks whether people support a “safer, cleaner” park plan. What can a margin of error say about the answers, and what can it not fix about the wording?
A poll’s sample size is increased fourfold, but its opt-in method stays the same. What happens to its nominal sampling margin, and why might the poll remain biased?