Nonresponse Bias

When Selected People Do Not Respond

In Undercoverage Bias, you learned that a sampling frame can leave some members of the population out before a sample is selected. Nonresponse is different: people are selected for the sample, but some do not provide the requested information. They might refuse, be unavailable, or start but not complete the survey.

A random sample is a strong starting point, but the results can still be biased if the people who respond differ from those who do not on the variable being measured. A survey about sleep, for example, might miss selected people who work overnight and are hard to reach during the survey team’s calling hours. The key question is not simply how many people declined; it is whether responding and not responding are related to the answers the survey seeks.

A refusal is one kind of nonresponse. It is not the same as a voluntary response sample. In voluntary response, people choose themselves into the sample, often by answering an open invitation. In a survey with nonresponse, people are selected first, and some of those selected do not participate. As discussed in Undercoverage Bias, undercoverage happens earlier still: people are missing or inadequately represented in the frame used to select the sample.

Response Rate: Useful, but Not a Bias Measure

The response rate describes what fraction of selected people completed the survey. For a simple survey in which everyone selected is eligible, calculate it as the number of completed responses divided by the number selected. A survey protocol should specify how to handle people who are ineligible or whose eligibility cannot be determined; the examples here assume that everyone selected is eligible.

A low response rate means that many selected people did not provide data, so there is greater reason to investigate possible nonresponse bias. But the response rate alone cannot tell you whether bias occurred or which way an estimate may be off. A low-response survey could have little bias if respondents and nonrespondents are similar on the variable of interest. A high-response survey could still be biased if the few who do not respond differ substantially.

This is similar to the caution about undercoverage: identify the people who are missing from the results and consider how they might differ on the measured variable. With nonresponse, those missing people were in the selected sample, but their answers are unavailable. Unless the scenario supplies relevant information, avoid claiming that refusals definitely raised or lowered an estimate.

Worked Example: Refusals in a Park Survey

Practical Ways to Reduce Nonresponse

Reducing nonresponse starts with making participation clear, respectful, and manageable. A well-designed contact plan gives selected people reasonable opportunities to respond without pressuring them. The goal is not to obtain answers at any cost; participation should remain voluntary and consistent with the ethical principles discussed in Ethical Considerations in Data Collection.

• Explain the request clearly. Briefly state who is conducting the survey, why the person was selected, how long participation is expected to take, and how the information will be used.
• Make responding convenient. Offer an accessible format, such as a short online survey or a phone option when appropriate. Provide a reasonable range of times to respond.
• Follow up respectfully. Send or make a limited number of reminders to people who have not responded. Different contact times may reach people who were unavailable earlier.
• Protect privacy honestly. Explain confidentiality protections accurately. Do not promise anonymity or security that the survey cannot guarantee.
• Consider a modest incentive. A small, appropriate incentive may encourage participation, but it should not be coercive or presented as a guaranteed way to eliminate bias.

A follow-up can improve participation and provide clues about nonresponse. For example, a second contact might reach people who were unavailable the first time. But people who respond after several attempts might still differ from those who never respond. Replacing nonrespondents with easier-to-reach people is not a reliable fix: replacements may have different characteristics and do not supply the missing selected individuals’ answers.

Use Known Differences Carefully

Sometimes a survey team knows that response rates vary across groups—for example, by grade, region, or a characteristic already recorded for everyone in the sample. Comparing these rates can identify where nonresponse is concentrated. If those groups may also differ on the survey variable, the respondent results could overrepresent one group and underrepresent another.

When a group’s size in the sample is known, an analyst may adjust a respondent estimate to reflect the sample’s group proportions rather than the respondents’ proportions. This approach can help if the groups are meaningfully related to the response variable and if respondents within each group are reasonably representative of nonrespondents in that same group. It cannot correct for unknown differences within groups, so an adjusted result still needs cautious interpretation.

Common Mistakes and AP Exam Tips

• Treating the response rate as the amount of bias. A response rate tells you how many selected people answered, not how different their answers are from nonrespondents’ answers. Explain what the rate suggests and what information is still missing.
• Claiming a direction without evidence. Say that nonresponse could bias a result when you do not know how nonrespondents differ. State a likely direction only when the scenario supports it.
• Confusing nonresponse with voluntary response or undercoverage. A full-credit answer identifies the mechanism: selected people do not respond, people choose themselves into a sample, or the frame leaves people out.
• Assuming reminders eliminate the problem. Reminders may increase participation, but those who continue not to respond may still differ from respondents. Describe the improvement without claiming that it guarantees an unbiased result.
• Reporting respondents’ answers as the population’s answers. Say whom the observed proportion describes. Generalizing to the population requires considering both how the sample was selected and whether nonresponse could matter.
• Using an adjustment without stating its assumption. If you reweight groups, explain that the adjustment relies on respondents representing nonrespondents within the groups used. It cannot account for differences the groups do not capture.

For a strong AP response, identify the selected sample, describe who did not respond, and connect possible differences to the variable being measured. Use precise, cautious language: “The refusals may bias the estimate if those who refused differ from respondents in their support for the proposal.” If the scenario establishes how they differ, explain the likely direction; otherwise, do not invent one.