Why Response Rates Alone Do Not Establish Bias

Why a Response Rate Is Not a Bias Score

A survey report may highlight that 90% of selected people responded. That is useful information: it describes how much of the selected sample provided data. But it does not tell us whether the people who replied gave outcomes like those of the people who did not. To assess possible nonresponse bias, we need to consider both how many responded and how respondents and nonrespondents differ on the variable of interest.

This distinction matters because a high response rate can still leave a consequential gap between the two groups, while a low response rate need not create a gap on a particular outcome. A response rate is one piece of evidence about a survey, not a direct measurement of its bias.

Survey organizations may use more detailed rules for eligibility, partial responses, and unresolved cases. The simple definition here makes the central idea clear: a rate summarizes response counts, whereas bias concerns how missing outcomes affect an estimate. Even a perfectly calculated response rate contains no direct information about the unobserved outcome values of nonrespondents.

How Outcome Differences Enter an Estimate

Consider a probability sample of \(n\) eligible units. Let \(r\) be the number who respond, and let \(n-r\) be the number who do not. For a quantitative outcome, write \(\bar{x}_R\) for the mean among respondents, \(\bar{x}_{NR}\) for the mean among nonrespondents, and \(\bar{x}_S\) for the mean across the entire selected sample. Let \(q=r/n\), the response rate. If we could observe the outcome for every selected unit, the full-sample mean would be a weighted average of the two group means:

A survey that uses only respondents estimates this full-sample mean with \(\bar{x}_R\). Subtracting the full-sample mean from the respondent mean shows exactly how the respondent–nonrespondent gap enters the discrepancy:

This identity gives a useful justification for considering both ingredients. The factor \(1-q\) is the nonresponse fraction, and \(\bar{x}_R-\bar{x}_{NR}\) is the difference in outcome means. If either the nonresponse fraction or the outcome gap is zero, this particular difference between the respondent mean and the full-sample mean is zero. A nonzero gap is scaled by the fraction of the sample whose outcomes are missing.

The identity describes the realized discrepancy within the selected sample, assuming the outcome is defined and measured in the same way for both groups. It is not, by itself, a calculation of the full survey’s bias relative to the population mean. The sample may differ from the population through sampling variability, and other problems such as undercoverage or measurement error may also matter. In practice, the nonrespondents’ outcome mean is often unknown, which is precisely why the response rate alone cannot settle the question.

A Lower Rate Does Not Automatically Mean More Bias

A lower response rate means a larger share of the selected sample has missing outcomes. That can leave more room for nonresponse to affect the estimate, but it does not say whether the missing outcomes actually differ from the observed ones. The outcome gap matters too. If respondents and nonrespondents have similar values on the outcome of interest, a substantial nonresponse fraction may produce little discrepancy in this simple comparison.

For categorical outcomes, the same logic applies to proportions. If respondents and nonrespondents have different proportions in a category, the respondent-only proportion can differ from the proportion across the full selected sample. The fraction who did not respond scales that difference; the response rate does not reveal the difference by itself.

What Evidence Can and Cannot Establish

The response rate is worth reporting because it describes the extent of unit nonresponse and helps readers understand how much of the selected sample supplied data. It is also a prompt to ask what is known about response patterns. For example, one can compare respondents and nonrespondents on information available for both groups, such as age or region, if those data were collected for the full sample. Such comparisons may reveal differences in observed characteristics.

However, matching on observed characteristics does not guarantee matching on the survey outcome. Respondents and nonrespondents could look similar on recorded background variables yet differ on an unrecorded experience or opinion. Conversely, differences in background characteristics do not by themselves determine the size or direction of error for a particular outcome. The relevant question is how the response process relates to the variable being estimated.

A response rate also does not tell us whether a respondent-only estimate is precise. Bias and sampling variability are distinct ideas: an estimate may vary from sample to sample even if nonresponse is not systematically related to the outcome, and a stable estimate may still be systematically different from the target. A high rate should not be treated as proof of low bias, and a low rate should not be treated as proof that an estimate is badly biased.

Common Mistake

Wrong reasoning: “The response rate was 95%, so nonresponse bias must be small.” The rate says that 5% of sampled eligible units did not respond. It does not say whether those units’ outcomes were close to the respondents’ outcomes. A small missing fraction can still matter if its outcomes differ substantially.

Correction: Report the rate as evidence about participation, then ask what is known about respondent–nonrespondent differences for the outcome being estimated. If those outcomes are unknown for nonrespondents, state that uncertainty rather than inferring that the bias is negligible from the rate alone. Apply the same caution in reverse: a low rate raises a question, but does not establish that the respondent estimate is biased.