Nonresponse Bias in Surveys

Why Participation Can Change a Survey Result

A survey may begin with a carefully selected sample and still produce a misleading estimate if some sampled people do not participate. The concern is not simply that fewer people answered than were invited. It is that the people who answered may differ from those who did not in ways related to the quantity being measured. If, for example, people with particularly high or low values of the outcome are more likely to respond, the average among respondents may not represent the population average.

This is a different problem from undercoverage. Undercoverage occurs when some members of the target population are absent from the sampling frame or have little chance of selection. With nonresponse, people are selected, but some do not take part. A survey can avoid undercoverage and still face nonresponse bias.

The phrase “in a way related to the outcome” is essential. If respondents and nonrespondents differ on some characteristic that has no meaningful connection to the survey outcome, that difference alone does not establish nonresponse bias for that outcome. Conversely, even a sample selected using a probability design can be at risk: known selection chances do not make participation unrelated to the outcome.

Participation and Outcome Differences Together

A simple equal-probability sample illustrates how the risk works. Imagine separating everyone in the initially selected sample into respondents and nonrespondents. Let \(n_R\) be the number who respond, \(n_N\) the number who do not, and \(n=n_R+n_N\) the original sample size. Let \(\bar{x}_R\) be the respondents’ mean outcome and \(\bar{x}_N\) the nonrespondents’ mean outcome. If the outcome were available for everyone in the selected sample, its mean would be

Define the response fraction as \(r=n_R/n\), so that \(1-r=n_N/n\). Substituting these fractions gives a weighted-average expression for the full selected sample.

The respondents’ mean differs from the full selected sample mean by

This derivation shows two ingredients of the difference: the fraction who do not respond, \(1-r\), and the difference in the outcome means, \(\bar{x}_R-\bar{x}_N\). If everyone responds, the first factor is zero. If respondents and nonrespondents have the same outcome mean, the second factor is zero. Either way, this particular mean difference is zero. If nonrespondents are a substantial fraction and their mean differs from respondents’ mean, the respondent mean can differ from the full selected-sample mean.

This equation is an exact decomposition for an equally weighted sample when the outcome is known for all its members. In a real survey, the outcome for nonrespondents is usually unavailable, which is precisely why the difference cannot simply be calculated from the returned questionnaires. The calculation describes the mechanism and direction of bias; it does not provide a general way to recover the missing outcomes.

Direction and Size Depend on the Outcome

Nonresponse does not always push an estimate upward or always push it downward. Its direction depends on whether respondents tend to have higher or lower values of the particular outcome. The same pattern of participation could overstate one outcome and understate another. For example, if people who are especially satisfied with a service are more willing to answer a survey, a respondent-only estimate of average satisfaction may be too high. That same participation pattern does not, by itself, tell us whether the estimate of another measure—such as average travel time—is too high or too low.

For a proportion, the same reasoning applies by treating a Yes response as 1 and a No response as 0. The mean of these values is the proportion answering Yes. If respondents and nonrespondents have different proportions of Yes outcomes, combining the two groups changes the overall proportion.

Diagnosing the Risk in a Survey Description

A survey description rarely reveals the outcomes of those who did not participate. Diagnosis is therefore a reasoned assessment of risk, not a claim that the exact bias is known. Look for a plausible connection between participation and the outcome. Ask who may be especially likely to answer, who may be hard to reach or less willing to take part, and whether those groups might have systematically different values on the survey measure.

Consider a campus survey asking students about use of a late-night shuttle. If the survey link is sent by email and students who regularly use the shuttle are especially motivated to respond, the respondent group may overrepresent frequent users. That creates a plausible risk of overestimating use. But if participation instead is more common among students dissatisfied with the shuttle, the direction could differ. The description alone may not establish which pattern occurred.

A careful diagnosis separates what is known from what is suspected. The survey may tell us how invitations were sent and which groups responded; those details can support or weaken a concern. They do not automatically reveal how nonrespondents would have answered the outcome question. It is therefore appropriate to write “there is a risk that the estimate is biased upward because frequent users may be more likely to respond,” rather than “the estimate is definitely too high.”

The relevant comparison is outcome-specific. A demographic difference between respondents and nonrespondents matters for a survey estimate if that demographic characteristic is related to the outcome in a way that affects the estimate. If an observed difference is unrelated to the outcome, it may not cause bias for that particular estimate. Similarly, a group difference that matters for one survey question may not matter for another.

A Common Mistake

A common mistake is to reason, “Many people did not respond, so the survey is biased.” A large number of nonrespondents can make bias possible, but it does not establish that the respondent outcomes differ from the nonrespondent outcomes. In the decomposition, a large \(1-r\) is multiplied by the outcome difference \(\bar{x}_R-\bar{x}_N\). If that difference is zero, the expression is zero despite the low response fraction.

The opposite claim is also mistaken: “The survey used a probability sample, so nonresponse cannot bias it.” Probability sampling governs who is selected; it does not guarantee who will participate. The correction in both cases is to ask whether participation may be associated with the outcome being estimated, and to state the evidence and uncertainty carefully. A description can establish a plausible risk without proving the existence, direction, or size of bias.