Improving Participation Is Not the Same as Removing Bias
A survey team can make it easier or more appealing for selected people to respond: send a reminder, try another contact method, offer a simpler way to complete the survey, or provide an incentive. These steps may increase the number of completed surveys. But they do not automatically make respondents similar to nonrespondents on the outcomes being studied.
The distinction matters because nonresponse bias depends on how respondents and nonrespondents differ on a survey outcome, not just on how many people respond. A higher response rate is useful operational information. To judge whether an intervention also reduces bias, one must ask who responded after the intervention and, where possible, how the people who remained missing differ.
The response rate is the proportion of eligible sampled units that provide a usable response, according to the survey’s stated definition of a usable response. If \(N\) eligible units are contacted and \(n_R\) respond, the response rate is \(n_R/N\). The denominator and the rule for counting a response should be stated consistently when response rates are compared.
What a Higher Response Rate Can—and Cannot—Tell You
A reminder may reach people who simply overlooked the first invitation. A follow-up call may reach people who rarely check email. A shorter survey may reduce the time needed to participate. These are plausible ways to remove practical barriers. Still, the people who respond to a follow-up might differ from those who respond immediately, and those who never respond may remain systematically different from everyone who does.
For a quantitative outcome, the connection can be stated precisely. Imagine that the outcome is defined for every unit in the eligible group, whether or not the survey observed it. Let \(\bar{y}_R\) be the mean outcome among respondents, \(\bar{y}_{NR}\) the mean among nonrespondents, and \(r\) the response fraction. The overall mean for that group is a weighted average of the two means:
Subtract the overall mean from the respondent mean:
This identity shows two ingredients in the difference between the respondent mean and the overall mean: the nonresponse fraction, \(1-r\), and the difference between respondents and nonrespondents on the outcome. A smaller nonresponse fraction reduces the first ingredient, but it does not tell us whether the second ingredient is large, small, or zero. In practice, the nonrespondents’ outcome mean is often unknown, so the identity does not usually give a direct bias calculation. It explains why a response rate alone cannot establish that bias has been removed.
Practical Ways to Encourage Responses
The appropriate strategy depends on why people are not responding, how they can be reached, and what the survey asks them to do. Strategies should be planned so that they make participation more accessible without pressuring people or changing what a question measures.
| Strategy | How it may help | What to check |
|---|---|---|
| Reminders | Prompt people who missed or postponed the original invitation. | Whether reminders reach additional kinds of people, and whether repeat contacts are proportionate. |
| Follow-up contacts | Try another time or permitted contact route for people who have not replied. | Whether the contact method is appropriate, accessible, and respectful of privacy. |
| Convenient response options | Offer practical ways to respond, such as a mobile-friendly form or a telephone option. | Whether options are genuinely accessible and whether different modes affect how people answer. |
| Incentives | Recognize time or effort and encourage completion. | Cost, fairness, ethical safeguards, and whether an incentive changes who participates. |
Convenience is not only about offering several buttons or formats. The survey should be manageable in length, available at suitable times, accessible to people with different needs, and clear about how responses and contact details will be handled. Incentives should be proportionate and should not be presented as a guarantee of benefit or as pressure to take part. These design choices can support participation, but changes in survey mode can also affect responses; a larger number of answers is not a reason to ignore possible measurement effects.
Worked Example: Calculating the Effect of a Reminder
A hypothetical survey team invites 800 eligible people by email. By the end of the first week, 400 have completed the survey. After a reminder, 120 more people complete it. Calculate the response rates before and after the reminder.
Before the reminder, the response rate is:
After the reminder, the number of completions is \(400+120=520\). The response rate is:
The response rate increased by \(65.0\%-50.0\%=15.0\) percentage points. This calculation supports the conclusion that more invited people completed the survey after the reminder. By itself, it does not show whether the additional respondents resemble the people who still did not respond, or whether the survey’s outcome estimate is now unbiased.
Assess Who Is Reached, Not Just How Many
When appropriate information is available for the full sample, response rates can be examined across relevant subgroups. For example, a team may know each sampled person’s region or age group from the sampling list. Differences in subgroup response rates can identify groups that remain less represented among respondents. This check is informative, but matching subgroup response rates would still not prove that respondents and nonrespondents within each subgroup have the same survey outcomes.
A stronger assessment is possible when outcome information for nonrespondents can be obtained through a suitable independent source or a carefully designed follow-up. Such information can help evaluate how the respondent group differs on the specific outcome of interest. It may not be available, and information about one outcome does not establish the absence of differences on every other outcome. Claims should therefore match what was actually measured.
Worked Example: More Responses, but a Remaining Composition Difference
Suppose a hypothetical sample contains 600 employed and 400 unemployed adults. Initially, 480 employed adults and 120 unemployed adults respond. After follow-up, 510 employed adults and 240 unemployed adults have responded. Calculate the subgroup and overall response rates, then consider what the change does and does not establish.
Initially, the employed response rate is \(480/600=80.0\%\), and the unemployed response rate is \(120/400=30.0\%\). The overall response rate is:
After follow-up, the employed response rate is \(510/600=85.0\%\), and the unemployed response rate is \(240/400=60.0\%\). The overall response rate is:
Participation increased in both subgroups, and the gap between their response rates narrowed from 50 to 25 percentage points. The respondents are still not in the same employment proportions as the full sample: employed people make up \(510/750=68.0\%\) of respondents, while they make up \(600/1000=60.0\%\) of the sample.
Suppose, for illustration, that a binary survey outcome is positive for 70.0% of employed people and 40.0% of unemployed people, and that those percentages apply equally to respondents and nonrespondents within each subgroup. The full sample’s positive proportion would be:
Under the stated within-subgroup assumption, the post-follow-up respondents’ positive proportion would be:
In this hypothetical calculation, the respondent proportion is 60.4%, compared with 58.0% for the full sample. The response rate improved, and the subgroup response-rate gap narrowed, yet the respondent group’s outcome proportion still differs from the sample’s. The assumption about outcome percentages within subgroups is crucial; without it, subgroup counts alone cannot determine the outcome bias.
Evaluate Costs and Trade-Offs
A practical evaluation should record more than the final response rate. Useful measures include the number of additional usable responses, the subgroups reached, time to completion, contact burden, direct cost, and any change in missing answers or survey completion quality. These measures help teams decide whether an intervention is worthwhile and whether further efforts should be targeted differently.
When comparing strategies, a simple cost-per-additional-response calculation can support planning. It must be described as a comparison with an appropriate baseline; a historical baseline may not predict exactly how the current sample would have responded without the strategy.
Worked Example: Considering an Incentive Cost
A hypothetical team expects 500 completed surveys from a planned group of 1,000 invitations, based on its planning baseline. It offers a $10 incentive to everyone who completes the survey and receives 600 completions. Assuming, for this planning calculation, that 500 completions would otherwise have occurred, estimate the incentive cost per additional completion.
The observed increase over the baseline is \(600-500=100\) completions. The total incentive cost is:
Under the stated baseline assumption, the cost per additional completion is:
The planning estimate is $60 per additional completion. It is not necessarily the causal effect of the incentive: the baseline may be inaccurate, and other circumstances could differ. Nor does the calculation show whether the extra respondents reduced bias. The team would still need to examine who responded and whether the survey remained appropriate and voluntary.
Common Mistake: Treating Participation as Proof of Representativeness
Wrong reasoning: “The response rate rose from 50% to 75%, so the nonresponse bias must have been eliminated.” This conclusion does not follow. The additional responses could come mostly from people whose outcomes resemble those of earlier respondents, while people with very different outcomes remain hard to reach. Even a high response rate does not reveal the outcomes of the people who did not respond.
Correction: Report the response-rate change accurately, then evaluate what is known about the respondents and the people still missing. Check response patterns across relevant subgroups and, when feasible, compare outcomes using appropriate independent information. State the limits of those checks. A higher response rate is an improvement in participation, not proof that nonresponse bias is absent.
Check Your Understanding
Answer using the distinctions and calculations in this tutorial.
- A survey response rate rises from 40% to 65% after follow-up. What does this establish, and what does it not establish about nonresponse bias?
- Why can subgroup response rates help evaluate a follow-up strategy, yet still fail to prove that outcome bias is absent?
- A sample has 300 eligible people, and 210 provide usable responses. What is the response rate? State the denominator used.
- A team offers a new response format. Name one participation measure and one possible measurement concern it should evaluate.
- What additional information would help assess whether people who remain nonrespondents differ on a particular survey outcome?