When Respondents Choose Themselves
In Convenience Sampling and Its Bias, you saw how selecting people because they are easy to reach can leave some groups underrepresented. A related problem occurs when an invitation is open and people decide for themselves whether to respond. Call-in polls and online polls often work this way: anyone who encounters the invitation can choose to participate, but the pollster does not select respondents at random.
This matters because people who choose to answer may differ from people who do not. In particular, those with strong feelings about an issue may be more motivated to call, click, or submit a response. If their views differ from those of less motivated people, the responses may give a distorted picture of the population’s opinions.
For a poll about a proposal, for example, both people who strongly support it and people who strongly oppose it might be especially eager to respond. People with mixed or less intense views may be less likely to take part. The resulting poll could overrepresent strong views, although the responses do not always shift in one predictable direction. Which view appears more common depends on who chooses to respond.
What Makes a Response “Voluntary”?
Look for who initiates participation. If a radio station asks listeners to call a number to vote on a question, the station has invited responses, but listeners choose whether to call. If a website posts a poll link and visitors click it only if they want to, the visitors choose themselves into the poll. These are voluntary response methods, even if the invitation reaches a large audience.
A voluntary response sample is not the same as a chance-based sample. In a simple random sample, chance selects individuals from a sampling frame. In an open poll, people decide whether to participate; their chance of appearing in the results is connected to whether they see the invitation and whether they choose to answer. A large number of responses does not turn those choices into random selection.
Voluntary response sampling can overlap with convenience sampling, but the terms point to different features. Convenience sampling emphasizes that the researcher recruits people who are easy to reach. Voluntary response sampling emphasizes that individuals choose themselves to respond. For instance, a poll link posted on a popular local website may be easy for visitors to access, and those visitors also choose whether to click and answer. Describing the poll as voluntary response identifies the self-selection that is central to this tutorial.
As discussed in Sources of Variability in Collected Data, different samples can produce different statistics because of sampling variability. Voluntary response bias is a separate concern: it comes from the way respondents enter the sample, not just from ordinary sample-to-sample fluctuation. Collecting more answers may make the poll’s result more stable among the people who respond, but it does not ensure that those people represent everyone the poll is about.
Worked Example: A Radio Call-In Poll
Worked Example: Support for a New Bus Route
Imagine a fictional town with 1,000 adults. In this constructed example, 400 adults support a proposed new bus route and 600 oppose it. A radio station asks listeners to call in and say whether they support the proposal. Of the 400 callers, 300 support the route and 100 oppose it.
State: The population of interest is all 1,000 adults in the town. The categorical variable is whether an adult supports the new bus route. The poll’s sample consists of the 400 adults who chose to call in.
Plan: This is a voluntary response sample because listeners decide whether to call. People with strong views about transportation might be more motivated to participate than people with less intense opinions. If supporters are especially motivated in this example, the poll may overrepresent support.
Do: The proportion supporting the route among callers is
So 75% of the callers support the route. In the constructed population, the proportion supporting it is
The caller proportion is \(0.75-0.40=0.35\), or 35 percentage points higher than the population proportion in this illustration.
Conclude: In this constructed example, the call-in poll overrepresents support for the bus route. The difference is plausible if supporters were more motivated to call than other adults. In an actual poll, the population proportion would usually be unknown, so the poll’s result alone would not reveal the size or direction of any bias. It would be accurate to report that 75% of the callers supported the route, but not to claim that 75% of all town adults supported it.
Strong Feelings Can Pull a Poll Toward the Extremes
An open poll may attract respondents from more than one side of an issue. The important concern is not simply that one opinion is present in the poll; it is that some people may be more likely to answer because they feel more strongly. Less engaged or undecided people may be less likely to respond. A poll can therefore overrepresent people with intense opinions without accurately measuring how common those opinions are in the population.
Do not assume that “strongly opinionated people respond” automatically means the result will show more opposition, more support, or more extreme answers in one particular direction. A poll can lean toward whichever group is more likely to participate. Without information about people who did not respond, the direction and size of the bias may be uncertain.
Worked Example: An Online Poll About School Start Times
A fictional school district has 500 families. For this illustration, 200 families favor keeping the current start time, 200 favor a later start, and 100 are undecided. The district’s website posts an open poll. Of the 100 families who choose to answer, 60 favor a later start, 30 favor keeping the current time, and 10 are undecided.
State: The population of interest is all 500 families in the district. The categorical variable is a family’s opinion about the school start time. The poll’s sample is the 100 families who chose to answer online.
Plan: The poll uses voluntary response because families decide whether to participate after seeing the website poll. Families with strong views about schedules may be more likely to respond than undecided families. The poll’s sample composition should therefore not be assumed to match the district’s composition.
Do: Among the respondents, the proportion favoring a later start is
In the fictional population, the corresponding proportion is
The respondent proportion is \(0.60-0.40=0.20\), or 20 percentage points higher in this constructed example. Also, 10% of respondents are undecided, while \(100/500=0.20\), or 20%, of the population is undecided.
Conclude: The online poll shows that 60% of the families who chose to respond favored a later start. In this illustration, that percentage is higher than the population proportion. The difference could occur if families with strong opinions about changing the schedule were more motivated to complete the poll. The poll does not establish that 60% of all district families favor a later start.
A High Response Count Does Not Remove Self-Selection
Suppose the school poll receives 2,000 responses instead of 100. That is a larger set of responses, but it is still voluntary if families decide whether to participate. Many responses can give a detailed description of the people who answered. They do not show that the people who did not answer would have responded in the same proportions.
This is the same general caution about sample size introduced in Convenience Sampling and Its Bias: size and representativeness are not interchangeable. A larger voluntary response sample may include more people with a particular view precisely because they are especially motivated to answer. Without a selection process that supports representing the population, the larger count alone does not justify generalizing the result.
A careful description distinguishes what the poll directly observed from what it hopes to learn. “Among the people who answered the website poll, 60% favored a later start” describes the respondents. “Sixty percent of all district families favored a later start” makes a broader population claim that the voluntary response poll does not establish.
Worked Example: Evaluating a Community Garden Poll
A neighborhood newsletter invites readers to click an online poll asking whether the community should fund a garden. There are 240 responses: 180 support funding and 60 oppose it. The newsletter reports, “Three quarters of neighborhood residents support the garden.”
State: The intended population is all neighborhood residents. The variable is whether a resident supports public funding for the garden. The 240 respondents are the people who chose to click the poll and submit an answer.
Plan: Because readers choose whether to respond, this is a voluntary response sample. Residents who care strongly about the garden could be more likely to notice or answer the poll. The number of responses does not establish that respondents reflect all neighborhood residents.
Do: The proportion supporting funding among respondents is
Thus, 75% of the respondents supported funding, and 25% opposed it. These percentages describe the 240 poll respondents. The poll provides no count of how many nonresponding residents support or oppose the garden.
Conclude: The newsletter’s statement generalizes beyond the group that answered. A more defensible report is, “Of the 240 people who chose to answer the online poll, 75% supported funding the garden.” Since people with strong views may have been more likely to respond, the poll cannot by itself establish the level of support among all neighborhood residents.
Common Mistakes and AP Exam Tips
- Calling an open poll random because anyone can participate. An open invitation is not random selection. A full-credit answer says that individuals chose whether to respond and identifies the poll as voluntary response.
- Assuming every volunteer has the same opinion. Voluntary response does not mean all respondents agree. The concern is that willingness to respond may be related to the measured opinion, making some views more likely to appear.
- Claiming a definite direction without support. Strong opinions may motivate people on either side. Explain which group might be more motivated only if the context supports that possibility; otherwise, say the direction of bias is uncertain.
- Treating many responses as proof of representativeness. A large number of self-selected respondents is still self-selected. Explain that response count does not show how nonrespondents would answer.
- Reporting a respondent percentage as a population percentage. State the group described by the statistic. “Among people who answered” is supported; “among all residents” requires a selection method that supports that generalization.
- Confusing voluntary response with convenience sampling. The methods can overlap. Explain the defining feature of the situation: respondents chose themselves, or the pollster recruited people because they were easy to reach, or both.
For a strong AP response, name the population and the actual respondents, identify that individuals chose whether to answer, and connect willingness to respond to the variable being measured. Then limit the claim to what the poll supports. For example: “Because listeners chose whether to call, people with strong views about the bus route may be overrepresented. The poll describes the callers, but it may not represent all adults in town.”
Check Your Understanding
For each situation, focus on how respondents entered the poll and what conclusion the results support.
- A television station invites viewers to text their opinion about a proposed stadium. Explain why the sample is voluntary response and identify one possible source of bias.
- A website poll receives 5,000 answers about a change to public library hours. Explain why the large response count does not establish that the results represent all residents.
- A poll finds that 80% of people who clicked an online poll support a new school policy. Write a sentence that reports this result without generalizing to all families.
- In a fictional town, 90 of 150 call-in respondents favor a traffic change. Find the proportion favoring the change among respondents. Explain why that value alone does not establish the townwide proportion.
- Explain how an open poll could overrepresent strongly held opinions without always leaning in the same direction.