Why the Way People Enter a Sample Matters
A poll can collect thousands of answers and still give a misleading picture of the people it is meant to describe. The important question is not only how many people answered, but how they came to be in the sample. If participation depends on who is easiest to reach or who is especially motivated to respond, the people observed may differ systematically from the target population.
Two common nonprobability sampling approaches are convenience sampling and voluntary response sampling. They are easy to carry out, but their results do not have the same design-based justification for generalizing to a population as a probability sample, where selection chances are known.
The two approaches can overlap. A survey link posted on a popular website may be convenient for the site’s visitors, while the people who click the link and complete the survey are also volunteering. To classify a sample, ask how the people were recruited and how they entered it. Were they selected because they were available, because they opted in, or both?
Recognizing the Two Sampling Approaches
Convenience sampling is about ease of access. Examples include interviewing people at the nearest shopping center, surveying students in one readily available class, or asking patients at one clinic to represent everyone with a condition. These participants may be appropriate for describing the people actually observed. The concern arises when the claim extends to a broader target population whose members were not similarly accessible.
Voluntary response sampling is about self-selection into participation. A survey may be announced on a website, broadcast, or public noticeboard, and people decide whether to respond. Those who care strongly about the issue, have more time, or feel especially motivated to express an opinion may be more likely to take part. If those characteristics are related to the survey answer, the respondents’ answers can differ from those of the target population.
The selection process does not prove that a sample is unrepresentative for every question. For example, a self-selected group could happen to have the same average as the target population on a particular measure. The key point is that convenience and opt-in methods do not, by themselves, establish that the observed group represents the target population. A large number of responses does not remove that limitation.
Worked Example: An Online Poll About a Proposed Rule
A local news site asks, “Should the city introduce a fee for downtown parking?” Anyone visiting its website can click a poll button. The site reports 2,016 votes in favor out of 3,200 votes and says that 63% of city residents support the fee. Classify the sample and check the reported percentage.
Sampling approach. Website visitors decide whether to vote, so the poll is a voluntary response sample. It is also restricted to people who encounter the site’s poll, making access a factor in who can participate.
Calculation. The percentage of recorded votes in favor is
The arithmetic supports the statement that 63% of the people who voted supported the fee. It does not establish that 63% of all city residents supported it. Residents who did not visit the site, did not see the poll, or did not choose to vote are not represented by the calculation in any known way.
How Self-Selection Can Change Sample Composition
A sample can differ from its target population because some kinds of people are more likely to enter it than others. Suppose people who feel strongly about a topic are more likely to respond to an open invitation than people with weaker views. If strong views are associated with a particular answer, strong views will be overrepresented among respondents, and the answer percentage may shift. This is a possible mechanism, not a claim that every voluntary response sample must have a particular direction of error.
A simple population breakdown makes the issue visible. The following hypothetical example specifies exact counts of people and answers. The percentage among respondents is calculated from those counts; it is not a ratio of expected random counts.
Worked Example: An Opt-In Survey and Population Support
Imagine a target population of 1,000 residents. Of these, 400 are highly engaged with a proposed policy and 600 are less engaged. In the full population, 70% of the highly engaged residents support the policy, while 30% of the less engaged residents support it. An open online survey attracts 280 highly engaged residents and 60 less engaged residents. Among the respondents, 80% of the highly engaged group and 40% of the less engaged group support the policy. Calculate support in the population and among respondents.
Population support. The highly engaged group contains \(400(0.70)=280\) supporters. The less engaged group contains \(600(0.30)=180\) supporters. Thus the population has \(280+180=460\) supporters, and its support percentage is
Support among respondents. Of the 280 highly engaged respondents, \(280(0.80)=224\) support the policy. Of the 60 less engaged respondents, \(60(0.40)=24\) support it. There are \(224+24=248\) supporters among \(280+60=340\) respondents, so
In this hypothetical population, the respondent percentage is about 26.94 percentage points higher than the population percentage. Both the group’s support rates and its composition contribute to the difference: highly engaged residents are more common among respondents (280 of 340) than in the population (400 of 1,000), and the two engagement groups have different support rates. The example shows how self-selection can produce a respondent group unlike the target population; it does not imply that every opt-in survey will overstate support.
A Short Justification: Selected-Group Means and Population Means
For a quantitative variable, the same idea can be expressed with means. Consider a finite target population of \(N\) individuals. Suppose \(m\) of them enter a selected group, and the remaining \(N-m\) do not. Let \(\bar{x}_{\text{selected}}\) be the selected group’s mean and \(\bar{x}_{\text{remaining}}\) the mean for everyone else in the target population. The population mean is the weighted average of these two group means:
Subtracting this population mean from the selected group’s mean gives
This identity is exact for the specified population groups. It shows that the selected group’s mean differs from the population mean when the selected and remaining groups have different means and the remaining group has positive size. If their means are equal, the selected-group mean equals the population mean, even if selection is related to other features of the individuals. Thus, a difference in who participates can affect the distribution of responses without necessarily changing the mean of every variable.
Interpreting a Convenience Sample
Convenience samples can be useful when the goal is to learn about the people who were actually contacted, to test survey logistics, or to explore a question before planning a stronger sampling design. The limitation appears when a report uses the sample to make a claim about people beyond those readily available.
Worked Example: Study Time Reported at a Library
A university wants to describe the average number of hours students study in its library per visit. A researcher asks 30 students leaving the library between 4 p.m. and 6 p.m. to report their study time for that visit. Their recorded times sum to 72 hours. The researcher reports that students at the university study an average of 2.4 hours per library visit. Evaluate the sampling method and the calculation.
Calculation for the observed group. The sample mean is the sum of the 30 recorded times divided by the number of students:
That calculation correctly describes the average reported time for these 30 students. The sample is a convenience sample: the researcher approached people who were easy to find at one library during a two-hour period. Students who use other study locations, visit at other times, or do not study in the library may differ. The method therefore does not justify treating 2.4 hours as an established average for all university students. The arithmetic can be correct while the population claim is unsupported by the selection method.
What a Published Claim Can and Cannot Establish
When reading a poll or survey report, identify the target population and the actual route into the sample. Then compare them. A report about “residents” based on visitors to one website has a broader target than its recruitment method directly covers. A report about “students” based on one class has a broader target than the people who were readily available.
For a convenience or voluntary response sample, the reported percentage or mean is a description of the observed respondents, unless additional assumptions or methods are supplied to support a broader conclusion. Because inclusion chances are not known from these sampling approaches, a conventional probability-sampling margin of error cannot be justified from the sample size alone. More respondents may make the description of that respondent group more detailed, but size alone does not ensure that the group resembles the target population.
| Question to ask | Why it matters |
|---|---|
| Who is the target population? | It identifies the group the report wants to describe. |
| Who could realistically enter the sample? | It reveals limits caused by ease of access or how the invitation was distributed. |
| Who chose whether to participate? | It helps identify self-selection and possible differences between respondents and others. |
| Does the report describe respondents or the wider target? | A percentage calculated from respondents does not by itself establish the same percentage in the target population. |
Common Mistakes
- Calling an opt-in poll a random sample because anyone could respond. An open invitation does not give every target-population member a known selection chance. People still choose whether to participate.
- Assuming a large sample must represent the target. A large self-selected group can still have a composition different from the target population. A high response count does not repair the recruitment method by itself.
- Treating a respondent percentage as a population percentage. The percentage may be correctly calculated for respondents while remaining unsupported as a claim about everyone in the target group.
- Assuming selection always changes every statistic. A selected group may differ from the population in some characteristics but have the same mean for a particular variable. Check what is being measured and how selection relates to it.
Check Your Understanding
For each question, identify the selection method and explain what can reasonably be concluded.
- A radio station invites listeners to text their opinion about a proposed tax. What kind of sample is formed, and why might respondents differ from all residents?
- A researcher surveys the first 25 people leaving a campus dining hall. Is this convenience sampling, voluntary response sampling, or both? Explain.
- An online poll receives 5,000 votes. What does the large number establish, and what does it not establish about the target population?
- In the mean identity, what condition makes the selected-group mean equal to the population mean when some people remain outside the selected group?
- A report states that 60% of its respondents support a policy. What information about the sampling method is needed before extending that percentage to a wider population?