One Survey, More Than One Possible Problem
A survey can go wrong at several points. Some people in the population may have little or no chance of being invited. Some invited people may not respond. Those who do respond may misunderstand a question or give an answer that differs systematically from what they would report under a more suitable measurement process. These are not interchangeable problems: to diagnose them, locate where in the survey process the potential error arises.
This matters when a description is incomplete. Knowing that a survey used an online questionnaire, for instance, does not tell us whether the sample was selected randomly, whether some people were excluded from the sampling frame, or whether answers were affected by the questions. A careful diagnosis states what the information establishes, what it only raises as a possibility, and what additional design facts would help resolve the uncertainty.
Locate the Source in the Survey Process
A useful first step is to trace the process from the population of interest to the recorded answers. Selection concerns who is represented in the data; measurement concerns what values are recorded for those represented units. This distinction is about the source of error, not whether the problem looks serious or whether a survey is described as “online,” “random,” or “anonymous.”
- Selection: Ask how the target population was defined, what sampling frame was used, how units were chosen, and who actually participated. A frame that omits part of the target population can create undercoverage. A convenience or voluntary response sample can have systematic differences from the population. If selected people do not respond, nonresponse can also lead to bias when participation is related to the survey characteristic.
- Measurement: Ask how the question was worded, how the answer was recorded, and under what conditions it was given. Leading or unclear wording, recall difficulty, social desirability, interviewer influence, and mode effects can affect reports even if the respondents were selected appropriately.
These categories can interact. For example, people with a particular experience may be more likely to answer a survey, and the wording may also prompt them to overstate that experience. The first concern is about which people contributed answers; the second is about the answers those people gave. Identifying both does not establish that either one actually shifted the estimate in a particular direction.
| Survey stage | Diagnostic question | Possible source |
|---|---|---|
| Defining and reaching the population | Could some members of the target population be absent from the frame or unable to be reached? | Undercoverage or another selection problem |
| Choosing and retaining participants | How were units invited, and who did not participate? | Selection differences, including nonresponse |
| Asking and recording | Could wording, recall, interviewer behavior, or survey mode systematically alter reports? | Measurement bias |
| Interpreting a reported flaw | Is there evidence of systematic distortion, or only a plausible risk? | Uncertainty requiring more design information |
A Short Justification for Keeping the Sources Separate
For a quantitative outcome, the distinction can be expressed directly. Suppose a target population has a true value \(y_i\) for each unit \(i\), but respondent \(i\) reports \(y_i^*=y_i+e_i\), where \(e_i\) is that respondent’s reporting error. Let \(R\) be the set of respondents, and let \(\bar{y}_R\) and \(\bar{y}_R^*\) be the averages of their true and reported values. If the target population mean is \(\mu\), then:
The first term compares the respondents’ true average with the target population mean. It can reflect selection into the respondent group. The second term is the average reporting error among respondents, a measurement component. This is an accounting identity, not a claim that the two components are always easy to estimate separately. It shows why a survey description should not use evidence about one stage as proof of a problem at another. For a categorical outcome, the same reasoning applies to reported proportions: who responds can change the represented group, while misclassification can change recorded category membership.
Worked Example: A Clinic Survey With Several Plausible Flaws
Hypothetical case. A clinic sends an online questionnaire to 400 patients who have accounts on its patient portal. In total, 180 complete it. The questionnaire asks, “How often do you make a healthy effort to exercise?” The clinic uses the answers to describe the exercise habits of all adults in its service area. Identify what is known, what is only a possible concern, and what information would help.
Define the target. The stated target is all adults in the service area. The invitation list includes patients with portal accounts, so it may omit adults who are not clinic patients or who lack portal accounts. This raises a coverage concern. To assess it, we would need to know how the portal list relates to the target population and whether excluded adults differ in exercise habits.
Consider participation. Of the 400 invited patients, 180 completed the questionnaire, so 220 did not complete it. This establishes substantial nonparticipation among those invited, but the count alone does not establish nonresponse bias. We would need information about how the 220 differ from respondents in relevant characteristics, or evidence about whether response is associated with exercise habits.
Consider measurement. “Make a healthy effort” is subjective and potentially suggestive. Respondents may interpret it differently or feel encouraged to report a socially approved answer. That wording creates a measurement concern, but the description does not prove that answers are systematically inflated. A clearer, more neutral question and information about how respondents interpret it would help assess the risk.
Conclusion. The case supports concerns about possible undercoverage, nonresponse, and measurement problems. It does not identify which one, if any, caused a biased estimate or by how much. Nor can we determine the direction of any distortion from the invitation and response counts alone.
Work From What the Case Actually Says
In ambiguous cases, separate established facts from plausible mechanisms. “The survey was sent to portal users” is a fact about the frame. “People without portal access exercise less” is an unverified claim about how excluded people differ. “The item asks about a healthy effort” is a fact about the wording. “Respondents therefore overreported exercise” is a claim about the effect on responses that needs evidence.
The distinction also prevents a common shortcut: treating “randomly selected” as a complete diagnosis. Random selection can support a sound selection process from a particular frame, but it cannot include people missing from that frame, guarantee that selected people respond, or prevent a question from influencing answers. Similarly, an anonymous survey may reduce one concern about disclosure, but it does not by itself establish that the sample represents the target population or that questions are understood consistently.
Worked Example: Random Invitations, Uneven Participation, and a Sensitive Question
Hypothetical case. A university randomly selects 300 students from its complete enrollment list and invites them to answer an anonymous survey about weekly alcohol consumption by telephone. A total of 210 students respond. The report gives the average among respondents. The case does not say whether the interviewers read a neutral question, how many attempts were made to reach nonrespondents, or whether students’ response rates varied by year of study.
Selection facts. Random selection from a complete enrollment list is evidence about how the initial invitees were chosen. It does not show that all invited students responded: 90 did not. Because nonrespondents may differ from respondents in alcohol consumption, nonresponse-related selection bias is possible. We need information about follow-up and relevant differences between respondents and nonrespondents to assess that possibility.
Measurement facts. Alcohol consumption is a sensitive topic, so social desirability or interviewer effects could affect reports. Anonymity may reduce concern about disclosure, but it does not prove that answers are accurate. We would need the exact question, interviewer instructions, privacy conditions, and evidence about how the mode affected reporting.
Conclusion. The initial random selection does not settle either issue. Uneven participation is a selection concern; possible underreporting during a telephone interview is a measurement concern. The description does not establish that underreporting occurred, that nonresponse shifted the mean, or which direction the overall estimate might move.
Ask for the Information That Would Resolve the Case
A useful diagnosis does not stop at naming a possible bias. It identifies what design information would distinguish competing explanations. The right question depends on the suspected stage:
- For coverage: Who belongs to the target population, what list or frame was used, and who could not appear on it?
- For participation: How many selected units responded, refused, could not be reached, or left items unanswered? What follow-up was attempted? Do response patterns relate to the outcome or to characteristics associated with it?
- For measurement: What were the exact questions and response options? Were they clear and neutral? Was the topic sensitive or difficult to recall? Who administered the survey, in what mode, and under what privacy conditions?
- For the claimed effect: Is there evidence comparing answers or participation patterns across relevant groups, modes, or question formats? A plausible mechanism is not, by itself, evidence of the direction or size of bias.
These questions are deliberately specific. “Was there bias?” is too broad to diagnose a mixed case. Finding out that selection was random may clarify initial selection but leave response and measurement unresolved. Finding out that the item was leading may clarify a measurement risk but say nothing about people who were never invited. Each answer should be tied to the process it describes.
Worked Example: A Voluntary QR-Code Survey at a Health Fair
Hypothetical case. At a health fair, organizers place a QR code beside the exit and invite visitors to rate how helpful the event was from 1 to 5. There are 500 visitors, and 75 submit a rating. The organizers want to summarize satisfaction among all visitors. The report does not describe the exact prompt or whether anyone was asked to respond in person.
Selection concern. Visitors choose whether to scan the code, so this is a voluntary response process. The 75 respondents may differ in satisfaction from the 425 who did not submit a rating. The response count establishes how many ratings were received, not whether respondents were more or less satisfied. A comparison with a follow-up of nonrespondents, if feasible, could help assess this difference.
Measurement concern. The five-point scale gives a way to record a rating, but we do not know the exact wording, the labels attached to each response option, or whether the prompt was presented neutrally. If the prompt praised the event before asking for a rating, that could influence reported satisfaction. We need the exact prompt and response choices to assess this risk.
Conclusion. Voluntary participation is a clear selection feature that creates a risk of response bias; its actual effect is unknown. Measurement bias cannot be diagnosed from the existence of a QR code or a numerical scale alone. The response process and the question wording require separate investigation.
Common Mistakes
- Calling every survey flaw “sampling bias.” A question that systematically changes what respondents report is a measurement concern. A frame that excludes part of the target population is a selection concern.
- Treating a risk as proof. “There may be social desirability” is not the same as showing that answers were shifted. State what is plausible and what evidence is missing.
- Inferring the direction from the flaw alone. A low response rate does not tell us whether the estimate is too high or too low. The direction depends on how respondents and nonrespondents differ on the outcome.
- Assuming random selection solves everything. Random selection addresses how units were chosen from a frame; it does not fix an incomplete frame, nonresponse, or measurement problems.
- Giving one label when several stages are implicated. A survey can have both selection and measurement concerns. Diagnose each separately rather than forcing the case into a single category.
Check Your Understanding
For each case, identify the stage of the process at issue and one fact that would help resolve uncertainty.
- A survey of city residents uses a mailing list that excludes people without a permanent address. Is this primarily a selection or measurement concern, and what further fact would help assess its importance?
- A randomly selected group is invited to report monthly spending, but many do not respond. Why does the number of nonrespondents alone not establish the direction of bias?
- A question asks, “How often do you responsibly follow your doctor’s advice?” What feature raises a measurement concern, and what wording information would you want to inspect?
- A questionnaire is anonymous and uses a complete list of eligible people, but it is administered by an interviewer. Which concerns might anonymity address, and which remain unresolved?
- Give one example of a survey in which selection and measurement concerns could both arise, and distinguish the mechanism for each.