When Answers Are Distorted
In Nonresponse Bias, you learned that selected people may not provide answers, and that nonresponse can bias results when respondents differ from nonrespondents on the variable of interest. A different problem occurs when people do respond but the answers they give are influenced by the survey process. A completed survey is not automatically an accurate survey.
The wording of a question, a respondent’s wish to make a favorable impression, or an interviewer’s behavior can all affect what someone reports. These influences matter because a survey is meant to measure a variable, such as a person’s opinion or reported behavior. If answers are systematically shifted away from what respondents would say in a neutral, comfortable setting, the results may misrepresent that variable.
Response bias is not the same as ordinary variation in answers. People can have different opinions, and a respondent can misunderstand a question by chance. The concern here is a consistent influence on answers—for example, a question that nudges many respondents toward the same choice. Often, a survey cannot reveal each respondent’s private “true” answer. Instead, we identify plausible sources of distortion and describe how they could affect the results.
As discussed in Biased and Leading Question Wording and Surveys and Designing Survey Questions, a question should be clear, focused, and neutral. This tutorial builds on those ideas by examining what may happen to the answers when question wording, social expectations, or the interviewer shapes the response.
Three Ways Answers Can Be Influenced
Question wording can influence answers through emotional terms, praise or criticism, assumptions, or phrasing that suggests a preferred position. Even small wording changes can affect how respondents understand an issue. If two versions ask about the same subject but one version uses more favorable language, their results may differ partly because of that language.
Social desirability is the tendency to give an answer that seems acceptable, responsible, or approved by others rather than one that fully reflects the respondent’s behavior or opinion. Someone might overreport a socially praised behavior, such as returning borrowed items on time, or underreport a behavior they expect others to disapprove of. The direction depends on what answer seems socially preferred in that situation.
Interviewer effects occur when an interviewer’s presence or behavior influences a respondent’s answer. Tone of voice, facial expression, appearance, reactions, or the way a question is delivered might make one answer seem more welcome. The respondent may adjust an answer to avoid awkwardness or gain approval, even if the interviewer does not intend to influence them.
Worked Example: Comparing Two Question Wordings
Worked Example: Does a Favorable Phrase Change Reported Support?
A fictional town selects a random sample of 240 adult residents to ask about a proposed community garden. The survey team randomly assigns 120 selected residents to each of two versions of a question. One asks, “Do you support the community garden proposal?” The other asks, “Do you support the beneficial community garden proposal?” Both versions offer the same “yes” and “no” response options. Of the residents receiving the neutral wording, 54 answer yes. Of those receiving the favorable wording, 78 answer yes.
State: The variable is whether an adult resident supports the proposal. The possible source of response bias is the word “beneficial,” which presents the proposal positively and may encourage a “yes” answer.
Plan: Compare the proportion answering yes in each wording group. The comparison is most informative because residents were randomly assigned to versions, the question topic and response options are otherwise the same, and the survey procedure is held constant. The random assignment helps make the groups comparable; it does not, by itself, make the sample representative of all residents.
Do: The proportion answering yes under the neutral wording is
Under the favorable wording, the proportion answering yes is
The difference, calculated as favorable wording minus neutral wording, is
The reported support proportion is 0.20, or 20 percentage points, higher with the favorable wording. As a check, the two groups each have 120 residents, and their yes counts differ by \(78-54=24\); \(24/120=0.20\), the same difference in proportions.
Conclude: In this sample, more residents answered yes when the question described the proposal as “beneficial.” The difference is consistent with the favorable word influencing responses. It does not establish how every resident would answer in a neutral setting or tell us which proportion is the proposal’s “true” level of support.
Social Desirability and Privacy
Social desirability is especially important for questions about behavior that people may feel judged for, such as following a rule, helping others, or keeping a commitment. A respondent might report the socially approved behavior more often than it actually occurs. Conversely, a respondent may avoid claiming a behavior that could seem boastful. Do not assume the direction without considering the situation and the expected social norm.
A private self-administered survey can sometimes make it easier for people to answer honestly about sensitive topics because they are not speaking directly to an interviewer. However, privacy does not guarantee truthful answers. Respondents may still misunderstand the question, forget what happened, or choose not to disclose information. Survey designers should explain privacy protections accurately, as discussed in Ethical Considerations in Data Collection.
Worked Example: Reporting a Socially Approved Behavior
A fictional school selects 100 students at random to ask whether they returned a borrowed library item by its due date the last time they borrowed one. The students are randomly assigned to answer either in a face-to-face interview or through a private, self-administered form. The question and response options are identical. In the interview group, 42 of 50 students answer yes. In the private-form group, 31 of 50 answer yes.
State: The variable is whether a student reports returning the most recently borrowed library item by its due date. Because returning an item on time may seem like the responsible answer, students speaking directly to an interviewer might feel pressure to report that they did so.
Plan: Compare the proportions answering yes across the two randomly assigned survey modes. The groups have the same size, question, and response options, so the main planned difference is whether a student answers face to face or privately. Random assignment supports a fair comparison of the modes in this sample, but the survey does not independently verify what students actually did.
Do: For the interview group, the proportion answering yes is
For the private-form group, it is
The interview-group proportion is higher by
That is a difference of 22 percentage points. Checking with counts gives \(42-31=11\) additional yes answers in groups of 50; \(11/50=0.22\), matching the difference in proportions.
Conclude: In this constructed example, students answering face to face were more likely to report returning their item on time than students using a private form. This pattern is consistent with social desirability or another survey-mode effect. It does not show that the interview answers are false or that the private-form answers are correct; no independent information about actual return behavior is provided.
How Interviewers Can Affect Responses
An interviewer can influence responses unintentionally. A sigh after one answer, a smile after another, or a noticeable difference in emphasis may signal which response seems preferred. Personal characteristics or visible affiliations could also affect how comfortable a respondent feels. A scripted question alone is not enough if interviewers deliver it differently or react differently to answers.
To reduce interviewer effects, use a consistent script, read questions in the same way, and avoid approving or disapproving reactions. Train interviewers to respond neutrally when a person asks for clarification, and give them a standard clarification that does not steer the answer. If interviewer identity or style differs between groups of respondents, those differences could be mixed up with real differences between the groups.
Worked Example: Different Answers With Two Interviewers
A fictional community survey selects 80 residents at random to ask whether they separate recyclable materials at home every week. The residents are randomly assigned in equal numbers to two interviewers. Both interviewers use the same written question and response options. One interviewer nods and says “That’s great” after residents answer yes; the other interviewer gives no reaction. With the first interviewer, 32 of 40 residents answer yes. With the second interviewer, 21 of 40 answer yes.
State: The variable is whether a resident reports separating recyclables at home every week. The concern is that positive reactions from the first interviewer may make a “yes” response seem more socially approved.
Plan: Compare the yes proportions. The equal group sizes, random assignment of residents to interviewers, and identical question wording support a useful comparison. The interviewers’ different reactions are a possible influence, but the survey does not check each resident’s recycling behavior independently.
Do: The yes proportion with the first interviewer is
With the second interviewer, the yes proportion is
The difference is
The first interviewer’s group has a yes proportion 0.275, or 27.5 percentage points, higher. The count check gives \(32-21=11\) additional yes answers among 40 residents, and \(11/40=0.275\).
Conclude: Residents interviewed by the interviewer who reacted positively reported weekly recycling more often. The difference is consistent with an interviewer effect, especially because the question was the same and residents were randomly assigned to interviewers. It is not proof that the interviewer’s reaction caused every difference; chance variation and other features of the interaction may also matter. Using a neutral response after every answer would make the survey procedure more consistent.
Common Mistakes and AP Exam Tips
- Calling every missing answer response bias. If selected people do not answer, consider nonresponse bias, as in the previous tutorial. If people answer but the survey process influences what they report, consider response bias.
- Assuming the bigger proportion is the truth. A higher reported proportion under one wording or mode is evidence of a difference in responses, not a way to identify which group gave the correct answers.
- Claiming a definite direction without a reason. Explain why a phrase, social expectation, or interviewer behavior could push answers in a particular direction. If the scenario does not support a direction, describe only the possible influence.
- Ignoring how the comparison was set up. If wording or mode is the only planned difference and people are randomly assigned to versions, the comparison is more informative. If the groups were formed in different ways, their answers may differ for reasons other than the survey feature.
- Confusing random selection with random assignment. As in Why Random Selection Matters and Scope of Inference: Four Combinations, random selection supports generalizing to the population sampled from, while random assignment helps make comparison groups comparable. They serve different purposes.
- Calling a question neutral just because it sounds polite. A courteous question can still include a favorable or unfavorable cue, an assumption, or language that encourages a particular response. Explain the specific feature that may influence the answer.
For a strong AP response, name the source of possible influence, connect it to the variable being measured, and state how it could affect answers. For example: “Because the question calls the proposal beneficial, it may encourage respondents to report support more often than neutral wording would.” That wording identifies a plausible mechanism without claiming that the survey reveals respondents’ unobserved true opinions.
Check Your Understanding
For each situation, identify the possible influence and explain what can and cannot be concluded from the reported answers.
- A survey asks, “Do you agree that the school should finally fix its inadequate lunch program?” Explain which wording may influence answers and why.
- Students are more likely to say they always follow a classroom rule when asked face to face than when answering a private form. Give one possible explanation and one limitation of that comparison.
- An interviewer smiles after one response option but not another. Explain how this could affect answers even if the question’s wording is neutral.
- A survey has a very high response rate, but its question uses strongly approving language. Explain why response bias could still be a concern.
- In a study, two randomly assigned groups receive different wordings and report different proportions. Explain what random assignment helps with and what the observed difference alone does not establish.