Why the Interviewer and Survey Mode Matter
A survey answer is not recorded independently of the process used to ask the question. A respondent might give a different answer to a person on the phone than when entering it privately online. A printed response scale, a spoken list of options, or an interviewer’s follow-up prompt may also affect what is reported. These influences matter when the goal is to use survey answers to describe a population accurately.
Interviewer and mode effects are forms of measurement bias: they arise when the process of collecting an answer systematically shifts what is recorded. They do not mean every response is wrong, and a difference between two survey modes does not automatically prove that either mode caused the difference. The key questions are what was measured, how responses could have been influenced, and whether the groups being compared are comparable.
How Interviewers Can Influence Answers
Interviewer effects can occur even when an interviewer intends to be neutral. Respondents may adjust an answer to appear polite, agreeable, knowledgeable, or socially acceptable. A respondent may also feel less comfortable disclosing a sensitive experience to another person than entering an answer privately. These influences are especially plausible for questions involving stigmatized behavior, personal finances, health, or opinions that the respondent believes may be judged.
The interviewer can influence answers through more than presence alone. Tone, facial expression, emphasis, pauses, and the way a question is repeated can signal what answer seems expected. A probe such as “Are you sure?” may prompt a respondent to revise an answer. Even neutral-sounding clarification can lead different interviewers to explain a question in different ways. If interviewers differ in how they ask or record responses, the answers may vary systematically with interviewer.
A carefully written question does not guarantee identical delivery. Standardized scripts, consistent training, and neutral procedures can reduce variation, but they cannot ensure that respondents experience every interview in exactly the same way. For sensitive items, a survey may allow respondents to answer privately even when other questions are administered by an interviewer.
How Survey Mode Can Influence Answers
A survey mode determines how a question and its response options are presented and how an answer is given. Modes can differ in privacy, pace, accessibility, and the information respondents can see while answering. In a phone interview, a respondent generally hears rather than sees the response options and must remember them. In a web survey, the options may be visible together, but the screen size or layout can affect how easily they are read. A paper questionnaire provides a visible record, while an in-person interview adds face-to-face interaction.
These differences can affect which answer is selected or whether a question is answered at all. A long list read aloud may be difficult to retain; a visual list may make all options available for comparison. An interviewer may clarify a question, while a self-administered format may leave a respondent unsure what it means. A private mode may make it easier to report a sensitive answer. Conversely, a respondent with limited internet access, difficulty reading a screen, or an accessibility need may find a particular mode harder to use.
A mode comparison can therefore combine several influences. It might reflect privacy, the presence of an interviewer, how options are displayed, or who is able and willing to respond in each mode. If the question wording or response options also change between modes, it becomes harder to attribute a difference specifically to the mode.
Worked Example: Comparing Reports Across Two Modes
In a hypothetical survey, 200 people are assigned to answer a question about a sensitive behavior by private web form, and 200 are assigned to answer the same question by phone with an interviewer. Suppose 72 web respondents and 54 phone respondents answer “Yes.” The wording and response options are kept the same.
The observed proportions are:
The difference is \(0.36-0.27=0.09\), or 9 percentage points. In this hypothetical comparison, the web group reported “Yes” more often. Because respondents were assigned to modes and the question was kept the same, the contrast provides evidence that the administration process may affect reporting in this setting.
The result does not show which respondents answered truthfully, nor does it establish that every respondent would give different answers in the two modes. It also does not identify which aspect of the process mattered: privacy, interviewer presence, or another feature could contribute. The conclusion should stay with the observed contrast and the design used to obtain it.
Define the Comparison Before Interpreting It
A useful comparison holds the target population and the question as steady as possible while changing the feature of administration under investigation. For example, to investigate an interviewer effect, compare otherwise similar procedures that differ in whether an interviewer is present. To investigate a broader mode effect, compare administration modes while keeping wording and response options consistent when feasible.
For a binary question, let \(\hat{p}_A\) and \(\hat{p}_B\) be the observed proportions answering “Yes” in modes A and B. The descriptive contrast is \(\hat{p}_A-\hat{p}_B\). A positive value means that the sample proportion was higher in mode A; a negative value means it was lower. This arithmetic describes the data, but by itself it does not explain why the proportions differ.
A mode contrast is more informative about measurement when the groups assigned to or using each mode are comparable in their underlying characteristics. If respondents choose their own mode, or if one mode is used for one subgroup and another mode for a different subgroup, the observed difference can reflect who answered as well as how the question was administered. A simple comparison cannot separate these explanations without suitable design or additional information.
Worked Example: Separating a Mode Contrast From Group Composition
A hypothetical survey uses phone and web administration. The phone group includes 80 younger and 20 older respondents. The web group includes 20 younger and 80 older respondents. Suppose the “Yes” rates are 40% among younger respondents and 80% among older respondents in both modes.
For phone, the number answering “Yes” is \(80(0.40)+20(0.80)=32+16=48\), so the rate is \(48/100=48\%\).
For web, the number answering “Yes” is \(20(0.40)+80(0.80)=8+64=72\), so the rate is \(72/100=72\%\). The overall web rate is 24 percentage points higher, even though within each age group the reported rates are identical across modes.
Here the aggregate contrast arises from different age compositions: the web group contains more older respondents, and older respondents have a higher “Yes” rate in this example. The contrast is not evidence that web administration changed how people answered. This illustrates why the composition of the groups must be considered before attributing a difference to mode.
A Simple Model of Systematic Measurement Shift
For a quantitative response, a simplified model makes the idea of systematic influence precise. Let \(T_i\) be the value that would represent the target quantity for respondent \(i\), and let \(Y_i\) be the value recorded under a particular survey procedure. Suppose, for illustration, that
where \(b_m\) is a systematic shift associated with mode \(m\), and \(\epsilon_i\) is respondent-specific error whose average is zero under the model. The model is a teaching simplification: real measurement effects need not be constant across respondents or additive. Its purpose is to distinguish a systematic shift from random variation.
Taking expected values under these assumptions gives:
Thus, if mode \(m\) shifts responses upward by a constant amount, the expected recorded value exceeds the target value by that amount. A consistent shift can remain even with a large sample; collecting more responses does not, by itself, remove systematic measurement bias. For categorical responses, the same idea can be expressed as a mode-specific difference in the expected proportion selecting a response, rather than an additive shift in the response itself.
Worked Example: Interpreting a Difference in Sensitive Reports
In a hypothetical comparison, 100 respondents answer a sensitive yes-or-no item privately online, and 100 answer it with an interviewer present. Suppose 30 of the online group and 18 of the interviewer group answer “Yes.” Assume the question wording and options are identical.
The sample proportions are:
The difference is \(0.30-0.18=0.12\), or 12 percentage points. The private online group reported “Yes” more often in these samples. If the groups were assigned to these procedures in a way that made them comparable, the result is consistent with interviewer presence affecting reporting.
If respondents instead selected their own mode, the groups may differ in ways related to their answers. The 12-point contrast would then be an association between mode and reported answer, not a clear estimate of an interviewer effect. Also, because the comparison changes privacy and interviewer presence together, it cannot determine which feature accounts for the difference.
How to Diagnose and Reduce These Effects
A practical diagnosis begins by specifying the response that the survey is intended to measure and identifying the features that differ across administration procedures. Ask whether wording, response options, privacy, interviewer behavior, or visual presentation changes. Then ask whether the people answering in each mode are comparable. A difference that persists when groups are comparable and other survey features are held steady is more informative about a mode or interviewer effect than a raw difference between self-selected groups.
Survey designers can reduce avoidable variation by using the same wording and response options across modes, training interviewers to read questions consistently, and specifying neutral rules for probes and clarification. They can also test questionnaires before full use, check whether response options are easy to understand and access in each format, and provide private ways to answer sensitive items where appropriate. These steps can limit particular sources of influence, but they do not guarantee that all modes produce identical measurements.
A mode difference is not automatically a flaw. A mode may be selected because it reaches people who would otherwise be difficult to contact or makes participation more accessible. The important point is to avoid treating responses from different modes as interchangeable without considering whether the mode changes measurement or the composition of respondents. When mode is related to the answer, reporting the mode and examining the differences helps readers judge what conclusions the data support.
Common Mistake
Wrong reasoning: “The phone and web percentages differ, so the survey mode caused the difference.”
Correction: A difference between observed percentages establishes a sample contrast, not its cause. Respondent groups may differ, wording or response options may not match, and more than one feature of administration may change at once. Describe the contrast first, then assess whether the design supports attributing it to interviewer presence or mode.
Check Your Understanding
Answer each question using the distinctions in this tutorial.
- Give one way an interviewer’s behavior could influence an answer even when the interviewer does not intend to do so.
- Why might a list of response options produce different answers when read aloud than when displayed on a screen?
- A phone survey and a web survey report different proportions, but respondents chose their own mode. What important alternative explanation should be considered?
- In a hypothetical comparison, 45 of 150 respondents answer “Yes” online and 30 of 150 answer “Yes” by phone. Calculate the difference in sample proportions in percentage points, and state what it does and does not establish.
- Why can collecting more responses fail to remove a systematic interviewer or mode effect?