Why Accurate Questions Can Still Produce Inaccurate Reports
A survey question can be clear and neutrally worded yet still produce an inaccurate answer. A respondent may not remember how many doses of a medicine they missed last month, or may hesitate to report behavior they think others will disapprove of. In both cases, the recorded answer may differ systematically from the behavior or event the survey aims to measure.
These are two distinct sources of measurement bias. Recall bias arises when memory affects reports about past events. Social desirability bias arises when a respondent’s wish to present themselves favorably affects what they report. Either may occur in health and behavior surveys, and either may lead to underreporting or overreporting. The direction depends on the question and circumstances; it should not be assumed without evidence.
Both are forms of measurement bias: the issue is how the characteristic is recorded, not necessarily who was selected for the sample. A sample can be selected appropriately and still yield distorted reports. Conversely, a discrepancy between reports and another measure does not by itself establish which mechanism caused it. A comparison measure may also be incomplete or inaccurate.
Recall Bias: Reconstructing the Past
Recall is more difficult when a question asks about a long period, events that happen often, events that are similar to one another, or details that were not memorable at the time. For example, a person may have trouble counting every missed dose of a medicine over several months. They might forget some missed doses, confuse one week with another, or remember an unusual lapse more readily than routine behavior.
Recall errors do not always move in one direction. Forgetting events can lead to underreporting, but telescoping—remembering an event as more recent than it was—can cause a respondent to include an event outside the requested period. A respondent might also estimate a count by relying on a typical week rather than remembering each occurrence. The question’s time window and the behavior being recalled both matter.
A short, clearly specified recall period can make retrieval easier, but it does not guarantee accuracy. A calendar, event cue, or contemporaneous record may help a respondent reconstruct events. These supports can reduce some memory demands, although they cannot ensure that every event was recorded or remembered correctly.
Worked Example: Reports of Missed Medication Doses
In a hypothetical survey, 80 patients are asked whether they missed at least one prescribed dose during the previous month. For this exercise, assume a contemporaneous medication log is available for each patient and treat it as the reference classification. The log indicates that 32 patients missed at least one dose and 48 did not. Comparing each survey answer with the corresponding log gives these counts: of the 32 log-positive patients, 18 report a missed dose and 14 do not; of the 48 log-negative patients, 6 report a missed dose and 42 do not.
The survey records 18 + 6 = 24 patients as having missed a dose. Its reported proportion is \(24/80=0.300\), or 30.0%. The reference proportion is \(32/80=0.400\), or 40.0%. The difference in reported proportions, survey minus reference, is \(0.300-0.400=-0.100\), a shortfall of 10.0 percentage points.
Among the 32 patients classified as having missed a dose by the log, 14 did not report one. That is \(14/32=0.438\), or 43.8% of the log-positive group. The pattern is consistent with missed events being forgotten, but it does not prove that recall bias caused the discrepancy. The log may itself omit doses, and other reasons for disagreement are possible. The conclusion is limited to this hypothetical comparison: survey reports were lower than the stated reference classification.
Social Desirability Bias: Reporting What Seems Acceptable
Some behaviors carry a positive or negative social judgment. Respondents may overstate behavior viewed as responsible or healthy, such as exercising regularly, and understate behavior viewed as risky or disapproved, such as smoking. They may also avoid reporting stigmatized experiences. These are possible patterns, not rules: people differ, and what seems desirable depends on the social setting and the question.
Social desirability bias is about the influence of perceived acceptability on the answer. It is not simply any inaccurate report, and it is not the same as refusing to answer. A person who leaves a question blank has not provided a self-report for that item; a person who answers but alters the report may contribute measurement error. A survey can include both kinds of problems, but they should not be treated as identical.
Privacy and neutral, nonjudgmental wording may reduce pressure to give an approved answer. For example, a question can make clear that a range of behaviors is common and that accurate answers are useful. Such steps may help, but they cannot guarantee truthful reporting or establish that any remaining discrepancy is due to social desirability.
Worked Example: Reporting Recommended Physical Activity
Suppose 120 adults complete a hypothetical survey asking whether they met a physical-activity recommendation during the past week. For this example, a separate activity log is treated as the reference measure. The log classifies 60 adults as meeting the recommendation and 60 as not meeting it. In the matched comparison, 54 of the 60 log-positive adults report meeting it, while 6 report that they did not. Of the 60 log-negative adults, 18 report meeting it and 42 report that they did not.
The survey total reporting that they met the recommendation is \(54+18=72\), so the survey proportion is \(72/120=0.600\), or 60.0%. The reference proportion is \(60/120=0.500\), or 50.0%. The difference is \(0.600-0.500=0.100\), an excess of 10.0 percentage points in the survey reports.
In this hypothetical comparison, more adults report meeting the recommendation than the log classifies as meeting it. That pattern could be consistent with socially desirable overreporting, since regular activity may be viewed favorably. It is not proof of that explanation: the log may not capture all activity, and the survey and log may measure the target differently. The calculation describes disagreement with the stated reference, not the cause of the disagreement.
What a Comparison Can Establish
When a reference measure is available, matched comparisons help describe the type and direction of disagreement. For a binary outcome, let \(N\) be the number of people with both measures. Relative to the chosen reference, let \(FP\) be the number who report the behavior but are reference-negative, and \(FN\) the number who do not report it but are reference-positive. The survey’s reported proportion minus the reference proportion is determined by the balance of these two kinds of disagreement.
To see why, let \(R_i\) equal 1 when person \(i\)'s survey report is positive and 0 otherwise, and let \(T_i\) equal 1 when the reference classification is positive and 0 otherwise. The difference between the two proportions is the sum of \(R_i-T_i\) divided by \(N\). Each false positive contributes \(+1\), each false negative contributes \(-1\), and every agreement contributes 0. Therefore:
This identity explains why a net difference can hide important detail. If false positives and false negatives are both numerous but similar in count, the overall proportions may look close even though individual reports often disagree with the reference. A matched table is therefore more informative than comparing two totals alone.
The reference measure must also be interpreted carefully. A diary can be incomplete; a record may not capture behavior outside a clinic or system; and a proxy measure may not be identical to the survey target. A comparison can reveal disagreement with that reference, but attributing disagreement to recall or social desirability requires a reasoned argument about the question, the behavior, the recall period, and the quality of both measures.
Distinguishing the Two Sources
The wording and context of a survey scenario can suggest which mechanism is most plausible. If the main challenge is remembering numerous past events or placing them within a time window, recall bias is a natural concern. If the question concerns behavior that respondents may want to conceal or portray favorably, social desirability bias is a natural concern. Both can operate at once: a respondent may forget some events and feel pressure to report fewer of them.
| Clue in the scenario | Likely concern | Reason |
|---|---|---|
| Many routine events over a long period | Recall bias | Events may be forgotten or counted inaccurately. |
| A behavior carrying stigma or moral judgment | Social desirability bias | Respondents may be reluctant to disclose it. |
| A behavior viewed as healthy or responsible | Possible social desirability bias | Respondents may overstate how often they do it. |
| A long time window for a sensitive behavior | Possibly both | Memory difficulty and pressure to shape the answer can coexist. |
Worked Example: Classifying a Survey Problem
A hypothetical clinic survey asks patients how many times they used a pain reliever without consulting a clinician during the past six months. Some patients may regard this behavior as unwise, and the question asks them to recall repeated events over a long period. Identify the two potential biases and explain what can be concluded from the scenario alone.
Recall concern. The six-month period and repeated events may make an accurate count difficult. Patients could forget occasions or misremember when they occurred.
Social desirability concern. If patients think the behavior will be judged negatively, they may report fewer occasions than they recall or prefer not to disclose it. This is a plausible risk, not a certainty about any individual respondent.
Conclusion. The scenario gives reasons to investigate both sources of measurement bias. It provides no comparison data, so it does not establish how many reports are inaccurate or whether the overall survey result is too high or too low.
Common Mistakes and Practical Safeguards
- Assuming the direction in advance. Long recall periods may lead to forgetting, but telescoping or estimation can also raise a count. State the likely direction only when the details support it.
- Treating a discrepancy as proof of a mechanism. A survey-reference difference shows disagreement, not its cause. Assess whether the reference is complete and measures the same concept.
- Equating social desirability with deliberate lying. Respondents may be uncertain, embarrassed, influenced by expectations, or simply interpreting the question differently. Avoid claims about intent without evidence.
- Assuming a small net difference means accurate reports. Positive and negative disagreements can cancel in an overall proportion. Examine matched responses when available.
Practical improvements should match the suspected source. For recall, use a clearly bounded and manageable time period, ask about specific events, and consider calendars or records. For social desirability, use neutral language, emphasize that honest answers are useful, and protect privacy. These steps can reduce some pressures or memory demands; they do not eliminate measurement error.
The key is to distinguish what is observed from what is inferred. A report that differs from a reference is a measurable discrepancy. Calling it recall bias or social desirability bias is an explanation that should be supported by the survey context and by the limitations of the reference measure.
Check Your Understanding
Answer each question by identifying the evidence and stating any limits on the conclusion.
- A survey asks people to count every headache they had during the previous year. What measurement problem is especially plausible, and why?
- A survey asks about a behavior that respondents may feel embarrassed to disclose. Which bias is a concern, and what direction might it produce?
- In a matched comparison, survey reports are higher than a reference measure. Does this establish social desirability bias? Explain.
- For a binary outcome, suppose \(FP=9\), \(FN=5\), and \(N=100\). Using the displayed identity, calculate the survey proportion minus the reference proportion.
- Give one way to reduce recall demands and one way to reduce social pressure. Why do neither guarantee accurate answers?