Why Audit a Published Survey?
A survey report may state that “most residents support the proposal” or that “students are worried about costs.” Those conclusions are meaningful only if we know who the survey aimed to describe and how the people whose answers were counted were reached. A careful reader does not need the full technical report to begin an audit: a few precise questions can reveal what the description supports, what it leaves unclear, and which sources of bias are plausible.
This tutorial provides a practical checklist for reading short survey descriptions. The goal is not to declare every survey good or bad from a few sentences. It is to connect each claim to the available evidence, separate confirmed design facts from unanswered questions, and avoid treating a large sample or a polished report as a guarantee of representativeness.
A Five-Part Checklist
Read the claim first, then examine the description using the following five questions. Record what the report explicitly says separately from what you infer. “Not stated” is a useful audit finding: it is more accurate than assuming that an unstated procedure was used.
Name the full group the report wants its conclusion to describe, including relevant boundaries such as location, age, or time period. A claim about “all residents” is broader than a survey of registered voters.
Identify the list, register, database, or set of units from which the sample could be drawn. Compare its coverage with the target population. People absent from the frame may have no chance of selection.
Look for a described selection mechanism: for example, a random selection from a frame, a systematic selection, or an invitation to volunteers. If the report says only that it “surveyed” a number of people, the selection method remains unknown.
Check how many units were invited or selected, how many completed the survey, and whether some selected units could not be reached or declined. A response rate summarizes participation, but by itself does not establish whether respondents and nonrespondents differ on the survey outcome.
Inspect the question wording, response options, survey mode, and time reference when these are reported. Then compare the conclusion with the population actually reached and the question actually asked.
These questions concern different parts of a survey. A suitable frame does not guarantee a sound selection method; a probability selection does not guarantee everyone responds; and a high response rate does not establish that the question measured the intended idea. Keeping the parts separate makes an audit more precise.
What the Response Information Can Establish
For a simple description, a response rate can be calculated as the number of completed responses divided by the number of eligible units invited, when both counts are available and the report defines eligibility clearly. The rate describes participation among those invited; it is not a measure of how closely the responses match the target population.
A report may not provide enough information to calculate this rate. For example, “500 people responded” gives a count of completions but not the number invited. Even when the rate is available, it does not tell us whether those who did not respond would have given different answers. That concern is nonresponse bias, and assessing it requires information about how response relates to the outcome, not just a participation percentage.
Also inspect what “response” means in the report. Someone may complete most questions but skip one item. The number of completed surveys and the number answering a particular question can therefore differ. A percentage about one question should be based on its stated denominator, not automatically on all survey invitations or all completed surveys.
Worked Example: A Citywide Opinion Poll
A hypothetical report says: “We randomly selected 800 names from the city’s current residential utility-account list. Of those invited, 520 completed the survey. Sixty-one percent of respondents favored adding a late-night bus route. The report describes the result as the view of all city residents.”
Target population. The claim refers to all city residents. The report does not narrow this to account holders or to adults, so “all residents” is the stated target, though the survey may not cover that entire group.
Frame. The sampling frame is the current residential utility-account list. Residents without an account in their own name may be absent; for instance, some may live in shared housing or have service included in rent. This is a plausible coverage concern because the frame may not include every resident. The description alone does not tell us how many residents are missing or whether their opinions differ.
Selection. The report explicitly says names were randomly selected from the list. That supports random selection within the stated frame; it does not, by itself, make the list identical to the target population.
Response information. There were 800 invitations and 520 completions, so the response rate is
This establishes that 65% of the invited names completed the survey, assuming all 800 were eligible as described. It does not establish that the other 35% would have answered similarly.
Measurement and claim. The report supplies no question wording or response options, so we cannot judge whether “favored” was measured neutrally or whether respondents had options beyond favoring or opposing the route. The result directly describes the 61% of respondents who favored it. Extending that percentage to all residents needs caution because of possible frame undercoverage and nonresponse; the short description does not quantify either concern.
Separate Facts, Concerns, and Unknowns
A strong audit avoids two opposite errors: accepting a broad claim without examining its design, and declaring a result biased merely because a possible problem can be imagined. Use three categories in your notes:
- Reported facts: details the report explicitly provides, such as the invitation method, number invited, or wording of a survey question.
- Plausible concerns: a design feature that could cause some relevant people or views to be represented differently, such as excluding part of the target population from the frame.
- Unknowns: information needed to judge the extent or direction of a concern but not given in the description.
This distinction matters because bias is a systematic tendency for a survey result to differ from the value it aims to describe. A possible mechanism is not proof that a particular survey’s result is biased, and the direction of any bias may be unclear. For example, knowing that some residents lack a utility account establishes a possible frame mismatch; it does not tell us whether those residents are more or less likely to favor a bus route.
A report that uses a voluntary online poll has a different selection concern from a random sample drawn from a defined frame. If people choose for themselves whether to take part, those with stronger interest or stronger views may be more likely to respond. Still, an audit should describe this as a plausible selection concern unless the report provides evidence about how participants differ. Do not label every nonrandom sample in exactly the same way: state the actual invitation or selection process reported.
Worked Example: An Open Website Poll
A hypothetical local news site posts a poll asking, “Should the town invest more in safer crosswalks?” Readers can click one of two answers. The site reports 2,400 votes, with 72% answering Yes, and says that “town residents overwhelmingly support the investment.”
Target population. The claim is about town residents. The report does not say whether only residents could vote.
Frame and selection. The poll is open to site visitors who choose to click. The reachable group is therefore site users who encounter the poll, not a defined list of all town residents. Participation is voluntary rather than described as a random selection. Residents who do not visit the site, do not see the poll, or do not choose to vote are not represented in the same way.
Response information. The report gives 2,400 votes but no count of people invited or exposed to the poll. A response rate cannot be calculated from this description. The number of votes does not tell us how many town residents had an opportunity to participate.
Measurement and claim. The question presents a particular investment in terms of “safer crosswalks,” and the two answer options may not capture uncertainty or other views. Without further wording or context, it is difficult to judge the size of any measurement effect. The 72% is the share of recorded votes answering Yes, not a defensible estimate of the share of all town residents who support the investment. The key limitation is the open, self-selected participation, not simply the fact that the poll has a large number of votes.
Why Sample Size Does Not Repair a Design Mismatch
A large number of responses can make a reported percentage numerically stable among the people who responded, but it cannot make an unrepresented group appear in the data. This can be seen without a complicated formula. Suppose the target population contains two groups, A and B, with different views. If the frame entirely excludes group B, then every selected unit comes from A, regardless of how many times the sample size is increased.
For example, if every member of A opposes a proposal and every member of B supports it, a sample of 100,000 people drawn only from A would still report opposition among its respondents. Increasing the sample reduces the chance that a random sample from A happens to differ from A’s own pattern, but it does not change the fact that B had zero chance of selection. Thus, greater sample size may reduce sampling variability within the covered group while leaving a coverage problem unresolved. This is why a report’s sample size must be read alongside its frame and selection method.
Worked Example: A Membership Survey with Item Nonresponse
A hypothetical professional association emails a survey to all 1,000 current members. There are 700 completed surveys. One question asks whether members support changing the annual meeting location; 560 respondents answer that item, and 350 of those answer Yes. The report says, “Most members support moving the meeting.”
Target population and frame. The apparent target is current association members, and the frame is the current membership list. The description does not support a claim about former members, prospective members, or professionals who do not belong to the association.
Selection and response. The survey was emailed to the entire list, rather than a sample being selected from it. Of 1,000 members invited, 700 completed the survey, giving a completion rate of
That overall figure does not reveal whether those who completed the survey differ in their opinions from the 300 who did not.
Item response and calculation. Only 560 of the 700 survey completers answered the location question. Among item respondents, the Yes percentage is
So a precise description is that 62.5% of the members who answered this item supported the change. It would be inaccurate to calculate \(350/700\) and describe that as the proportion of item respondents who supported it: 140 completers did not answer this question. It would also overstate what is known to say that 62.5% of all members support the change. Unit nonresponse and item nonresponse leave different unanswered groups, and the short report gives no evidence about their opinions.
A Short Justification for the Checklist
Each checklist item concerns a link needed to extend a result beyond the observed answers. The target population specifies the group the claim concerns. The frame and selection method indicate which units could enter the data and how they entered. Response information distinguishes selected or invited units from those whose answers were observed. Measurement details clarify what those answers mean. If any link is missing or mismatched, the report may still accurately describe its respondents, but a broader population claim requires additional justification.
The logic can be stated compactly: a person can contribute an answer only if they are covered or reached, selected or invited, and then respond to the relevant item. A failure at any stage can make the observed answers differ systematically from the target group’s answers. The audit does not assume that such a difference occurred; it identifies where the report would need evidence to rule it out or assess its importance.
Common Mistakes
- Confusing respondents with the target population. “Sixty-one percent of respondents” and “61% of all residents” are different claims. The latter needs support beyond the raw respondent percentage.
- Assuming a random sample solves every problem. Random selection from a frame helps describe that frame; it cannot include people missing from the frame or ensure selected people respond.
- Treating response rate as a bias score. A low response rate signals that many selected people did not respond, but it does not by itself show the direction or size of nonresponse bias. A high rate does not prove the remaining nonrespondents are irrelevant.
- Calling an unknown a fact. If the report does not state how participants were chosen, do not assume they were randomly sampled—or assume the opposite. Record the selection method as not reported.
- Letting a large count replace design details. Many responses from a narrow or self-selected group do not automatically represent a broader population.
- Using the wrong denominator. The number answering a particular item may be smaller than the number invited or the number completing the whole survey. State which group a percentage describes.
Check Your Understanding
For each question, distinguish what the survey description establishes from what it leaves uncertain.
- A report claims to describe all county adults but sampled only people listed in a library newsletter database. What are the target population and the frame, and what possible concern follows?
- A poll reports 900 completed responses but does not state how many people were invited. Can its response rate be calculated? Explain.
- A newspaper website lets visitors choose whether to answer a poll. What does the reported percentage directly describe, and why might it not describe all residents?
- A survey invites 400 people; 300 complete it, but only 240 answer one particular item. Which denominator should be used to report the percentage answering that item, and why?
- A report states that it randomly selected people from a complete list of target-population members, but 40% did not respond. What does the selection information support, and what does it not establish about the final respondents?