Designing Questions People Can Answer Fairly
In Experiments and Why They Can Show Cause, you learned that an experiment deliberately assigns treatments, while an observational study records conditions that already exist. A survey is a way to collect information by asking people questions. How those questions are written and arranged matters: unclear or unbalanced questions can make answers hard to interpret or can nudge respondents toward particular answers.
A fair survey question gives respondents a reasonable opportunity to report what they think, know, or do without confusion or pressure. Fairness does not mean every respondent must interpret a question identically, but good design reduces avoidable differences in interpretation. As in Deciding Which Variables to Measure, start by being clear about what information the investigation needs. Then check that the question actually collects that information.
A useful question should be understandable to the people being surveyed, focus on one main idea, and specify important details such as a timeframe. Response options should fit the question and make it possible for different respondents to select an answer that genuinely describes them. These checks do not guarantee that a survey represents a population; they concern the quality of the questions themselves.
A Fairness Check for a Survey Question
When reviewing a question, consider its wording and its answer choices together. A question may sound clear but have response options that do not fit it. Or the response choices may be sensible, while the question itself leaves respondents unsure what period or behavior to consider.
State what information the survey should collect, such as a person’s number of visits to a library during a specified month.
Look for undefined terms, unclear time periods, or multiple questions joined together. Ask whether respondents can tell exactly what they are being asked.
Look for praise, blame, assumptions, or emotional language that could make one answer seem more acceptable than another.
Check that categories do not overlap, that likely answers are covered, and that options such as “not sure” or “prefer not to answer” are considered when appropriate.
Ask whether an earlier question or the order of choices could affect how respondents understand or answer this question.
A small pretest can reveal confusing wording or missing choices. Ask people what they thought the question meant, not just whether they liked it.
Wording: One Clear Target at a Time
A respondent needs to know what to answer. A question about “regular exercise,” for example, leaves room for different interpretations: one person might count a walk to school, while another might count only planned workouts. If the intended variable is exercise sessions, define what counts as a session and specify a time period.
A question should also avoid asking about two things at once. “How satisfied are you with the cost and convenience of the bus service?” asks about both cost and convenience. Someone may be satisfied with one but not the other, with no clear way to provide that answer. Separate questions can measure the two variables.
Use wording the intended respondents can understand. Technical terms, vague adjectives such as “often,” and broad phrases such as “a healthy amount” may mean different things to different people. A specified timeframe or a brief definition can help respondents use the same reference point. However, added details should clarify the question rather than praise or criticize a particular answer.
This tutorial introduces a broad fairness check; the next tutorial examines biased and leading wording more closely. For now, the key question is whether the language makes it easier to answer accurately or pushes respondents toward an answer.
Response Options: Make Room for Reasonable Answers
For a question with fixed response options, two useful goals are that the choices be mutually exclusive and collectively exhaustive. Mutually exclusive choices do not overlap: a respondent should not fit equally well into two categories. Collectively exhaustive choices cover the plausible answers for the group being surveyed.
For example, age categories “10–15,” “15–20,” and “20–25” overlap at ages 15 and 20. A respondent who is 15 may not know which category to choose. Categories “10–14,” “15–19,” and “20–24” do not overlap, but may still fail to cover everyone if the survey includes people younger than 10 or older than 24. The options should be designed for the actual population of respondents.
Sometimes an “other” option with a space to specify an answer helps cover responses that the designer did not anticipate. “Not sure,” “not applicable,” or “prefer not to answer” may also be appropriate, depending on the purpose and sensitivity of the question. They are not automatic additions to every survey: including irrelevant choices can distract or make a question harder to use.
Response scales should match the question. If asking about agreement, offer a consistent set of agreement choices, with a clear direction from disagreement to agreement. If asking for a count, categories should represent counts rather than opinions. If a middle option is included, it should have a clear meaning, such as neither agreeing nor disagreeing, rather than serving as an unexplained catch-all.
Order and the Surrounding Survey
Question order can affect responses. An earlier question may bring a topic to mind, change how a later question is understood, or make a particular response seem more expected. For instance, asking respondents to describe recent delays before asking them to rate a transit service may make delays especially prominent as they form their rating.
A thoughtful survey usually begins with clear, relevant questions that are easy to answer and places more sensitive or demanding questions later, when appropriate. Related questions may be grouped together, but the group should not frame a later answer as if a particular opinion were expected. Keep instructions clear when the survey changes topics.
The order of response options can matter too, especially in a long list. A balanced scale should use consistent labels and direction. If the order of a list of options has no natural sequence and one position might attract more attention, survey designers may rotate or randomize the order when the survey method allows. This is a design choice, not a fix for unclear wording.
Pretesting is especially useful for detecting order problems. Ask a few people who resemble the intended respondents to complete the survey and explain how they interpreted particular questions. If answers change after a preceding question, or people interpret the same phrase in different ways, the designer has evidence that the survey needs revision.
Worked Example: Counting Visits to a Community Garden
Worked Example: Counting Visits to a Community Garden
A community group wants to record how frequently neighborhood residents use a shared garden. Its draft question is: “Do you regularly visit our wonderful community garden?” The response choices are “Yes” and “No.” Evaluate the question and propose a fairer version.
The intended variable appears to be how often a resident visits the garden, but “regularly” is not defined. One person might count a visit each week as regular, while another might require daily visits. “Wonderful” expresses approval and may make a positive answer seem more welcome. The yes-or-no choices also do not measure frequency and leave the meaning of a “Yes” response unclear.
A more useful question would be: “How many times did you visit the community garden during September? Count any visit, even if it lasted only a few minutes.” Response options could be “0,” “1,” “2–3,” “4–7,” “8 or more,” and “Not sure.” The defined month gives respondents a shared timeframe, and the instructions specify what counts as a visit. The categories are nonoverlapping and cover counts of zero or more; “Not sure” gives respondents a way to report uncertainty instead of guessing.
The group should verify that September is the intended period and that respondents can recall visits over that month. If the survey is conducted well after September, a shorter or more recent timeframe may be easier to remember. The revision makes the measured information more precise without assuming that residents like or dislike the garden.
Worked Example: Rating a School Lunch Service
Worked Example: Rating a School Lunch Service
A student committee wants to measure students’ satisfaction with the time they spend waiting in the lunch line. It proposes the question: “How satisfied are you with the fast, convenient lunch service and the quality of the meals?” The choices are “Satisfied,” “Neutral,” and “Dissatisfied.” Explain what needs improvement and give a revised question with suitable options.
The question combines at least two topics: waiting time and meal quality. It also calls the service “fast” and “convenient” before asking students to judge it. A student who thinks the line is slow but the meals are good cannot give a clear answer. The committee should separate the topics and avoid describing the service in a way that already suggests a favorable judgment.
For the intended variable, a revised question could be: “During the past five school days, how satisfied or dissatisfied were you with the amount of time you waited in the lunch line?” Possible choices are “Very satisfied,” “Somewhat satisfied,” “Neither satisfied nor dissatisfied,” “Somewhat dissatisfied,” “Very dissatisfied,” and “I did not use the lunch line during those days.” Each choice has a distinct meaning, the scale runs in a consistent order, and the last choice accommodates students for whom the question does not apply.
The committee should ask about meal quality in a separate question if that information is also needed. It should also check whether “past five school days” is easy for students to use as a reference period. The revised waiting-time question does not measure how students feel about the food, and its results should not be presented as if it did.
Worked Example: Reviewing Question Order
Worked Example: Reviewing Question Order
A city youth council wants to ask teens how safe they feel using a local park. Its draft survey first asks, “Have you ever witnessed vandalism or another upsetting incident in the park?” It then asks, “How safe do you feel when visiting the park?” Explain a possible order effect and suggest a fairer approach.
The first question directs attention to upsetting incidents. That context could make incidents especially noticeable when a respondent answers the safety question. A person might report feeling less safe after being prompted to recall an incident, even if the intention was to measure their overall impression before that reminder.
The council could ask the general safety question first: “During the past month, how safe or unsafe have you felt while visiting the park?” It could use a balanced scale such as “Very safe,” “Somewhat safe,” “Neither safe nor unsafe,” “Somewhat unsafe,” and “Very unsafe,” plus “I did not visit the park during the past month.” If the council also needs information about witnessed incidents, it can ask that in a separate question afterward.
The council should not conclude that changing the order eliminates every possible influence. It should pretest the full survey with teens similar to the intended respondents and ask them how they interpreted the questions. If order is central to the investigation, the council could compare versions with different orders, while recognizing that a small pretest mainly helps uncover problems rather than establish a population-wide effect.
Common Mistakes and AP Exam Tips
- Checking only whether a question is grammatically correct. A sentence can be readable but still combine multiple topics, use vague terms, or omit a timeframe. Explain how the wording affects what respondents can report.
- Calling options “complete” without checking who will answer. The choices must cover plausible responses for the actual respondent group. An age survey for students, for example, may need different categories from an age survey for all adults.
- Using categories that overlap. If a respondent fits two options, the categories are not mutually exclusive. Set clear, nonoverlapping boundaries.
- Adding every possible fallback option to every question. Include “not sure,” “not applicable,” or “prefer not to answer” when it suits the question and respondent group; make sure the option has a clear purpose.
- Ignoring the surrounding questions. The question being evaluated may seem neutral by itself, but an earlier question can affect what respondents have in mind. Consider the sequence as well as the wording.
- Revising without explaining why. For a strong AP response, identify a specific design problem, describe how it could affect answers, and give a concrete improvement. For example: “The choices ‘10–15’ and ‘15–20’ overlap at 15, so a respondent who is 15 could fit both. Change them to ‘10–14’ and ‘15–19’.”
When evaluating a survey, be specific. Name the unclear phrase, overlapping categories, missing option, or order concern. Then connect the proposed change to the information the survey is meant to collect. “Make the question less biased” is too general to show what should change; explain which words or choices cause the problem.
Check Your Understanding
For each situation, identify a fairness concern and describe a specific improvement.
- A survey asks, “Do you often use the school gym?” What detail could be added to make the question easier to answer consistently?
- A question asks, “How satisfied are you with the library’s hours and its study spaces?” What is the problem with measuring these topics together?
- A survey offers age ranges “12–16,” “16–20,” and “20–24.” Explain the issue and revise the category boundaries.
- A survey asks about respondents’ overall opinion of a park immediately after asking whether they have witnessed a frightening incident there. Describe a possible order effect and one way to address it.
- When might “I did not use this service during the past month” be a useful response option? Explain what kind of question it should accompany.