Start With What the Question Is Asking
A research question helps determine how data should be collected. If researchers want to learn what people think or report doing, a survey may be a good tool. If they want to study conditions that already exist, they can use an observational study. If they want to find out whether a treatment causes a change, they may be able to conduct an experiment.
These choices are related, but they are not always three mutually exclusive categories. A survey is a way to collect information by asking people questions. An observational study gathers information without assigning treatments; researchers might use a survey to measure existing behaviors or opinions in such a study. An experiment deliberately imposes treatments and measures responses. Researchers might use a survey to measure an outcome in an experiment, but the study is classified as an experiment because of the imposed treatments, not because of the survey.
As discussed in Observational Studies and Their Limits, measuring an existing condition does not establish that it caused an outcome. As discussed in Experiments and Why They Can Show Cause, an experiment imposes treatments, and random assignment can support a cause-and-effect conclusion. The previous tutorial, Scope of Inference: Four Combinations, also explains why random selection and random assignment answer different questions. Keep those distinctions in mind when choosing a method.
A Practical Way to Choose
Read the research question closely. Does it ask what people prefer, believe, remember, or report? Does it ask how existing variables are related? Or does it ask whether changing one condition would affect an outcome? Those clues point toward different methods.
Decide whether the question asks for a description, a comparison or association, or an answer about cause and effect.
If researchers want to test an effect and can assign treatments appropriately, an experiment may fit. If they will only measure conditions that already exist, the study is observational.
For opinions, preferences, or behaviors that people can report, a survey may be useful. Existing records or direct observation may be more suitable for other variables.
Explain why it fits the question. Then distinguish what the method can show from what it cannot establish.
A survey is not automatically the best choice just because the participants are people. For example, researchers might record attendance from school records rather than ask students to remember how many days they attended. Conversely, a preference is usually not visible in a record or direct observation; researchers may need to ask people.
Worked Example: Estimating a Behavior With a Survey
Worked Example: Estimating a Behavior With a Survey
A school wellness committee asks, “What proportion of students at North Ridge High School bring a reusable water bottle to school at least three days in a typical week?” Which data collection method is appropriate?
This question asks for a description of a behavior across a defined population: students at North Ridge High School. Students can report how often they bring a bottle, so a survey is a reasonable method. The committee should define “typical week” and “bring” clearly so students interpret the question consistently. It should also use fair wording, as explained in Surveys and Designing Survey Questions.
To support a conclusion about all students, the committee could randomly select students from an up-to-date school roster and invite them to respond. Random selection is different from asking whoever happens to be nearby or posts a response to an open link. As described in Generalizing Results to a Population, a random sample can support generalizing to the population sampled from, subject to issues such as nonresponse and coverage.
The committee is not assigning students to bring bottles or not bring them. This is not an experiment. The survey can estimate the proportion who report the behavior, but it cannot establish that bringing a reusable bottle causes any other outcome.
Worked Example: Studying an Association With Survey Data
Worked Example: Studying an Association With Survey Data
A student research team asks, “Among students in the district, is the number of hours spent playing video games on school nights associated with the number of hours of sleep they get?” Which method fits?
The question asks whether two variables are associated. Students already have their own gaming and sleeping habits; the researchers are not assigning those habits. The study should therefore be observational, not an experiment. A survey could be the tool for collecting both self-reported variables. The team should define the time period and the meaning of “hours of sleep,” and should recognize that people may not recall or report their behavior exactly.
A careful plan would specify which students the question concerns, use a suitable sampling method, and ask neutral, clear questions. If the researchers randomly selected students from a complete district roster, the selection method could support generalizing the observed association to the population represented by that roster, subject to nonresponse and other limitations. If they used volunteers, they should not automatically generalize the results to all district students.
Even with a random sample, the team would not have assigned gaming time or sleep time. The data could show an association in context, but could not establish that gaming causes students to sleep less. Other variables, such as homework demands or work schedules, could be related to both. This example shows how a survey can be the collection method within an observational study.
Worked Example: Testing a Possible Cause With an Experiment
Worked Example: Testing a Possible Cause With an Experiment
A library wants to know, “Does sending a text reminder increase the proportion of students who return borrowed books by the due date?” Which method best matches the question?
The question asks whether a treatment—sending a reminder—changes an outcome—returning a book by its due date. If it is practical and appropriate to assign reminders, an experiment is a strong choice. For example, the library could randomly assign participating students who borrow books either to receive the usual information or to receive the usual information plus a text reminder. It would then record whether each book was returned on time.
The researchers have imposed different conditions, so this is an experiment. Random assignment can help make the groups comparable, supporting a cause-and-effect conclusion about the reminder for the participants in the experiment. Random assignment does not, by itself, make the participants representative of all students. As the previous tutorial explains, generalizing beyond participants depends on how they were selected.
A survey asking students whether they think reminders would help would answer a question about their opinions, not whether reminders actually increase on-time returns. The proposed experiment measures the behavior in question and compares outcomes under assigned conditions. Researchers should also consider whether they can obtain the necessary contact information and permissions, and should apply the reminder consistently within the assigned group.
Worked Example: Asking About Preferences
Worked Example: Asking About Preferences
A school dining committee asks, “Which of three proposed vegetarian lunch options do students prefer?” Which method is appropriate?
Preference is an opinion, so a survey is a suitable way to collect it. The committee can present the three options with neutral descriptions and ask each respondent to select one. If students may prefer more than one option or have no preference, the response choices should allow for those possibilities. Clear and balanced wording matters; a description that praises one option more than the others could lead responses.
The committee should decide whether it wants the preferences of all enrolled students or only those who eat lunch at school. That decision defines the population of interest. A randomly selected sample from the chosen population can better support generalizing the results to that population than an open invitation that mainly attracts students with strong opinions.
This survey measures stated preferences. It does not test whether students would actually choose or eat the options in a real lunch setting. If the committee wanted to compare actual selection or waste under different menus, it would need a different plan. The method should match the outcome the question names.
Worked Example: Comparing Existing Conditions
Worked Example: Comparing Existing Conditions
A community group asks, “Among students at two nearby schools, is commuting by bicycle associated with arriving late to school?” Which method fits?
Students already have their commute patterns, and researchers should not assign students to bicycle to school. This is an observational question. The group could obtain commute information through a survey and lateness information from attendance records, if it has appropriate access and permissions. Using records for lateness may be more accurate than asking students to recall it.
The group should define “commuting by bicycle” and the period over which lateness will be counted. It should also consider that the two schools may differ in distance, start time, or other factors related to both commuting and lateness. The study can describe whether bicycle commuting and lateness are associated in the data, but it cannot establish that bicycle commuting caused the observed difference.
This is an observational study even if a survey is used for one variable. If the group instead asked students whether they like bicycling to school, it would be asking about a preference and could use a survey to describe that opinion.
Common Mistakes and AP Exam Tips
- Treating “survey” and “observational study” as competing labels in every situation. A survey is a way to ask questions; an observational study is defined by researchers measuring existing conditions without assigning treatments. A survey can be part of an observational study.
- Choosing an experiment just because the question uses the word “effect.” Researchers need to be able to impose and assign treatments appropriately. If the condition already exists or cannot responsibly be assigned, an observational study may be more appropriate, but it cannot establish cause by itself.
- Using a survey when the question asks about actual behavior but a better measure is available. Self-reports can be useful, especially for private or hard-to-observe information, but records or direct measurement may better capture some outcomes.
- Assuming a survey automatically represents a population. How participants are selected matters. An open invitation or volunteer response is not the same as a random sample from a defined population.
- Confusing the data collection method with the strength of a conclusion. A survey can collect data for a descriptive study or an observational study. Neither survey wording nor a large number of responses creates random assignment or proves causation.
For a full-credit explanation, name the method and connect it to the research question. For instance: “Because the question asks whether an assigned reminder changes on-time returns, a randomized experiment is appropriate if reminders can be assigned responsibly. Random assignment can support a cause-and-effect conclusion for the participants, while the sampling method determines whether findings can be generalized.” For an existing behavior, say that an observational study can measure an association but cannot establish cause.
Check Your Understanding
For each research question, choose a suitable method and give a brief reason. For observational studies and experiments, identify whether treatments would be assigned.
- What proportion of residents in a town say they use public transportation at least once a week?
- Among students, is the number of hours spent working at a paid job associated with the number of hours spent on homework?
- Does a new practice schedule improve free-throw accuracy compared with the team’s usual schedule?
- Which design of a community recycling flyer do residents find easiest to understand?
- Is participation in an after-school club associated with attendance at school?