Start With the Question You Need to Answer
In “Four Statistical Practices in Context,” you saw that a useful investigation connects its question, data collection, analysis, and conclusion. This tutorial focuses on the first two links: sharpening the question and choosing a plan to collect data that can answer it. A plan should not be chosen just because it is convenient. It should fit the claim you hope to make.
A broad topic such as “student wellness” is a starting point, not yet a statistical question. To make it answerable, decide who or what the investigation concerns, what information will be recorded, and what feature of the data you want to learn about. Then decide how to obtain observations from the relevant people or process.
The wording matters. “Does a reminder help?” does not say who receives it, what kind of reminder is meant, or what “help” means. “Among students who volunteer for a study-skills program, does assignment to receive a daily text reminder rather than no reminder change the number of planned study sessions completed in two weeks?” identifies a group, a comparison, and an outcome that can be recorded.
A well-formulated question also keeps its intended scope clear. A survey of one school can directly describe the students represented by its sampling plan; it does not automatically answer a question about every student in the region. Likewise, an experiment with volunteers can compare treatments for those participants, but volunteer recruitment alone does not make them representative of a broader population.
Match the Data-Collection Design to the Goal
A sample survey asks a selected group of people questions to learn about a population. It is a natural choice when the goal is to estimate or describe a population’s answers, such as the proportion of residents who use a public trail. The selection method matters: a survey of people who happen to be on the trail may not represent residents who rarely or never visit it.
An observational study records information about individuals or conditions without assigning a treatment. It can describe a group or examine an association, but an observed association by itself does not establish that one variable caused changes in another. For example, recording how much sleep students report and how many late assignments they have does not assign anyone a sleep schedule; other differences between students could be related to both measurements.
An experiment deliberately assigns treatments or conditions to individuals and measures a response. When treatments are randomly assigned, differences in the response between groups can provide evidence about a cause-and-effect relationship for the individuals and conditions studied. Random assignment is different from random sampling: random assignment helps create comparable treatment groups, while random sampling helps obtain a sample that represents a population.
Sometimes a census—collecting information from every individual in the population—is possible, but it can be costly, slow, or impractical. Even when everyone is invited to respond, missing answers can still limit what the results represent. A carefully planned sample may be more useful than an attempted census with substantial nonresponse.
Choose a Sampling Plan That Fits the Population
A simple random sample (SRS) of a specified size is selected so that every possible sample of that size has an equal chance of selection. A complete list of the population, such as an enrollment roster, can provide a sampling frame: the list from which individuals are selected. A random-number generator can then select the individuals. An SRS is not the same as asking whoever is easiest to reach.
Other probability sampling plans can be useful when the population has meaningful groups or is spread across many locations. In a stratified random sample, divide the population into relevant groups, called strata, and randomly select individuals from each stratum. This ensures that each stratum is represented in the sample. In a cluster sample, divide the population into natural groups, randomly select some groups, and collect data from all individuals in the selected groups. Clusters can make data collection more practical when individuals are geographically spread out.
A systematic sample selects individuals at a regular interval from an ordered list, after choosing a random starting point. For example, a researcher might randomly choose a starting position and then select every twentieth name. Before using this approach, consider whether the list’s order has a repeating pattern related to the question; a pattern could make the selection unrepresentative.
Convenience samples include individuals who are easy to reach. Voluntary response samples rely on people choosing to participate, often after an open invitation. Both methods can be affected by selection bias: people included or willing to respond may differ in relevant ways from those not included. A large sample does not, by itself, fix a biased selection process. Also consider nonresponse among selected individuals and whether the question or measurement method could lead to inaccurate answers.
Worked Example: A Survey About School Bus Use
Worked Example: Estimate Students’ Usual Transportation
Situation. A fictional high school wants to describe how its students usually travel to school. The school has 1,200 students, with 300 students in each of grades 9–12. The broad request “find out about transportation” needs a specific outcome and a plan.
Formulate the question. The population is all 1,200 students enrolled at the school. The individuals are students, and the variable is each student’s usual way of traveling to school, recorded in categories such as school bus, family vehicle, walking, or bicycle. A focused question is: “What proportion of currently enrolled students in grades 9–12 usually travel to school by school bus?” This specifies the group, the response, and the feature to describe.
Choose a sampling plan. Because the school wants a result for all grades and grade might be related to transportation, use a stratified random sample. Treat each grade as a stratum. From each grade’s roster, use a random-number generator to select 20 students, for a total sample of 80. Selecting from every grade ensures that each grade is included. The school should ask the same clearly worded question of each selected student and make reasonable efforts to obtain responses from those who do not answer at first.
An alternative would be an SRS of 80 students from the full enrollment roster. That is also a probability sample, but it might include relatively few students from a particular grade just by chance. A survey link posted for anyone to answer would instead be a voluntary response plan: students who care strongly about transportation could be especially likely to reply. That plan would not be as defensible for describing all students.
Connect the plan to the conclusion. If the stratified sample is carried out as planned and response is sufficiently complete, its results can provide evidence about the school’s enrolled students. The plan does not support a conclusion about students at other schools. If some selected students do not respond, the school should consider whether their transportation patterns might differ from those who did respond. The question and plan fit because the recorded categorical response directly answers the population-description goal.
Worked Example: Test a Study Reminder
Worked Example: Plan a Randomized Experiment
Situation. A fictional learning center wants to know whether a daily text reminder changes how consistently students complete planned study sessions. It recruits 40 students who agree to participate for two weeks. Each student records the number of planned sessions completed, from 0 to 10.
Formulate the question. The explanatory variable is assigned reminder condition: a daily text reminder or no reminder. The response variable is the number of planned study sessions completed during the two weeks. A focused question is: “For the 40 participating students, does assignment to receive a daily text reminder rather than no reminder affect the number of planned study sessions completed over two weeks?”
Plan the experiment. Use a chance process to assign 20 participants to the reminder group and 20 to the no-reminder group. Give both groups the same definition of a planned study session, the same recording instructions, and the same two-week period. The only planned difference between the groups should be the reminder condition. At the end, collect each participant’s recorded number of completed sessions.
Random assignment helps make the groups comparable before the treatment, reducing the chance that a pre-existing difference systematically determines who receives reminders. The equal group sizes also make the design straightforward to carry out. The plan measures a specific response rather than leaving “study more” open to different interpretations.
State what the design can support. If the reminder group completes more sessions on average, the randomized comparison can provide evidence that the reminder affected completion for these participants during this two-week period. This design does not automatically show that the same effect would occur for all students: participants volunteered, so they may differ from students who did not join. It also does not answer whether reminders affect grades or study habits over a longer period, because neither outcome is being measured.
The key design choice follows from the causal wording “does assignment ... affect.” Merely surveying students about whether they use reminders and how much they study would be an observational study. Students who choose to use reminders could differ from those who do not, so that comparison alone would not isolate an effect of reminders.
Worked Example: Study Sleep and Late Assignments
Worked Example: Plan an Observational Study
Situation. A fictional student research team wonders whether sleep and late assignments are related among students at a high school. The team cannot assign students how much to sleep and wants to study current habits.
Formulate the question. The population is students enrolled at the school. The team will record each participating student’s usual hours of sleep on school nights and the number of assignments turned in late during the previous four weeks. A focused question is: “Among students enrolled at this school, what association is there between usual school-night sleep and the number of assignments turned in late during the previous four weeks?”
Choose a plan. The team can draw a random sample from the enrollment roster, invite the selected students to participate, and ask all respondents the same questions. It should define “usual school-night sleep,” specify how students should report hours, and use the same four-week window for the late-assignment count. Those definitions make the intended measurements clearer. The team should also protect privacy because assignment records and sleep reports may feel personal.
The team is recording existing characteristics, not assigning sleep amounts or changing students’ deadlines. Therefore, this is an observational study. A random sample can help the team obtain information about students on the roster, although nonresponse or inaccurate self-reports could limit the result. The study could describe an association in the observed data, but it could not establish that sleep causes fewer late assignments. Workload, health, or other factors could be related to both sleep and assignment completion.
If the team instead invited students to complete an open online survey, the respondents would be self-selected. That may still produce interesting descriptions of those who answer, but it is a weaker basis for claims about all enrolled students. The sampling plan and the wording of the conclusion must remain connected.
Common Mistakes and AP Exam Tips
- Asking a topic instead of a statistical question. “Study habits” names a subject but not what will be learned. Full-credit wording identifies the population and the variable or comparison of interest.
- Using an outcome that is not defined. Words such as “healthy,” “successful,” or “helpful” can mean different things. State what will actually be recorded, such as reported hours of sleep or number of completed sessions, and over what period.
- Confusing random sampling and random assignment. Randomly selecting participants helps with representing a population; randomly assigning treatments helps with a cause-and-effect conclusion. Name the method that was actually used and what it supports.
- Calling an observational comparison an experiment. Recording who already uses a reminder does not assign reminders. Without assigned treatments, a difference between groups may reflect other differences between the people in them.
- Assuming a large sample eliminates bias. Many responses do not fix undercoverage, voluntary response, or nonresponse. Explain which people could be missed or more likely to participate and how that could affect the intended description.
- Making the conclusion broader than the plan. A sample from one school does not directly represent every school, and volunteers do not necessarily represent all students. State the group and setting the design actually covers.
On an AP response, make the chain explicit: give a focused question, name the individuals and variables, describe how the sample or treatment groups will be formed, and explain what the resulting evidence could support. For a limitation, name the design feature and its consequence. For example, “Because participants volunteered, they may differ from students who did not volunteer, so the results may not generalize to all students” is more informative than simply saying “the sample is biased.”
Check Your Understanding
For each prompt, connect the question to a suitable data-collection plan and an appropriately limited conclusion.
- A community center wants to estimate the proportion of local adults who use its fitness room. What population, variable, and sampling method could be specified?
- Why might surveying only adults who are inside the fitness room give a misleading answer to the community center’s question?
- A researcher records students’ existing use of a planner and their assignment completion. Is this an experiment or an observational study? What kind of conclusion is appropriate?
- A teacher wants to test whether a new review activity affects quiz scores. What should be randomly assigned, and what response should be measured?
- Explain the difference between random sampling and random assignment, including the kind of conclusion each can help support.