How to Critique a Study Description
A study description may report a striking difference between groups, but the difference alone does not tell us what caused it or to whom it applies. To critique the design, trace how the groups came to differ, what was measured, and what alternative explanations remain. The goal is not to dismiss a study whenever it has a limitation. It is to identify both the evidence it provides and the boundary of the conclusion it can support.
A useful critique answers four connected questions: What kind of design was used? What could make the groups differ besides the exposure or treatment of interest? What features strengthen or weaken the comparison? And what is the strongest conclusion warranted by the design and the people actually studied?
These categories describe how a study was conducted, not how impressive its results appear. A study can use a large sample and precise measurements yet remain observational. Conversely, an experiment can use random assignment but have limitations in recruitment, measurement, or follow-up. Each feature addresses a different question.
| Design feature | What it can strengthen | What it does not establish by itself |
|---|---|---|
| Random assignment to conditions | A causal comparison between assigned conditions, if the experiment is carried out appropriately | That participants represent a wider population |
| Random sampling from a population | Representativeness for that population, subject to nonresponse and sampling limitations | A causal effect of an exposure |
| A control or comparison group | A reference for judging what happened under another condition | That the groups were comparable in every important respect |
| Repeated measurements or careful follow-up | Clearer information about timing and measured outcomes | That no unmeasured factor explains the results |
A Practical Critique Sequence
Read the methods before interpreting the headline. In particular, look for the action verbs: did researchers assign, randomly assign, select, recruit, or simply record? Then determine what counts as one participant or experimental unit, what the groups received or experienced, how outcomes were measured, and when measurements were taken.
If researchers assigned the condition, it is an experiment; if participants' exposures were already present or self-selected, it is observational. Note whether assignment was random.
Identify the groups, the outcome, the time period, and the quantity being compared. Do not substitute a different outcome or population for the one actually studied.
Ask what differences between groups could influence the outcome. Consider how each plausible confounder could be associated with both group membership and the measured result.
Consider assignment, comparison conditions, measurement, follow-up, recruitment, and missing outcomes. A strength in one area does not erase a limitation in another.
State whether the result is a sample description, an association, or evidence of a causal effect of an assigned treatment. Then name important limits on scope.
Worked Critiques
Worked Example: A Self-Selected Exercise Program
Hypothetical study. A clinic follows 120 adults for six months. Sixty choose to join a walking program, and 60 do not. At the end of follow-up, the program group has an average systolic blood-pressure decrease of 5 mm Hg; the other group has an average decrease of 1 mm Hg. The study reports that the difference in average change is 4 mm Hg.
Design. This is a prospective observational study: researchers followed participants over time, but participants chose whether to join. They were not assigned to the walking program.
Comparison and arithmetic. The reported difference in average decrease is \(5-1=4\) mm Hg. This describes the contrast between these two groups over the stated six months. It does not, on its own, show that walking caused a 4 mm Hg decrease.
Plausible confounding. People who choose a walking program may differ in baseline health, motivation, medication use, diet, or access to care. For example, if participants who joined were also more likely to start medication or change their diet, those changes could contribute to the blood-pressure difference. The study description does not tell us whether these factors were measured or adequately accounted for.
Strengths and limits. Following participants over six months establishes the order in which participation and later measurements occurred more clearly than a single cross-sectional snapshot would. But the self-selected groups may differ for reasons other than program participation, and the description does not establish that the 120 adults represent a broader population.
Strongest conclusion. In this hypothetical group, adults who chose the walking program had an average decrease 4 mm Hg greater than adults who did not join over six months. The result is an association, not proof that the program caused the difference.
Worked Example: A Randomized Reminder Experiment
Hypothetical study. Researchers recruit 80 volunteers through campus notices to test weekly text reminders intended to encourage exercise. They randomly assign 40 volunteers to receive reminders and 40 to receive no reminders. After eight weeks, the reminder group reports an average of 154 minutes of exercise per week, compared with 118 minutes in the no-reminder group.
Design. This is an experiment with random assignment. The treatment is assignment to receive reminders; the comparison condition is no reminders. It is not a randomized sample of all students or adults, because the volunteers came from campus notices.
Observed comparison. The difference in reported average exercise is \(154-118=36\) minutes per week. Thus the assigned-reminder group reported 36 more minutes per week on average at the end of the study.
Why random assignment matters. If assignment is genuinely random, each volunteer has the same chance of being placed in either condition. Before assignment, characteristics such as starting exercise habits are therefore not systematically used to decide treatment. Random assignment tends to make such characteristics comparable across groups, although a particular assignment can still leave chance imbalances.
Here the equation describes a pretreatment characteristic measured in the two groups: under random assignment, the expected difference in its group averages across all possible assignments is zero. It does not say the observed groups must have identical characteristics, nor that the outcome difference must be zero. It explains why assignment helps rule out systematic baseline differences as an explanation in the long run.
Strengths and limits. Random assignment supports a causal comparison of the assigned conditions if participants' outcomes are measured comparably, follow-up is adequate, and one participant's assignment does not substantially alter another's outcome. Because exercise was self-reported, reporting differences could affect the measured outcome. The description also does not say whether all 80 completed follow-up. And volunteers recruited at one campus may differ from people who did not volunteer.
Strongest conclusion. Among these volunteers, assignment to weekly reminders produced evidence of an increase in reported exercise relative to no reminders over eight weeks, with an observed difference of 36 minutes per week. The design supports a causal interpretation for the effect of assignment in this experiment more strongly than the observational comparison did. It does not establish the same effect for all students or for people outside the recruited group.
Worked Example: A Policy Change Across Districts
Hypothetical study. One school district introduces a later start time in September. A neighboring district keeps its usual schedule. Researchers compare average student absences during the autumn term: 6 days per student in the district with the later start and 8 days in the district with the usual schedule.
Design. Researchers did not assign districts to start times. This is a nonrandomized comparison of existing policy conditions, often described as a quasi-experiment when researchers study a policy change as it occurs. Calling it a quasi-experiment does not mean that districts were randomly assigned.
Observed comparison. The district with the later start recorded \(8-6=2\) fewer absences per student on average during the term. This is a difference between district averages, not a difference between otherwise identical individual students.
Possible confounding. Districts may differ in transportation, student health, attendance rules, family circumstances, or other policies. Any of these could be related to both which schedule a district uses and how often students are absent. If the later-start district also changed its attendance policy, for example, that change could affect recorded absences independently of start time.
Strengths and limits. The comparison concerns a real policy in its usual setting, and researchers can measure a practical outcome. However, the policy was not randomly assigned to districts, and this description gives no evidence that the districts were comparable before the change. A one-term comparison cannot separate the schedule's possible effect from pre-existing district differences or other changes occurring at the same time.
Strongest conclusion. During the observed autumn term, the district using the later start had an average of 2 fewer absences per student than the neighboring district. The contrast is consistent with a possible benefit, but this description alone does not establish that the later start caused the difference.
Why the Conclusion Depends on the Design
The examples show that a numerical difference is not the same thing as a causal effect. In the exercise-program study, participants selected their exposure, so the groups could differ in ways that also affect blood pressure. In the reminder experiment, random assignment gives a design-based reason to attribute a difference to the assigned condition, provided the experiment is implemented and measured appropriately. In the district comparison, a policy change may be informative, but nonrandomized districts can differ in other relevant ways.
Two distinctions help keep the conclusion precise. First, random assignment concerns how experimental conditions are allocated and strengthens causal comparisons. Random sampling concerns how units are selected from a target population and informs how broadly sample results may generalize. One does not automatically provide the other. Second, a study's strongest conclusion is not necessarily the boldest possible explanation. It is the strongest statement that survives the design's plausible alternative explanations and the limits of its participants and measurements.
A useful final check is to complete this sentence: “In the units studied, group A had [specified outcome] compared with group B during [time period]; because [design feature], the result [does or does not] support a causal interpretation, and it applies most directly to [population or setting].” If a study is observational, describe an association rather than asserting that the exposure caused the outcome. If it is randomized, identify the assigned treatment and avoid generalizing beyond the studied setting without support.
Common Mistake
Wrong reasoning: “The group with the better outcome received the treatment, so the treatment worked.” This skips the question of how groups were formed. People may have selected their own exposure, or different locations may have adopted different policies; pre-existing differences can help explain the outcome.
Correction: Identify assignment before making a causal claim. Random assignment can make a causal interpretation more credible by balancing other factors in expectation. Without it, state the observed association, identify plausible confounders, and avoid treating a comparison as proof of cause. Even with random assignment, check whether the outcome was measured well, follow-up was adequate, and the conclusion stays within the study's scope.
Check Your Understanding
For each question, give a reason based on the study design rather than relying only on the reported result.
- A survey finds that people who drink more tea report fewer headaches. Researchers did not assign tea consumption. What kind of conclusion is supported, and name one possible confounder.
- A randomized experiment reports different average outcomes in its two groups. What does random assignment strengthen, and what does it not guarantee about the participants?
- A study randomly samples people from a city but observes their existing diets. Does random sampling by itself establish that diet caused an observed health difference? Explain.
- Two districts have different attendance averages after only one adopts a policy. Give one reason the comparison may be confounded and state a cautious conclusion.
- In a randomized reminder study, why is it important to distinguish the effect of assignment to reminders from the claim that reminders would work equally well for everyone?