Why a Comparison Can Be Misleading
Suppose people who exercise regularly have better health outcomes than people who do not. It is tempting to attribute the difference to exercise. But the two groups might also differ in age, starting health, or another characteristic related to the outcome. If that other characteristic is unevenly distributed between the groups and also affects the outcome, the comparison mixes together more than one difference.
This problem is called confounding. It matters because a numerical difference between groups does not, by itself, tell us which difference produced the result. Before interpreting a comparison, ask not only how the groups differ in their treatment or explanatory variable, but also whether they differ in other factors related to the response.
For example, if people who exercise more are younger on average, and age is related to a health outcome, age may be a confounding factor in a comparison of exercise levels and that outcome. It is not enough that age is present in the data. The concern is that age differs between the exercise groups and is also related to the response.
Confounding does not mean that the recorded values or calculations are wrong. Group percentages can be calculated correctly and still give a misleading picture of what the treatment itself accounts for. Nor does noticing a possible confounder tell us exactly how much it changed the result. It signals that the observed comparison mixes influences that cannot be separated from that comparison alone.
Worked Example: Exercise, Age, and Health
Worked Example: Exercise, Age, and Health
In a hypothetical observational study, researchers compare the percentage of people experiencing a health complication among those who exercise regularly and those who do not. They also record whether each person is in a younger or older age group. The counts are:
| Age group | Regular exercise | Complication | No complication | No regular exercise | Complication | No complication |
|---|---|---|---|---|---|---|
| Younger | 90 | 9 | 81 | 20 | 4 | 16 |
| Older | 10 | 4 | 6 | 80 | 40 | 40 |
First, calculate the overall percentages. Among those who exercise regularly, \(9+4=13\) of \(90+10=100\) people have a complication:
Among those who do not exercise regularly, \(4+40=44\) of \(20+80=100\) people have a complication:
The overall complication rate is 31 percentage points lower in the regular-exercise group. But the age composition is very different: 90 of 100 regular exercisers are younger, compared with only 20 of 100 people who do not exercise regularly.
Now compare within each age group. Among younger people, the complication rates are \(9/90=10\%\) for regular exercisers and \(4/20=20\%\) for those without regular exercise, a 10-percentage-point difference. Among older people, the rates are \(4/10=40\%\) and \(40/80=50\%\), again a 10-percentage-point difference.
Age is a plausible source of confounding here: the exercise groups have very different age compositions, and complication rates also differ between age groups. The overall 31-percentage-point contrast combines the differences in exercise status and age composition. The table alone does not establish that exercise caused either the overall difference or the within-age differences.
A Simple Algebraic Explanation
A short model makes the entanglement explicit. Suppose \(T\) is an indicator of treatment group (\(T=1\) for treated and \(T=0\) for untreated), \(Z\) represents another factor, and \(Y\) is a numerical response. For illustration, assume an additive model:
Here, \(\tau\) is a constant treatment contribution to the response, \(\gamma\) is the contribution associated with \(Z\), and \(\varepsilon\) represents other influences. Assume that the mean of \(\varepsilon\) is the same in the two treatment groups. Taking the mean response in the treated group and subtracting the mean response in the untreated group gives:
The final expression separates the treatment contribution, \(\tau\), from the contribution associated with the groups’ difference in \(Z\). If the mean of \(Z\) differs between groups and \(\gamma\) is not zero, the observed mean difference includes both terms. This is the basic algebra of confounding in this simplified model. Real studies may be more complex, but the central issue is the same: group differences in another response-related factor can become entangled with the treatment comparison.
Worked Example: Tutoring and Academic Performance
Worked Example: Tutoring and Academic Performance
In a hypothetical school comparison, students who received tutoring and students who did not are grouped by their prior achievement. The response is whether each student passes a later exam.
| Prior achievement | Tutored students | Pass | Not pass | Students without tutoring | Pass | Not pass |
|---|---|---|---|---|---|---|
| Lower | 80 | 32 | 48 | 20 | 10 | 10 |
| Higher | 20 | 16 | 4 | 80 | 72 | 8 |
Overall, \(32+16=48\) of 100 tutored students pass, a pass rate of \(48/100=48\%\). Among students without tutoring, \(10+72=82\) of 100 pass, a pass rate of \(82/100=82\%\). A quick comparison might suggest that tutoring is associated with a lower pass rate.
But prior achievement is distributed very differently between the groups. Lower-prior-achievement students make up \(80/100=80\%\) of the tutored group, but only \(20/100=20\%\) of the group without tutoring. Prior achievement is also related to passing: in both groups, the higher-prior-achievement students have a higher pass rate.
Within the lower-prior-achievement group, the pass rates are \(32/80=40\%\) for tutored students and \(10/20=50\%\) for students without tutoring. Within the higher-prior-achievement group, the rates are \(16/20=80\%\) and \(72/80=90\%\), respectively. These observed comparisons still do not isolate a tutoring effect: students may differ in other ways, and the study is observational. The overall 48% versus 82% comparison is especially difficult to interpret because it also reflects the very different distribution of prior achievement.
The correct diagnosis is not “tutoring lowers achievement.” It is that the crude comparison is confounded by prior achievement, which is related both to who receives tutoring and to passing. Further design or analysis would be needed to separate these influences.
Worked Example: Random Assignment and Chance Imbalance
Worked Example: Random Assignment and Chance Imbalance
A hypothetical small experiment randomly assigns 20 students to a new study program or usual study support. Before assignment, each student is classified as having lower or higher prior achievement. The response is a numerical score improvement. The group summaries are:
| Prior achievement | New program: students | Mean improvement | Usual support: students | Mean improvement |
|---|---|---|---|---|
| Lower | 8 | 5 points | 2 | 6 points |
| Higher | 2 | 1 point | 8 | 2 points |
The overall mean improvement in the new-program group is the weighted mean:
The overall mean in the usual-support group is:
Thus, the overall difference is \(4.2-2.8=1.4\) points in favor of the new-program group. Yet in each prior-achievement category, the displayed mean improvement is one point lower in the new-program group: \(5-6=-1\) among lower-prior-achievement students, and \(1-2=-1\) among higher-prior-achievement students.
The groups have a pronounced chance imbalance in prior achievement: 8 of 10 new-program students are in the lower category, while only 2 of 10 usual-support students are. Since the summaries show different mean improvements by prior achievement, this imbalance can affect the overall contrast. In an actual experiment, random assignment makes such systematic group differences unlikely over repeated assignments, but it does not guarantee that every small sample will be balanced. This hypothetical result is a reason to interpret a small experiment’s overall comparison cautiously, not proof that the program is ineffective.
Recognizing Confounding in Practice
A practical diagnosis asks three linked questions. First, do the treatment groups differ in another variable? Second, is that variable related to the response? Third, could the groups’ difference in that variable contribute to the observed contrast? If the answer to all three is yes, the treatment comparison is potentially confounded. A variable that differs between groups but has no relationship to the response does not explain the outcome difference in this way.
Identify the treatment or explanatory variable, the response, and the groups whose outcomes are being compared.
Ask whether the groups differ in background characteristics, starting measurements, or circumstances relevant to the response.
A difference matters as a potential confounder when that factor is also related to the response.
Describe the observed comparison, but do not attribute it to the treatment alone when its influence is entangled with another factor.
Random assignment is important because it assigns treatment independently of participants’ characteristics in the assignment process. Across repeated random assignments, this tends to prevent systematic confounding by both measured and unmeasured characteristics. In any one experiment, especially a small one, chance imbalances can still occur. In an observational study, people or circumstances determine group membership rather than random assignment, so systematic group differences may be more likely. The design tells us what to examine; it does not make every group comparison automatically interpretable.
Common Mistake
- Wrong reasoning: “The treated group has a better outcome, so the treatment produced the improvement.” This treats the observed difference as if treatment were the only difference between groups.
- Correction: Check whether another factor differs between groups and is related to the response. If it does, the observed comparison may combine the treatment contrast with that factor’s influence. State what the data show, and avoid assigning the whole difference to treatment alone.
A related mistake is to assume that a variable is a confounder merely because it is measured or mentioned. The relevant issue is not the number of variables in a study. It is whether a factor is associated with differences in treatment groups and is also related to the response in a way that can complicate the comparison.
Check Your Understanding
For each question, explain your reasoning rather than giving only a yes-or-no answer.
- A health comparison finds different recovery rates for people who exercise regularly and those who do not. The groups also differ substantially in age, and recovery rates vary with age. What makes age a potential confounder?
- In the tutoring example, calculate the overall pass rate in each group and describe why those rates are difficult to interpret as a tutoring effect.
- Would a characteristic that differs between treatment groups necessarily be a confounder if it is unrelated to the response? Explain.
- What does random assignment do to the risk of systematic confounding, and why can a small randomized experiment still have an imbalanced sample?
- Write one cautious sentence describing what an observational comparison can show when a plausible confounder is present.