Diagnosing Confounding in Study Scenarios

Why Diagnose Confounding?

When two groups have different outcomes, the difference may reflect the treatment—or it may partly reflect who received it. Students who choose tutoring, patients who receive a medication, and employees who work remotely can differ from comparison groups in ways that also affect the response. Diagnosing confounding means identifying a plausible alternative explanation for some or all of the observed difference.

The task is not to prove from a short description that confounding definitely occurred. Instead, identify a candidate variable, explain how it could be related to both treatment and response, and use the direction of those relationships to judge whether it might make the observed treatment association look larger, smaller, or even opposite to the treatment’s effect.

Definition: A confounder is a variable that is a common cause of, or is otherwise associated with, both the treatment or exposure and the response, and is not caused by the treatment. It can make the treatment groups differ in their expected responses for reasons other than the treatment itself. A confounded comparison mixes the treatment contrast with these pre-existing group differences.

In a scenario, call the treatment or exposure \(A\), the response \(Y\), and a candidate confounder \(C\). A useful first check is whether \(C\) plausibly occurs before treatment and helps explain who receives which treatment. Then ask whether \(C\) also predicts the response. The confounder need not be the only cause of either variable, and a scenario usually supports a plausible diagnosis, not certainty.

A variable that changes because of the treatment is different. For example, if a program changes study time and study time then affects exam scores, study time may be on the pathway from program to score. It is not a pre-existing confounder of that treatment effect. Adjusting for a variable on the pathway can remove part of the effect one is trying to describe.

A Direction Check

The direction of potential distortion follows from two questions: which kind of person is more likely to receive the treatment, and which kind of person tends to have a higher response? The answer depends on how the response is coded. “Higher” may mean better, such as a higher recovery rate, or worse, such as a higher disease risk.

For a simple sign check, suppose \(A\) is a binary treatment, \(C\) is a binary pre-treatment characteristic, and the response has an additive conditional mean with no interaction:

E(Y\mid A,C)=\beta_0+\beta_1A+\beta_2C.

Here \(\beta_1\) is the difference in mean response between treatment levels at the same value of \(C\), while \(\beta_2\) is the response difference associated with \(C\) at the same treatment level. Write \(q_a=P(C=1\mid A=a)\), the proportion with \(C=1\) among those with treatment status \(a\). Averaging the conditional mean over each treatment group gives:

\begin{aligned} E(Y\mid A=1)-E(Y\mid A=0) &=\beta_1+\beta_2(q_1-q_0). \end{aligned}

The first term is the common within-\(C\) treatment contrast in this model. The second is the composition difference: how much the groups differ in \(C\), multiplied by how \(C\) relates to the response. If \(\beta_2(q_1-q_0)\) has the same sign as \(\beta_1\), the unadjusted difference is pushed farther in the treatment-effect direction. If it has the opposite sign, the difference is pulled toward zero and might even change sign.

This is a sign-checking model, not a guarantee that a real study follows a linear relationship. It assumes a common treatment contrast across levels of \(C\), and it does not establish that \(\beta_1\) is causal. A causal interpretation also requires an appropriate study and adequate control of other relevant differences. If the scenario does not say how \(C\) relates to treatment or response, its direction may be impossible to determine.

Name the contrast.
State the treatment or exposure, the comparison group, and the response. Clarify whether a higher response is favorable or unfavorable.

Propose a pre-treatment candidate.
Choose a characteristic that could differ between the groups before treatment and could influence who receives or chooses the treatment.

Trace both links.
Explain how the candidate relates to treatment receipt and how it relates to the response, holding the treatment comparison in view.

Combine the directions.
Ask whether the treatment group has more or less of the candidate, and whether that candidate tends to raise or lower the response. State whether this could exaggerate, mask, or reverse the observed contrast.

Worked Scenarios

Worked Example: Tutoring and Exam Scores

Suppose a hypothetical campus report compares exam scores for students who chose optional tutoring with scores for students who did not. A plausible candidate confounder is academic preparation before tutoring began. Students who feel less prepared may be more likely to seek tutoring, while stronger preparation tends to predict higher exam scores. The report has not randomly assigned students to tutoring.

To make the direction concrete, imagine that among students with lower preparation, the mean score is 60 without tutoring and 68 with tutoring. Among students with higher preparation, the corresponding means are 80 and 88. In both preparation groups the tutoring-associated difference is \(+8\) points. Suppose 20% of the tutoring group, but 70% of the no-tutoring group, had higher preparation.

The hypothetical group means are:

\begin{aligned} \bar{Y}_{\text{tutoring}}&=0.20(88)+0.80(68)=72,\\ \bar{Y}_{\text{no tutoring}}&=0.70(80)+0.30(60)=74. \end{aligned}

The unadjusted difference is \(72-74=-2\) points, even though the within-preparation differences are both \(+8\) points. Lower preparation is more common in the tutoring group and predicts a lower score, so it pulls the crude comparison downward. In this hypothetical example, that imbalance more than masks the positive within-group difference and reverses its sign. This diagnosis is conditional on the stated numbers; it does not show that tutoring caused an eight-point increase in a real setting.

Worked Example: Medication and Recovery

Imagine a hypothetical observational comparison of patients who received a medication with patients who did not, using recovery as the response. A plausible confounder is illness severity before treatment. More severely ill patients may be more likely to receive the medication, and severe illness may make recovery less likely regardless of treatment. This pattern is sometimes called confounding by indication: the reason for treatment also predicts the outcome.

Suppose the recovery rates are 80% with medication and 70% without it among mildly ill patients, and 50% with medication and 40% without it among severely ill patients. Thus, within each severity group, the medication-associated difference is \(+10\) percentage points. Suppose 80% of the treated patients are severely ill, compared with 20% of the untreated patients.

The overall recovery rates are:

\begin{aligned} \text{Treated rate}&=0.20(0.80)+0.80(0.50)=0.56,\\ \text{Untreated rate}&=0.80(0.70)+0.20(0.40)=0.64. \end{aligned}

The crude difference is \(0.56-0.64=-0.08\), or \(-8\) percentage points. Severity is more common among treated patients and predicts lower recovery, so the imbalance makes the medication group look worse in this comparison. The crude result does not establish that the medication harms recovery: the hypothetical within-severity differences point in the other direction. Nor do these numbers alone establish a beneficial causal effect; other group differences could remain.

Worked Example: Remote Work and Productivity

Suppose a hypothetical workplace report compares productivity scores for employees working remotely with scores for employees working on site. A possible confounder is job role or autonomy. Roles that allow remote work may also involve more independent tasks, and those roles may have different productivity scores even if work location has no effect.

For a simplified calculation, suppose employees in high-autonomy roles average 82 points remotely and 80 on site, while employees in lower-autonomy roles average 62 remotely and 60 on site. Assume 80% of the remote group, but only 20% of the on-site group, have high-autonomy roles.

The two overall means are:

\begin{aligned} \bar{Y}_{\text{remote}}&=0.80(82)+0.20(62)=78,\\ \bar{Y}_{\text{on site}}&=0.20(80)+0.80(60)=64. \end{aligned}

The crude difference is \(78-64=14\) points, while the within-role differences are both \(+2\) points. High autonomy is more common among remote employees and is associated with higher scores, so this imbalance could make the apparent remote-work advantage substantially larger than the within-role contrast. In an actual workplace, role may not be measured well, the relationship may differ across jobs, and other explanations may remain.

When the Direction Is Unclear

A careful diagnosis separates what the scenario supports from what remains unknown. It may establish that a candidate differs between groups but say nothing about how the candidate relates to the response. Or it may say the candidate predicts the response without revealing which treatment group has more of it. In either case, naming the candidate as a possible confounder may be reasonable, but claiming that it exaggerates or masks the association is not justified yet.

Also check whether the candidate is measured before treatment. A variable measured after treatment might be a consequence of the treatment, a mediator, or a response-related variable rather than a confounder. The timing and the causal story matter; a variable’s statistical association with treatment and response alone does not tell the whole story.

Common Mistake

Wrong reasoning: “The tutoring group had lower scores, so tutoring must have lowered scores,” or, conversely, “preparation is related to scores, so it must explain the whole difference.” Neither conclusion follows. The first ignores who selected tutoring; the second assumes an effect size and direction not given by the scenario.

Correction: State the candidate confounder and both links explicitly. For example: “Lower prior preparation may make students more likely to seek tutoring, and it tends to predict lower exam scores. Because lower preparation is therefore more common in the tutoring group, it could pull the crude score comparison downward.” This identifies a plausible direction without claiming that the confounder is the only explanation or that its contribution is known exactly.

Key takeaway: A direction-of-confounding argument needs all three pieces: the candidate’s relationship to treatment, its relationship to response, and which group has more of the candidate. If any piece is missing, say that the direction cannot be determined from the information given.

Check Your Understanding

For each scenario, identify a plausible candidate confounder and explain what additional information is needed to judge the direction of distortion.

A campus compares attendance at optional review sessions with final exam scores. Students’ motivation before the course may affect both session attendance and scores. Describe the two links and what group imbalance would be needed to predict a direction.
A health service compares recovery among patients who received an early intervention with recovery among those who did not. Explain why initial severity could confound the comparison.
A workplace compares sick days among employees with flexible schedules and employees with fixed schedules. Suggest a pre-existing candidate confounder and state how its relationships would determine the direction of distortion.
Why is study time measured after a tutoring program began not automatically a confounder of the program’s effect on exam scores?
A proposed confounder differs between treatment groups, but the scenario does not say whether it raises or lowers the response. What can you conclude about the direction of potential distortion?