Observed Differences Are Not Automatically Causal Effects
Suppose patients who receive a treatment have better outcomes on average than patients who do not. That comparison is informative about the groups as observed, but it does not by itself tell us what the treatment caused. The treated and untreated groups may have differed before treatment: for example, people with a better prognosis may be more likely to seek treatment, or clinicians may preferentially treat people who are more ill. Either pattern can make the outcome means differ even if treatment has no effect.
The key distinction is between who received treatment and what treatment did. The first is visible in an observational comparison; the second is a contrast between potential outcomes. The introductory material on potential outcomes defines those outcomes and the causal effect for each unit. Here we focus on how selection into treatment can separate the observed group contrast from those causal effects.
We assume the treatment and outcomes are well-defined, and that one unit's treatment does not change another unit's outcome. These conditions allow us to use the potential-outcome notation and consistency stated above. The algebra below also assumes the relevant means exist. It does not assume treatment was randomly assigned or that treated and untreated units had the same baseline prognosis. In fact, the point is to see what can happen when they differ.
How Selection Enters the Observed Contrast
By consistency, the observed mean among treated units is the mean of their treated potential outcomes, \(E[Y(1)\mid A=1]\). The observed mean among untreated units is the mean of their untreated potential outcomes, \(E[Y(0)\mid A=0]\). Add and subtract \(E[Y(0)\mid A=1]\) to express the difference as two components:
The first component is the average treatment effect among the treated: their average difference between their two potential outcomes. The second is a selection component: the difference in the untreated potential outcome that the treated and untreated groups would have had. It is a comparison of baseline potential outcomes, not an effect of treatment.
This decomposition is an identity, not an assumption that either component can be separately calculated from observed outcomes. For treated units, \(Y(0)\) is unobserved; for untreated units, \(Y(1)\) is unobserved. The identity helps explain why the raw contrast may not equal the causal effect, but it does not solve the missing-counterfactual problem.
Worked Example: A Positive Difference With No Treatment Effect
Consider a hypothetical population with 100 treated and 100 untreated people. Each person is either at high or low baseline risk. Suppose every high-risk person would have an outcome of 10 under either treatment state, and every low-risk person would have an outcome of 2 under either state. Thus, for every person, \(Y(1)=Y(0)\): treatment has no causal effect. However, 80% of the treated group is high-risk, compared with 20% of the untreated group.
Treated mean. The treated group contains 80 high-risk and 20 low-risk people. Its mean outcome is
Untreated mean. This group contains 20 high-risk and 80 low-risk people. Its mean outcome is
The observed treated-control difference is \(8.40-3.60=4.80\) outcome units. Yet each person's treatment effect is \(Y(1)-Y(0)=0\), so the average effect among the treated is also 0. The full 4.80-unit difference comes from the selection component: before considering any treatment effect, the treated group has more people with the higher untreated outcome.
Baseline Differences Can Work in Either Direction
Selection does not necessarily make the treatment look beneficial. Its direction depends on how treatment selection relates to the outcome. If a higher outcome is desirable and healthier people are more likely to receive treatment, selection can make treatment appear beneficial even when it changes no outcomes. If clinicians reserve treatment for people with worse prognoses, the same treatment can appear harmful in a raw comparison despite having no effect.
Worked Example: A Treatment That Appears Harmful
Imagine a hypothetical setting in which the outcome is a health score, with higher scores indicating better health. For every person, the score would be 90 under either treatment state for a low-severity case and 60 under either state for a high-severity case. Treatment therefore has no effect for anyone. Clinicians give treatment to 80 high-severity and 20 low-severity patients. Among untreated patients, 20 are high-severity and 80 are low-severity.
The mean score among treated patients is
The mean score among untreated patients is
The observed difference, treated minus untreated, is \(66.0-84.0=-18.0\) health-score points. That difference does not show that treatment lowered anyone's score: the stipulated potential outcomes are equal for every person. Instead, treatment is more common among patients whose scores would have been lower even without treatment. The negative raw contrast reflects that baseline composition.
These examples use simple risk categories to make the mechanism visible. Baseline differences need not be categorical, and the selection component need not be as large as the treatment effect. A raw contrast combines what happened under the treatment received with differences between the people who received each option. It can exaggerate, hide, or reverse the direction of a causal effect.
Worked Example: No Effect Does Not Guarantee Equal Trial Means
To distinguish systematic selection from chance variation, consider a hypothetical randomized trial of six units. Their outcomes are 2, 4, 6, 8, 10, and 12 under either treatment state, so treatment has no effect for any unit. Three units are assigned at random to treatment and three to control. In one possible assignment, the treated units have outcomes 2, 4, and 6, while the controls have 8, 10, and 12.
The realized treated mean is \((2+4+6)/3=4\), and the control mean is \((8+10+12)/3=10\). The observed difference is \(4-10=-6\) outcome units, even though every individual effect is zero. This is a chance imbalance in a particular assignment, not selection of treatment based on prognosis.
Across all equally likely assignments of three of these six units to treatment, each group has expected mean \((2+4+6+8+10+12)/6=7\). Thus the expected treated-control difference under this random assignment is \(7-7=0\). Random assignment removes systematic allocation according to prognosis in expectation; it does not require the two realized groups to have identical outcomes in every trial.
How to Read an Observational Comparison
The raw difference remains a valid description of the observed groups. The error is to label it a treatment effect without a reason to believe the groups' potential outcomes would otherwise be comparable. When allocation depends on health status, access, preference, clinician judgment, or another prognostic feature, the treatment groups may begin with different expected outcomes. The resulting contrast can therefore contain a selection component.
A useful first diagnostic is to examine pretreatment characteristics that may predict the outcome and compare their distributions across treatment groups. Large differences can reveal plausible routes for selection and warn that the raw contrast is difficult to interpret causally. But balance on measured characteristics alone does not establish that selection is absent: other prognostic factors may not have been measured, and similar averages can hide different distributions.
There is also a limit to what outcome data alone can diagnose. Because each unit reveals only the outcome under the treatment actually received, the unobserved potential outcomes are unavailable for direct comparison. Consequently, seeing similar observed means does not prove that there is no confounding; selection can offset a treatment effect. Seeing different means does not prove that treatment caused the difference; baseline selection can produce it.
| Approach | What the contrast describes | What to keep in mind |
|---|---|---|
| Raw observational comparison | Mean outcomes among those observed to receive each treatment | May combine treatment effects with baseline selection |
| Randomized comparison | Mean outcomes in groups formed by random assignment | Randomization removes systematic selection in expectation, but chance differences remain in a particular trial |
| Comparison after accounting for measured baseline features | A contrast constructed to address observed differences | It cannot by itself rule out selection related to unmeasured features; the assumptions and method require separate justification |
The table distinguishes design from analysis. Randomization determines treatment assignment by design. In an observational study, an analyst may use measured baseline information to make groups more comparable, but that does not automatically remove all confounding. The later material on exchangeability and adjustment develops the conditions for interpreting adjusted contrasts; those conditions should not be assumed merely because an analysis includes covariates.
Common Mistake and Key Takeaway
- Wrong reasoning: “The treated group had a higher mean outcome, so treatment improved outcomes.” This treats a difference between groups as if it were the contrast between the same units under two treatment states.
- Correction: Ask whether the groups could differ in their outcomes even if neither received treatment. If treatment selection is related to baseline prognosis, their untreated potential-outcome means may differ. The observed contrast then includes that selection difference as well as any effect among the treated.
- Another mistake: “The groups have similar measured baseline averages, so there is no confounding.” Similarity on recorded features is useful diagnostic evidence, but it does not establish similarity in all outcome-relevant features.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In the first example, what is the observed treated-control difference, and what is its causal interpretation?
- Suppose the treated group's untreated potential-outcome mean is 12 and the untreated group's untreated potential-outcome mean is 9. If the average effect among treated units is 2, what is the observed treated-control difference?
- Can a randomized trial have unequal observed treatment and control means when treatment has no effect? Explain.
- Why does similar measured baseline balance not prove that an observational contrast is unconfounded?
Solutions. (1) The observed difference is \(8.40-3.60=4.80\) units. Since every person's treatment effect is zero, the difference is entirely due to the groups' different baseline composition. (2) The selection component is \(12-9=3\), so the decomposition gives \(2+3=5\) outcome units. (3) Yes. Random assignment can produce chance imbalances in a particular trial; randomization makes the expected contrast reflect the treatment effect, but does not force each realized contrast to equal it. (4) Measured balance says nothing definitive about unmeasured prognostic features, and averages can fail to capture all distributional differences. It is a diagnostic, not proof that the potential outcomes are comparable.