Why the Distinction Matters
When treatment assignment is not randomized, people receiving treatment may differ systematically from those who do not. A common reasoning error is to say that the outcome was recorded under the treatment each person actually received, so the observed comparison must reflect the treatment effect. The first part of that reasoning concerns consistency; the second concerns whether treatment assignment is confounded. They are different claims.
Consistency is necessary for connecting observed data to potential outcomes. But by itself it says nothing about how treatment was assigned, whether treated and untreated people were comparable, or what their unobserved outcomes would have been under the other treatment. The distinction matters because an observational data set can satisfy consistency perfectly and still fail to identify a causal effect.
Consistency Is a Link, Not an Assignment Rule
Let \(A\) be a binary treatment indicator, where \(A=1\) means treated and \(A=0\) means untreated. For each person, let \(Y(1)\) be the outcome that person would have under treatment and \(Y(0)\) the outcome that person would have without treatment. Only one of these two potential outcomes is observed for each person.
This definition links the observed outcome to one potential outcome. It does not say that treatment assignment is independent of either potential outcome. In particular, consistency does not imply exchangeability, the condition that treatment assignment is unrelated to the potential outcomes (unconditionally, or within specified pretreatment covariate groups). If assignment depends on a characteristic related to prognosis, the treatment groups may differ in potential outcomes even when every observed outcome matches the treatment received.
For example, suppose clinicians are more likely to give a treatment to patients with more severe symptoms, and severity also predicts the outcome. Each treated patient's recorded outcome can still equal that patient's \(Y(1)\), and each untreated patient's outcome can still equal that person's \(Y(0)\). Yet a comparison of the two groups mixes treatment effects with pre-existing differences in severity.
What an Observed Mean Difference Contains
The separation can be written directly. Assume consistency and that both treatment groups occur, so their means are defined. The observed difference in group means is
The equality follows because consistency gives \(Y=Y(1)\) among those with \(A=1\), and \(Y=Y(0)\) among those with \(A=0\). Add and subtract \(E[Y(0)\mid A=1]\) to separate the pieces:
The first term is the average treatment effect among the treated: their average difference between \(Y(1)\) and \(Y(0)\). The second term is a selection difference: how the treated and untreated groups would have differed, on average, if both had been untreated. Consistency justifies the initial substitution, but does not make the selection difference zero. With confounded assignment, that term can be positive or negative.
For comparison, the population average treatment effect is \(E[Y(1)-Y(0)]\). Even if the selection difference were zero, the observed contrast in this decomposition is the effect among the treated, not automatically the population average effect. An identification argument needs assumptions connecting the assignment groups and the target population, as well as consistency and the required treatment overlap. The unconditional exchangeability condition \( (Y(0),Y(1))\perp A \), together with positive probability of each treatment, is one sufficient route to identifying the population average effect from the unadjusted observed means. Consistency alone is not that route.
Worked Example: Consistency With Confounded Assignment
Consider a hypothetical population of 100 patients. Let \(U=1\) indicate high initial severity and \(U=0\) low initial severity. There are 50 patients in each severity group. Their potential outcomes, measured in symptom-score points, are:
| Severity | \(Y(0)\) | \(Y(1)\) | Treatment effect \(Y(1)-Y(0)\) |
|---|---|---|---|
| Low (\(U=0\)) | 10 | 12 | 2 points |
| High (\(U=1\)) | 15 | 17 | 2 points |
Suppose 40 of the 50 high-severity patients and 10 of the 50 low-severity patients receive treatment. The other patients do not. The observed outcomes are consistent: a treated patient's outcome is the listed \(Y(1)\), and an untreated patient's outcome is the listed \(Y(0)\).
Among the 50 treated patients, 40 have outcome 17 and 10 have outcome 12. Their observed mean is
Among the 50 untreated patients, 10 have outcome 15 and 40 have outcome 10. Their observed mean is
The observed difference is \(16-11=5\) points. But every patient's treatment effect is \(2\) points, so the population average treatment effect is \(2\) points. The difference is due to assignment as well as treatment: high-severity patients, whose outcomes are higher under either treatment, are much more common in the treated group.
The decomposition confirms the arithmetic. The treated patients' average treatment effect is 2 points. Their average untreated potential outcome is \(\{40(15)+10(10)\}/50=14\) points, while the untreated group's average untreated potential outcome is 11 points. Thus the selection difference is \(14-11=3\) points, and the observed contrast is \(2+3=5\) points. Consistency holds throughout; it does not remove the 3-point selection difference.
The Same Observed Data Can Fit Different Effects
A further way to see the problem is to ask what the observed data leave unknown. For a treated person, consistency reveals \(Y(1)\) but not \(Y(0)\); for an untreated person, it reveals \(Y(0)\) but not \(Y(1)\). Those missing potential outcomes are exactly what individual treatment effects require.
Worked Example: Two Causal Stories With Identical Observations
Suppose a hypothetical data set contains four people. Two received treatment and had observed outcomes 12 and 14; two did not and had observed outcomes 8 and 10. The observed mean difference is
One possible completion of the missing potential outcomes assigns effects of 2 units to all four people: the treated people's \(Y(0)\) values are 10 and 12, and the untreated people's \(Y(1)\) values are 10 and 12. For each person, the observed outcome still matches the potential outcome under the treatment received. Under this completion, the average treatment effect is
A different completion keeps every observed value the same but assigns \(Y(0)=0,2\) to the two treated people and \(Y(1)=20,22\) to the two untreated people. The treatment effect is then 12 units for each person, so the average treatment effect is 12 units. Consistency still holds: none of the observed outcomes changed, and each remains the potential outcome matching the treatment received.
The two completions therefore agree on the observed data and on consistency but disagree about the average causal effect: one gives 2 units and the other 12 units. In the second completion, assignment is strongly related to the potential outcomes: the treated people have lower \(Y(0)\) values than the untreated people. The observed data cannot distinguish these causal stories without additional assumptions or design information. This is a nonidentification argument, not a claim that either completion describes a real study.
Random Assignment Does Not Mean Exact Balance
Consistency is also present in randomized trials. What random assignment adds is a design-based reason to expect treatment assignment to be unrelated to potential outcomes, rather than a guarantee that treatment and control groups will be identical in every realized sample. The next example keeps these ideas separate.
Worked Example: A Chance Imbalance in a Randomized Trial
Imagine a hypothetical finite population of 100 people: 50 have a low baseline value and 50 a high baseline value. Each person's untreated outcome is 10 points for low baseline and 20 points for high baseline. Treatment adds exactly 4 points for everyone. Thus the population average treatment effect is \(4\) points.
Suppose 50 people are assigned at random to treatment. By chance, the treated group contains 20 low-baseline and 30 high-baseline people; the control group contains 30 low-baseline and 20 high-baseline people. Consistency means the treated outcomes are 14 and 24 points, while the control outcomes are 10 and 20 points. The observed means are
The realized difference is \(20-14=6\) points, rather than the population treatment effect of 4 points. Here, the groups differ in baseline composition by chance. That chance imbalance does not mean randomization failed or that the assignment mechanism systematically selected people based on their potential outcomes. Across assignments generated by the randomization procedure, treatment assignment is independent of the fixed potential outcomes; in this particular assignment, the groups are not exactly balanced. This differs from the earlier observational example, where the assignment rule deliberately favored one severity group.
How to Diagnose the Mistake
Consistency is a statement about the relationship between treatment received and outcome observed. It is not a diagnostic test for confounding. In practice, the observed data alone usually cannot verify consistency's counterfactual claim, because each person reveals only one potential outcome. Researchers can, however, inspect whether the treatment is clearly defined and whether the recorded treatment and outcome correspond to the intended intervention and follow-up.
Warning signs that the causal comparison needs more than consistency include treatment selection based on prognosis, substantial pretreatment differences between groups, or a study design that does not justify exchangeability. These signs do not quantify confounding by themselves, and absence of visible differences does not prove exchangeability. A randomized design can support exchangeability through the assignment mechanism; in an observational study, covariate adjustment requires a defensible exchangeability assumption given measured pretreatment covariates. Neither approach can be replaced by simply noting that outcomes were measured after treatment.
The treatment definition also matters. If “treated” combines materially different versions of an intervention, the statement \(Y=Y(A)\) may be ambiguous: which treatment version's potential outcome does \(Y(1)\) denote? Clarifying versions addresses whether consistency is meaningful. It does not, by itself, resolve confounding in who receives treatment.
Common Mistake and Correct Reasoning
- Wrong reasoning: “Each treated person's observed outcome is their treated potential outcome, so the treated-versus-untreated mean difference is causal.”
- Correction: The equality \(Y=Y(A)\) only identifies the potential outcome corresponding to the treatment received. The observed contrast can also include a selection difference between groups. A separate assumption or design argument is needed to address that difference.
- Wrong reasoning: “If a data set satisfies consistency, it cannot be confounded.”
- Correction: Consistency and confounding concern different features. Consistency concerns outcome measurement under the received treatment; confounding concerns the relation between treatment assignment and potential outcomes.
- Wrong reasoning: “A randomized trial must have equal observed group means before treatment.”
- Correction: Random assignment balances potential outcomes in the assignment mechanism, not necessarily every baseline variable in a particular realized sample. Chance imbalances can occur.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In one sentence, state what consistency means for a person with \(A=0\).
- A treated group has a higher observed mean than an untreated group. Why does consistency alone not show that treatment caused the difference?
- In the first worked example, how much of the 5-point observed difference is the selection difference?
- In the four-person example, why do two different average treatment effects remain compatible with the same observed data?
Solutions. (1) For a person with \(A=0\), consistency says \(Y=Y(0)\). (2) Consistency links each observed outcome to the potential outcome under the treatment received, but assignment may be related to potential outcomes; the observed difference can include a selection difference. (3) It is 3 points: the treated group's average \(Y(0)\) is 14 points and the untreated group's average \(Y(0)\) is 11 points, so \(14-11=3\); adding the treated group's 2-point effect gives the observed 5-point contrast. (4) Each person reveals only the potential outcome matching the received treatment. The unobserved potential outcomes can be completed in different ways, preserving all observed values and consistency while changing the average treatment effect.