What Measured Adjustment Cannot Fix: Unmeasured Confounding

Why Adjustment Has a Limit

Adjustment compares treated and untreated units after accounting for recorded pretreatment covariates. Its causal interpretation depends on an important condition: within each covariate pattern, treatment assignment must be unrelated to the potential outcomes. If a pretreatment cause of both treatment and outcome is not recorded, that condition can fail even when the recorded covariates look perfectly balanced.

This is the central distinction: balance describes the distributions of variables we measured; exchangeability describes the relationship between treatment and potential outcomes. Balance on measured covariates can be useful evidence about the observed data, but it cannot by itself establish that all relevant common causes have been measured.

Definition: Let \(A\) be a binary treatment, with \(A=1\) for treated and \(A=0\) for untreated. Let \(Y(1)\) and \(Y(0)\) be the potential outcomes under treatment and no treatment, and let \(Y=Y(A)\) be the observed outcome under consistency. Let \(X\) denote measured pretreatment covariates and \(U\) an unmeasured pretreatment variable. Conditional exchangeability given \(X\) means \((Y(1),Y(0))\perp A\mid X\). An unmeasured confounder is a variable such as \(U\) that helps determine both treatment and outcome and leaves this condition false after conditioning on \(X\).

A variable need not be observed to be important. For example, severity of illness may affect both whether a patient receives a treatment and the outcome that would occur under either treatment choice. If severity is not recorded, comparing patients with the same values of the recorded covariates may still compare patients with different underlying severity. The resulting outcome difference can combine a treatment effect with a difference in prognosis.

What Adjustment Identifies—and What It Requires

The average treatment effect is \(\tau=E[Y(1)-Y(0)]\). A standard adjustment argument identifies it from observed outcomes when three conditions hold: consistency, conditional exchangeability given the measured \(X\), and positivity. Consistency connects the observed outcome to the potential outcome for the treatment actually received. Exchangeability says that, within values of \(X\), treated and untreated units are comparable in their potential outcomes. Positivity requires that both treatment options occur with positive probability at each relevant value of \(X\).

When these conditions hold, the average potential outcome under each treatment can be expressed using observed conditional means:

E[Y(a)] = E_X\{E[Y\mid A=a,X]\}, \qquad a\in\{0,1\}.

The difference between these two standardized means identifies \(\tau\). The key step is exchangeability: it permits replacing a conditional mean of the potential outcome \(Y(a)\) with the mean among units observed to receive \(a\). If an unmeasured common cause remains associated with treatment within levels of \(X\), that replacement is not justified.

Key distinction: Covariate balance means that the treated and untreated groups have similar distributions of the recorded \(X\). Conditional exchangeability requires that, within \(X\), treatment assignment be unrelated to the potential outcomes. The first does not imply the second.

Worked Examples

Worked Example: Perfect Measured Balance, Residual Confounding

Consider a hypothetical population with a binary measured covariate \(X\) and a binary unmeasured risk factor \(U\). Suppose the outcome, measured in points, follows the simple model \(Y(a)=10+4X+8U+2a\). Thus, holding \(X\) and \(U\) fixed, treatment increases the outcome by 2 points for every unit. Within each value of \(X\), imagine 100 treated and 100 untreated units. Among treated units, 75 have \(U=1\); among untreated units, 25 have \(U=1\). The proportions with \(X=1\) are equal between the treatment groups.

The recorded covariate is balanced by construction: both groups have the same distribution of \(X\). But within either \(X\) stratum, the average outcome among treated units is

$$ 10+4X+8(0.75)+2 = 18+4X. $$

The corresponding untreated mean is

$$ 10+4X+8(0.25) = 12+4X. $$

The within-\(X\) observed contrast is therefore \((18+4X)-(12+4X)=6\) points. Standardizing across \(X\) does not change that contrast: the adjusted comparison is 6 points. Yet the model’s causal effect is \(Y(1)-Y(0)=2\) points for every unit. The adjusted contrast exceeds the true effect by 4 points because the treated group has a 0.50 higher prevalence of \(U=1\), and \(U\) raises the outcome by 8 points: \(8(0.75-0.25)=4\). Balance on \(X\) has not made the groups exchangeable.

Worked Example: An Omitted Predictor That Does Not Bias the Contrast

Omitting a variable does not automatically create confounding. In a second hypothetical setting, suppose \(Y(a)=20+5X+6U+3a\), where \(U\) predicts the outcome but is not associated with treatment within levels of \(X\). Specifically, among both treated and untreated units at each \(X\), 40% have \(U=1\). The outcome is measured in points.

For any fixed value of \(X\), the treated mean is \(20+5X+6(0.40)+3=25.4+5X\). The untreated mean is \(20+5X+6(0.40)=22.4+5X\). Their difference is \(3\) points. The model also gives \(Y(1)-Y(0)=3\) points for every unit, so the adjusted comparison recovers the causal effect in this example.

Here, \(U\) is prognostic because it changes the outcome, but it does not produce residual confounding: its prevalence is the same in treated and untreated units within \(X\). The relevant issue is not simply whether a variable predicts the outcome. It is whether an omitted variable leaves treated and untreated units with different potential outcomes after conditioning on the variables used for adjustment.

A Simple Bias Decomposition

A short derivation makes the mechanism explicit. Suppose, for illustration, that the outcome follows the additive model \(Y(a)=m(X)+\tau a+\gamma U\), with a constant treatment effect \(\tau\), and that \(U\) is binary. Within a fixed value of \(X=x\), the observed treated-minus-untreated mean difference is

\begin{aligned} &E[Y\mid A=1,X=x]-E[Y\mid A=0,X=x]\\ &=\tau+\gamma\{P(U=1\mid A=1,X=x)-P(U=1\mid A=0,X=x)\}. \end{aligned}

To see why, take the conditional expectation of \(m(x)+\tau A+\gamma U\) in each treatment group. The \(m(x)\) terms cancel, the treatment terms differ by \(\tau\), and the remaining difference is \(\gamma\) multiplied by the difference in the prevalence of \(U=1\). Thus the bias in this particular model is the product of the outcome association \(\gamma\) and the residual prevalence difference. If either factor is zero, this source of bias is zero. This formula is a model-specific explanation, not a general formula for all forms of confounding.

Worked Example: Quantifying a Hypothetical Residual Bias

Suppose a hypothetical adjusted contrast is 5 points. A subject-matter analysis suggests that an unmeasured binary risk factor may raise the outcome by 4 points, and that its prevalence may be 0.20 higher in treated than untreated units within recorded covariate strata. Under the additive model above, the bias is

4(0.20)=0.80\text{ points}.

Since the treated group is expected to have more of a factor that raises the outcome, the bias is positive. The corresponding contrast after subtracting this hypothetical bias is \(5-0.80=4.20\) points. This calculation does not prove that the causal effect is 4.20 points: the prevalence difference and outcome association are assumptions used for a sensitivity calculation, not quantities learned from the adjusted contrast alone.

Diagnostics, Alternatives, and Limitations

Measured balance can be assessed by comparing the distributions of recorded pretreatment covariates across treatment groups, including within strata when adjustment is conditional. A remaining difference in \(X\) signals that the observed adjustment has not achieved its measured-balance goal. But even exact balance on every recorded covariate does not reveal whether an unrecorded \(U\) differs between groups. There is no general diagnostic using only the observed treatment, outcome, and \(X\) that can verify conditional exchangeability.

A credible analysis therefore needs more than a balance table. Investigators should use subject-matter knowledge to identify plausible common causes, measure important pretreatment variables where possible, and state why the remaining variables are expected not to undermine exchangeability. Sensitivity analysis can show how conclusions would change under specified levels of unmeasured association, as in the hypothetical calculation above; it cannot establish which level is true without additional information.

The main alternative is a design that makes treatment assignment independent of baseline potential outcomes, such as random assignment. In a randomized trial, assignment balances baseline causes on average by design, including causes that were not measured, although chance imbalance can occur in a particular sample. In observational data, adjustment can support a causal comparison only if the measured covariates are sufficient for exchangeability. No choice of adjustment formula can recover information about an omitted cause simply by balancing the variables that were recorded.

The limitation is substantive as well as statistical: whether an important cause was omitted cannot usually be settled from the observed data alone. More flexible outcome models, matching, or more detailed balance on recorded covariates may address modeling choices or measured imbalance, but they do not automatically remove bias from an unmeasured common cause. The causal conclusion remains conditional on the adequacy of the adjustment set and the assumptions supporting it.

Common Mistakes

“The measured covariates are balanced, so the treatment groups are exchangeable.” This confuses an observed-data comparison with a condition about potential outcomes. Balance says nothing definitive about variables that were not measured.
“Every omitted predictor of the outcome causes confounding.” Not necessarily. An omitted predictor can be prognostic without biasing the comparison if it is not associated with treatment within the adjustment strata.
“Adjustment removes all confounding.” It can remove confounding represented by adequately measured and appropriately adjusted covariates. Residual confounding can remain if a common cause is omitted, measured poorly, or handled in a way that does not achieve exchangeability.
“A sensitivity calculation verifies the correction.” It does not. Such a calculation shows what follows from chosen assumptions about the unmeasured factor; it cannot confirm those assumptions from the adjusted estimate alone.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In one covariate stratum, an unmeasured binary factor occurs in 60% of treated units and 30% of untreated units. If it raises the outcome by 5 units in an additive model, what is the residual bias contribution?
Can a variable be a strong predictor of the outcome without creating confounding? State the condition that would allow this.
What does exact balance on all recorded covariates establish, and what does it not establish?
Why does the standardization argument fail if conditional exchangeability given \(X\) is false?

Solutions. (1) The contribution is \(5(0.60-0.30)=1.5\) outcome units, positive because the risk factor is more prevalent in the treated group. (2) Yes. If the variable is not associated with treatment within levels of \(X\), it need not create confounding, even if it predicts the outcome. (3) It establishes equality of the observed distributions of the recorded covariates; it does not establish that potential outcomes are independent of treatment within those covariates. (4) Without exchangeability, the potential outcome mean among units observed to receive \(a\) need not equal the potential outcome mean for all units at the same \(X\), so the observed conditional mean cannot replace it in the identification formula.