Common Mistake: Adjusting For Every Measured Covariate

Why “adjust for everything” can change the question

In a longitudinal study, a covariate measured after an early treatment may be relevant in two different ways. It may carry part of that treatment’s effect forward, and it may help determine who receives a later treatment. Adjusting for it can therefore help separate the later treatment from confounding while also removing part of the earlier treatment’s effect. The problem is not that adjustment is always wrong; it is that adjustment can target a different contrast from the one of interest.

The key distinction is between a covariate’s measurement time and its causal role. A variable observed before a later treatment decision is not necessarily a baseline covariate for the whole study. If an earlier treatment has already changed it, conditioning on it can block a pathway from that earlier treatment to the outcome.

Definition: In this tutorial, overadjustment for an earlier treatment’s total effect means conditioning on a post-treatment variable that lies on a causal pathway from that earlier treatment to the outcome, thereby removing some or all of that pathway from the contrast being estimated. The same variable may still be needed to control confounding of a later treatment’s effect.

A total effect allows downstream variables to take the values that follow from the treatment being compared. A controlled direct effect instead compares treatments while fixing specified downstream variables. Neither target is automatically preferable; the scientific question determines which is relevant.

A longitudinal structural example

Consider two treatment decisions. \(A_1\) is an early treatment, \(L_1\) is a covariate measured afterward, \(A_2\) is a later treatment, and \(Y\) is the outcome. For a transparent calculation, take the treatments and covariate to be continuous. Suppose the data-generating equations are

\begin{aligned} L_1 &= 2A_1+U_L,\\ A_2 &= L_1+U_2,\\ Y &= 3A_1+4L_1+5A_2+\varepsilon. \end{aligned}

Assume \(A_1\) is nondegenerate, all variables have finite second moments, and \(U_L,U_2,\varepsilon\) are mutually independent, mean-zero errors that are also independent of \(A_1\). Assume \(\operatorname{Var}(U_L)>0\) and \(\operatorname{Var}(U_2)>0\). These conditions make the regressors noncollinear and, importantly, imply that the conditional mean of the outcome given \(A_1,L_1,A_2\) is the stated linear expression. Merely assuming that an error has mean zero, or is uncorrelated with \(A_1\), would not by itself establish that conditional mean.

The arrows represented by these equations include \(A_1\to L_1\), \(L_1\to A_2\), and \(L_1\to Y\). Thus \(L_1\) is affected by the earlier treatment and predicts the later treatment as well as the outcome. It is a confounder for the later treatment–outcome relationship, but it is also part of the pathway by which \(A_1\) can affect \(Y\).

Worked Example: The Total Effect Versus the Controlled Direct Effect

Compare setting \(A_1\) to \(a\) rather than \(a'\) while allowing \(L_1\) and \(A_2\) to evolve according to their equations, with comparing the same early treatments while holding \(L_1\) and \(A_2\) fixed.

Under the first comparison, substituting the equations for \(L_1\) and \(A_2\) into the outcome equation gives

\begin{aligned} Y &=3A_1+4(2A_1+U_L)+5(2A_1+U_L+U_2)+\varepsilon\\ &=21A_1+9U_L+5U_2+\varepsilon. \end{aligned}

The errors have mean zero, so the mean outcome when the early treatment is set to \(a\), with later variables allowed to follow their equations, is \(21a\). The total-effect contrast for \(a\) versus \(a'\) is therefore \(21(a-a')\).

If instead \(L_1\) and \(A_2\) are both fixed at the same values in the two comparisons, the outcome equation changes by \(3(a-a')\). That is the controlled direct contrast in this model. The contrast is \(3(a-a')\), not \(21(a-a')\), because the pathways through \(L_1\) and \(A_2\) have been held fixed.

For a one-unit increase in \(A_1\), the total contrast is 21 and the controlled direct contrast is 3. The difference, 18, is the contribution of the pathways through the downstream variables in this specified model.

What the regression adjustments calculate

The structural equations also let us calculate the population regression coefficients exactly. Here a regression includes an intercept, and its coefficients are population least-squares coefficients. The distinction between a conditional mean and a regression projection matters: a regression coefficient is not automatically a conditional causal effect.

Result: If \(Y=\alpha+\beta^\mathsf{T}X+\varepsilon\), \(E[\varepsilon\mid X]=0\), the second moments are finite, and \(E[(X-E[X])(X-E[X])^\mathsf{T}]\) is nonsingular, then the population linear regression of \(Y\) on an intercept and \(X\) has slope vector \(\beta\) and intercept \(\alpha\).

To verify the result, the population least-squares slope is the solution to the normal equations

\operatorname{Cov}(X,Y) = \operatorname{Var}(X)\,\beta_{\mathrm{OLS}}.

Using the stated conditional-mean assumption, \(E[\varepsilon\mid X]=0\) implies \(\operatorname{Cov}(X,\varepsilon)=0\). Hence \(\operatorname{Cov}(X,Y)=\operatorname{Var}(X)\beta\). Nonsingularity gives \(\beta_{\mathrm{OLS}}=\beta\); the intercept follows from \(E[Y]=\alpha+\beta^\mathsf{T}E[X]\). This is a regression result under explicit assumptions, not a general rule that adjustment recovers causal effects.

Worked Example: Adjusting for Both \(L_1\) and \(A_2\)

Fit the population regression of \(Y\) on \(A_1,L_1,A_2\). The structural equation and independence assumptions give

E[Y\mid A_1,L_1,A_2]=3A_1+4L_1+5A_2.

Because \(\operatorname{Var}(U_L)>0\), \(L_1\) is not determined by \(A_1\). Because \(\operatorname{Var}(U_2)>0\), \(A_2\) is not determined by \(L_1\). With nondegenerate \(A_1\), the three regressors are linearly independent in the population. The regression therefore has coefficients 3 on \(A_1\), 4 on \(L_1\), and 5 on \(A_2\).

The coefficient 3 is the outcome contrast for changing \(A_1\) while holding \(L_1\) and \(A_2\) fixed. It is not the total contrast of 21 calculated above. This coefficient interpretation relies on the conditional-mean equation; a mean-zero error alone would not justify it.

Worked Example: Adjusting for \(L_1\) but Omitting \(A_2\)

Now regress \(Y\) on \(A_1\) and \(L_1\), leaving \(A_2\) out. Since \(A_2=L_1+U_2\), rewrite the outcome as

Y=3A_1+9L_1+(5U_2+\varepsilon).

The residual term \(5U_2+\varepsilon\) is independent of \(A_1,L_1\) and has mean zero. Thus the population regression coefficient on \(A_1\) is 3, and the coefficient on \(L_1\) is 9. In this particular model, omitting \(A_2\) does not change the \(A_1\) coefficient after \(L_1\) is included, because the remaining part of \(A_2\) is \(U_2\), independent of both regressors.

This calculation is model-specific. In other data-generating systems, omitting \(A_2\) can change the coefficient. Its purpose is to show that even a regression that does not include every measured covariate can remove the earlier treatment’s pathway if it conditions on \(L_1\).

The total contrast can also be read from the regression of \(Y\) on \(A_1\) alone. Substitution gives \(Y=21A_1+9U_L+5U_2+\varepsilon\), and the combined error is independent of \(A_1\) with mean zero. The population slope is therefore 21. In this example, the unadjusted regression recovers the total contrast because \(A_1\) is independent of the errors; that convenience should not be generalized to observational studies with baseline confounding.

How to decide what to adjust for

Start with the target contrast, then place each candidate covariate in the treatment timeline. A variable’s being measured is not, on its own, a reason to include it in an outcome regression. Ask whether it precedes the treatment whose effect is being estimated, whether earlier treatment could have changed it, and whether it predicts both a later treatment and the outcome.

Name the treatment contrast.
Specify whether the aim is the total effect of an early treatment, a direct effect holding later variables fixed, or the effect of a later treatment given its prior history.

Order treatments and covariates in time.
In the example, \(A_1\) precedes \(L_1\), and \(L_1\) precedes \(A_2\). This ordering makes it possible for \(L_1\) to be both treatment-affected and relevant to later treatment assignment.

Check the role of each adjustment variable.
If a variable lies on a pathway for the target total effect, conditioning on it changes that target. If it confounds a later treatment, failing to address it may compromise the later-treatment comparison.

Match the analysis to the target.
Do not select an adjustment set simply by counting measured covariates. In feedback settings, ordinary outcome regression and a marginal structural model can target different quantities; the latter is designed for marginal regime contrasts when treatment-affected covariates also confound later treatment.

Diagnostics, alternatives, and limitations

A useful diagnostic is a timeline or causal diagram that marks when each variable is measured and which treatment could affect it. Then explicitly trace the pathways from the target treatment to the outcome. If adjustment conditions on a variable along one of those pathways, ask whether the intended effect is total or controlled direct. In the numerical example, the check is concrete: the model’s total contrast is 21, while the regression conditioning on \(L_1\) and \(A_2\) returns 3.

Model fit, small standard errors, or a large number of adjusted variables cannot establish that the chosen coefficient answers the desired causal question. Nor does a comparison of adjusted and unadjusted coefficients diagnose the problem by itself: differences can arise from confounding, model specification, or other features. The causal ordering and target contrast must guide interpretation.

The main alternative is not simply to omit every post-treatment covariate. If \(L_1\) is needed to control confounding of \(A_2\), ignoring it in a later-treatment analysis can leave that comparison confounded. A marginal structural model is an alternative framework for estimating marginal effects across treatment histories while accommodating this feedback structure; its assumptions and estimation details require separate treatment. Ordinary regression remains appropriate for clearly specified conditional or controlled contrasts when its model and assumptions match that target.

The linear example isolates the issue rather than representing every longitudinal study. Real data may have nonlinear relationships, binary treatments, measurement error, or unmeasured confounding. The numerical difference between 21 and 3 is a consequence of the displayed equations and assumptions, not a universal amount or direction of bias. If the regressors are collinear, separate regression slopes may not be identifiable; if the conditional error-mean assumption fails, the stated conditional regression coefficients need not follow.

Common mistake

Wrong reasoning: “\(L_1\) predicts \(A_2\) and \(Y\), so it is a confounder; therefore I should always adjust for it when estimating the effect of \(A_1\).” This argument notices only the later treatment decision. In the example, \(L_1\) is also caused by \(A_1\). Conditioning on it holds fixed a variable through which the early treatment changes the outcome, removing that part of the total effect.

Correction: State which effect is being estimated before deciding whether to condition on \(L_1\). For the total effect of \(A_1\), the contrast allows \(L_1\) and \(A_2\) to respond to \(A_1\); for a controlled direct effect, fixing them is part of the definition. For the later treatment effect, \(L_1\) may be needed to address confounding. The same covariate can therefore be appropriate for one target and inappropriate for another.

Key takeaway: “Adjust for every measured covariate” is not a causal strategy. A post-treatment covariate can control confounding of a later treatment and simultaneously carry part of an earlier treatment’s effect. Choose the adjustment approach by the causal contrast, not by an automatic rule.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the displayed structural example, what is the one-unit total effect of \(A_1\) when \(L_1\) and \(A_2\) are allowed to evolve?
What does the coefficient of \(A_1\) in the regression of \(Y\) on \(A_1,L_1,A_2\) equal, and what contrast does it represent in this model?
Why are mean-zero errors alone insufficient to establish the stated conditional mean in the full regression?
Give the two different roles \(L_1\) plays in the example.

Solutions. (1) Substitution gives \(Y=21A_1+9U_L+5U_2+\varepsilon\), so the one-unit total contrast is 21. (2) The coefficient is 3; with \(L_1,A_2\) fixed, it is the controlled direct contrast per unit of \(A_1\). (3) Mean zero gives \(E[\varepsilon]=0\), not necessarily \(E[\varepsilon\mid A_1,L_1,A_2]=0\). The example assumes mutual independence of the errors and \(A_1\), which supplies the needed conditional-mean property. (4) \(L_1\) is affected by \(A_1\), so it carries an earlier-treatment pathway; it also predicts \(A_2\) and \(Y\), so it is relevant to confounding of the later treatment’s effect.