Covariate-Adjusted Difference-in-Differences

Why Adjust Difference-in-Differences?

Difference-in-differences (DiD) compares the change in outcomes for a treated group with the change for a comparison group. If the groups have different distributions of characteristics that predict outcome trends, their average changes can differ even when treatment has no effect. Covariate adjustment addresses this problem by comparing treated units with control units who have similar pre-treatment covariates, then averaging those comparisons over the treated group.

The adjustment does not make the design causal by itself. Its interpretation still depends on an assumption about how untreated outcomes would have changed, now stated conditionally on covariates. Adjustment also requires control units at relevant covariate values. The target described here is the average treatment effect on the treated (ATT), not an effect for an arbitrary population.

Setup and Identification Assumptions

Consider two periods, $t=0$ before treatment and $t=1$ after treatment. Let $D_i=1$ indicate that unit $i$ is treated in period 1, and let $D_i=0$ indicate that it is in the comparison group. Let $Y_{it}(d)$ denote unit $i$'s potential outcome in period $t$ under treatment status $d$. The observed outcome is $Y_{it}=Y_{it}(D_i)$. Define the observed outcome change as $\Delta Y_i=Y_{i1}-Y_{i0}$. Let $X_i$ be a vector of covariates measured before treatment.

The basic setup assumes treatment does not affect the pre-treatment outcome, treatment status is well defined, and observed outcomes are consistent with the corresponding potential outcomes. In a panel, the same units are observed in both periods. The outcome and covariates must also be measured comparably across groups and periods. For repeated cross-sections, the conditional means must describe comparable populations in each group-period cell; changes in sample composition that remain after conditioning on $X$ can undermine the interpretation.

Conditional parallel trends: For covariate values $x$ represented among treated units, the mean change in untreated potential outcomes is the same for treated and comparison units: $$ E[Y_{i1}(0)-Y_{i0}(0)\mid D_i=1,X_i=x] = E[Y_{i1}(0)-Y_{i0}(0)\mid D_i=0,X_i=x]. $$ Overlap: At covariate values occurring among treated units, there must be comparison units from whom the untreated trend can be learned. In population notation, $P(D_i=0\mid X_i=x)>0$ on the treated covariate support.

Conditional parallel trends is a substantive identifying assumption, not a consequence of including $X_i$ in a regression. It says that after comparing units with the same covariates, the control group supplies the treated group’s average change in the absence of treatment. It need not hold for every individual; it is a restriction on conditional means. Covariates should be measured before treatment so that they are not themselves affected by treatment.

Definition: The ATT is the average effect of treatment among treated units in period 1: $$ ATT=E[Y_{i1}(1)-Y_{i1}(0)\mid D_i=1]. $$

From Conditional Trends to an Adjusted Estimator

Define $m_0(x)=E[\Delta Y_i\mid D_i=0,X_i=x]$, the control-group mean outcome change at covariate value $x$. Under consistency and no treatment in period 0, the treated group’s observed change consists of its untreated change plus its treatment effect. Conditional parallel trends lets us replace the treated group’s unobserved mean untreated change with $m_0(x)$. Averaging that comparison over the treated covariate distribution gives the identification result.

\begin{aligned} ATT &=E[\Delta Y_i-\{Y_{i1}(0)-Y_{i0}(0)\}\mid D_i=1]\\ &=E[\Delta Y_i\mid D_i=1] -E\!\left[E[Y_{i1}(0)-Y_{i0}(0)\mid D_i=1,X_i]\mid D_i=1\right]\\ &=E[\Delta Y_i\mid D_i=1]-E[m_0(X_i)\mid D_i=1]. \end{aligned}

The first equality uses the fact that treated units have no treatment effect in the pre-period, so their outcome change minus their untreated potential-outcome change is their period-1 treatment effect. In the last equality, conditional parallel trends replaces the treated group’s conditional mean untreated change with the comparison group’s conditional mean observed change. Overlap makes that comparison-group mean estimable at treated covariate values.

A sample analogue fits or estimates $m_0(x)$ using comparison units, predicts the untreated change for each treated unit, and subtracts the average prediction from the treated group’s average observed change. This is often called regression adjustment or standardization. The essential feature is the averaging distribution: predictions are averaged over the treated units, because the target is the ATT.

Worked Example: Standardizing Across Two Covariate Strata

Suppose a hypothetical panel study compares a program group with a comparison group. Let $X$ indicate whether a unit is in a high-baseline-risk stratum. Among treated units, 30% are high-risk and 70% are low-risk. Their mean outcome changes are 8 and 5 points, respectively. The comparison group’s mean changes are 4 points for high-risk units and 2.5 points for low-risk units. Assume the strata are defined by pre-treatment information, there are comparison units in both strata, and conditional parallel trends is plausible within each stratum.

First calculate the treated group’s observed mean change: $$ 0.30(8)+0.70(5)=2.40+3.50=5.90\text{ points}. $$ The estimated untreated change for the treated group is the comparison trend averaged using the treated stratum shares: $$ 0.30(4)+0.70(2.5)=1.20+1.75=2.95\text{ points}. $$ Therefore, the standardized DiD estimate is $$ 5.90-2.95=2.95\text{ points}. $$ Under the stated identifying assumptions, this estimates an ATT of 2.95 outcome points for the treated units.

For contrast, suppose the comparison group is 20% high-risk and 80% low-risk. Its overall mean change is $0.20(4)+0.80(2.5)=2.80$ points, so the unadjusted difference in mean changes is $5.90-2.80=3.10$ points. The two estimates differ because the comparison group’s covariate composition is not the treated group’s composition. The adjusted calculation aligns the trend average with the treated target population.

Regression Adjustment and Standardization

A practical regression-adjustment estimator specifies a model for the comparison group’s change, $m_0(x)$, estimates its parameters using comparison units, and predicts changes for the treated units. For example, a linear specification might be $m_0(x)=\beta_0+\beta_1x$. Flexible terms or interactions can be used when the relationship between covariates and untreated trends is not well represented by a straight line. Whatever model is chosen, the estimated prediction must be averaged over the treated units to target the ATT.

One can also estimate conditional changes separately by treatment group and covariates. In population terms, let $m_d(x)=E[\Delta Y_i\mid D_i=d,X_i=x]$. Then the covariate-standardized contrast is $E[m_1(X_i)-m_0(X_i)\mid D_i=1]$. For treated units, $m_1(x)$ is their observed mean change at $x$; the comparison group supplies $m_0(x)$. Under conditional parallel trends, their difference is the conditional treatment effect on treated units at $x$, and averaging over treated units gives the ATT.

A regression coefficient on a treatment indicator is not automatically this standardized ATT. For example, a model that imposes a single treatment effect across covariate values may yield an effect for a model-dependent comparison, especially if the model is misspecified or treatment effects vary with $X$. A direct way to preserve the ATT target is to predict the untreated change for each treated unit and average those predictions, then subtract that average from the treated group’s observed mean change.

Worked Example: Predicting Untreated Changes With a Linear Model

In a hypothetical evaluation, $X$ is a pre-treatment score. A linear model fitted to comparison units estimates their mean outcome change as $\widehat m_0(x)=1+0.5x$, in outcome points. Three treated units have scores $X=2,4,6$ and observed changes of 5, 7, and 8 points. Suppose the fitted model is an adequate description of comparison-group trends over this score range, and comparison units provide support there.

The predicted untreated changes for the treated units are $$ \widehat m_0(2)=1+0.5(2)=2,\quad \widehat m_0(4)=1+0.5(4)=3,\quad \widehat m_0(6)=1+0.5(6)=4. $$ The treated mean observed change is $(5+7+8)/3=20/3=6.667$ points. The mean predicted untreated change is $(2+3+4)/3=3.000$ points. Thus the regression-adjusted estimate is $$ \widehat{ATT}=\frac{20}{3}-3=\frac{11}{3}=3.667\text{ points}. $$ This is an estimate for the treated units represented by these three covariate values, conditional on the model and DiD assumptions; it is not a causal conclusion if those assumptions fail.

Why Covariates Must Be Pre-Treatment

Adjustment is intended to compare units using characteristics that precede treatment, not to hold fixed consequences of treatment. A post-treatment variable can lie on the causal pathway from treatment to outcome. Conditioning on it can remove part of the total effect the ATT is meant to capture. In other settings, conditioning on a post-treatment variable can also induce bias by selecting on a variable affected by treatment and other causes of the outcome.

Worked Example: Adjusting for a Treatment-Induced Mediator

Consider a hypothetical program that increases training participation $M$, which in turn raises an outcome $Y$. Suppose the structural relationships are $M=D+V$ and $Y=2D+3M+U$, where $U$ and $V$ are unaffected by treatment and their distributions do not change across periods. Treatment raises $M$ by one unit and has a direct effect of 2 outcome units.

Substituting the mediator equation into the outcome equation gives $Y=2D+3(D+V)+U=5D+3V+U$. Thus, under these relationships, changing $D$ from 0 to 1 raises the outcome by $5$ units in total: $2$ direct units plus $3$ units through the increase in training. A DiD that conditions on post-treatment $M$ would compare at a fixed mediator value and can remove the mediated $3$-unit pathway, leaving a contrast corresponding to the direct effect rather than the total ATT. The adjusted target has changed. Therefore, $M$ should not be used as an ordinary adjustment covariate when the goal is the total treatment effect.

Common Mistakes and Practical Checks

Using the comparison group’s covariate shares. The target is the ATT, so the predicted untreated changes must be averaged over the treated group’s covariate distribution.
Assuming adjustment guarantees parallel trends. Covariates can make conditional parallel trends more plausible, but their inclusion does not prove the assumption.
Extrapolating beyond overlap. If treated units have covariate values without comparable controls, their untreated trends are not identified from the observed comparison group without additional modeling assumptions.
Adjusting for post-treatment variables. A variable affected by treatment may be a mediator or collider; conditioning on it can introduce bias or change the effect being estimated.
Reading a regression coefficient without checking its target. The ATT calculation requires standardization over treated units; a coefficient from a restrictive regression is not automatically that quantity.

A useful report states the target population, the pre-treatment covariates used, how comparison trends were modeled, and whether treated covariate values were supported by comparison units. The identifying claim should be explicit: after conditioning on these covariates, the treated and comparison groups would have had equal mean outcome changes in the absence of treatment. That claim remains substantive even when the calculation is straightforward.

Check Your Understanding

Answer each question using the definitions and assumptions in this tutorial.

Write the conditional parallel-trends assumption using potential outcomes and pre-treatment covariates.
Why are predicted comparison-group changes averaged over the treated covariate distribution when estimating the ATT?
In a two-stratum example, treated shares are 40% and 60%, and comparison trends are 3 and 1 points. What is the standardized untreated trend for the treated group?
What problem arises if treated units have covariate values for which there are no comparison units?
Explain why controlling for a post-treatment mediator can change a DiD estimate’s target.