Common DiD Specification and Interpretation Errors

Why Small Specification Choices Matter

A difference-in-differences (DiD) analysis compares how an outcome changes over time in a treated group with how it changes in a comparison group. Several common mistakes can make a calculation or regression answer a different question: comparing only post-period levels, leaving out the group-by-period interaction, controlling for a variable measured after treatment, or reporting an interaction without identifying its scale.

The arithmetic of a DiD is often simple. The harder task is ensuring the model actually represents that arithmetic and that the resulting contrast has the intended interpretation. A regression coefficient is not automatically a causal effect: causal interpretation also requires a suitable design and assumptions.

The Two-Group, Two-Period Specification

Let \(G=1\) identify the treated group and \(G=0\) the comparison group. Let \(P=1\) denote the post-intervention period and \(P=0\) the pre-intervention period. Write \(\bar{Y}_{gp}\) for the mean outcome in group \(g\) and period \(p\). The sample difference-in-differences is the treated group's change minus the comparison group's change.

\widehat{\mathrm{DiD}} = (\bar{Y}_{11}-\bar{Y}_{10}) - (\bar{Y}_{01}-\bar{Y}_{00})

The corresponding saturated regression is \(Y=\beta_0+\beta_1G+\beta_2P+\beta_3(GP)+\varepsilon\). “Saturated” here means that the four group-period cells can have four distinct fitted means. The interaction \(GP\) equals 1 only for treated-group observations in the post period.

Definition: In the two-group, two-period regression, the interaction coefficient \(\beta_3\) is the difference between the treated and comparison groups' changes in mean outcome: \(\beta_3=(\bar{Y}_{11}-\bar{Y}_{10})-(\bar{Y}_{01}-\bar{Y}_{00})\). It is a DiD contrast on the outcome's modeled scale.

This coefficient has a causal interpretation only under design assumptions. In particular, the groups' mean untreated potential outcomes must have followed parallel trends: absent the intervention, the treated group's mean outcome would have changed by the same amount as the comparison group's mean outcome. The interpretation also requires that treatment has not affected outcomes in the pre period, that treatment and outcome are defined consistently, and that changes in the observed groups or their composition do not invalidate the comparison. These assumptions justify a causal reading; the regression formula alone does not establish them.

Error 1: Comparing Post-Period Levels Instead of Changes

The post-period gap, \(\bar{Y}_{11}-\bar{Y}_{01}\), compares treated and comparison groups after the intervention. It is not generally the DiD. If the groups already differed before the intervention, the post-period gap includes that pre-existing difference. DiD subtracts the pre-period gap, thereby comparing changes rather than post-period levels alone.

Worked Example: A Post-Period Gap Is Not the DiD

Suppose a hypothetical program is introduced for one group. The mean outcome is measured in points, with these cell means: treated group, 50 before and 62 after; comparison group, 45 before and 52 after. Assume for the calculation that the same outcome and group definitions are used in both periods.

The treated group's change is \(62-50=12\) points. The comparison group's change is \(52-45=7\) points. Thus,

\widehat{\mathrm{DiD}}=(62-50)-(52-45)=12-7=5\text{ points}.

The post-period treated-control gap is \(62-52=10\) points. But the pre-period gap was \(50-45=5\) points, so the post gap includes the initial 5-point difference. Subtracting that baseline gap gives \(10-5=5\) points, the same DiD contrast.

Under parallel trends and the other design assumptions, 5 points can be interpreted as the estimated treatment effect on the treated group. The cell means alone do not verify those assumptions, so without them the result is a descriptive difference in changes, not necessarily a causal effect.

Error 2: Omitting the Interaction

The interaction is what allows the treated group's change to differ from the comparison group's change. A model with group and period indicators but no interaction instead imposes an additive restriction: the group gap is the same in both periods, or equivalently, both groups have the same change in their fitted means. That restriction removes the very contrast DiD is intended to estimate.

In a balanced four-cell data set, the treatment-group coefficient in this additive model is the average of the two observed group gaps. It is not the difference between changes. With unequal cell sizes or additional regressors, the exact fitted coefficient depends on the model and data weighting; it still does not become the usual DiD interaction merely because the regression includes group and period indicators.

Worked Example: What an Additive Model Imposes

Consider hypothetical cell means, in points, with the same number of observations in each cell: comparison group 20 before and 24 after; treated group 30 before and 40 after. The changes are \(40-30=10\) points for the treated group and \(24-20=4\) points for the comparison group.

The DiD is

(40-30)-(24-20)=10-4=6\text{ points}.

Now fit the additive model \(Y=\beta_0+\beta_1G+\beta_2P+\varepsilon\), without \(GP\). With equal cell sizes, the fitted group coefficient is the average group gap:

\hat{\beta}_1 = \frac{(30-20)+(40-24)}{2} = \frac{10+16}{2} = 13\text{ points}.

The fitted period coefficient is the average change: \((4+10)/2=7\) points. The additive model therefore describes a constant 13-point group gap and a common 7-point time change. The observed group gaps, however, are 10 points before and 16 points after. The model cannot represent that difference; its implied interaction is zero. The coefficient 13 is not the 6-point DiD.

For the standard two-period DiD regression, include the interaction explicitly, such as \(G\), \(P\), and \(G \times P\). In settings with multiple periods or covariates, the model may require additional terms, but the principle remains: verify that the specification allows the contrast of interest rather than silently imposing equal changes.

Error 3: Adjusting for a Post-Treatment Variable

A covariate measured after treatment begins is not automatically a suitable adjustment variable. Treatment may change it. Controlling for such a variable can remove part of the treatment effect that operates through it, or can introduce bias if it is also affected by causes of the outcome. The adjusted coefficient may therefore target a different quantity—or fail to have a causal interpretation—than the total effect of interest.

This issue is distinct from adjusting for baseline covariates. Covariate adjustment based on pre-treatment information can be useful under appropriate assumptions. By contrast, deciding to control for a post-treatment variable requires a clear causal target and additional assumptions; including it because it predicts the outcome is not enough.

Worked Example: Adjustment Can Remove a Mediated Effect

Consider a hypothetical intervention indicator \(D\), a post-treatment mediator \(M\), and an outcome \(Y\) measured in points. Suppose the data are generated by \(M=D+U\) and \(Y=10+2D+3M+\varepsilon\), where \(U\) varies across people and is independent of \(D\), and \(\varepsilon\) has mean zero and is independent of both \(D\) and \(U\). Thus, \(M\) is affected by the intervention but is not perfectly determined by it.

Under these stated structural assumptions, changing \(D\) from 0 to 1 increases \(M\) by 1 unit. The direct effect through the \(2D\) term is 2 outcome points, and the mediated contribution through \(M\) is \(3 \times 1=3\) points. The total effect is therefore

2 + 3(1)=5\text{ outcome points}.

A regression that adjusts for \(M\) can, under this correctly specified model and the independence assumptions above, recover the coefficient 2 on \(D\). That coefficient represents the controlled direct effect in this example, not the 5-point total effect. Since \(M\) varies within each value of \(D\), the coefficients are not separated by perfect collinearity; nevertheless, adjustment changes the target by holding the mediator fixed.

In a DiD analysis, the same warning applies to variables measured after the intervention: if treatment affects them, adjustment may block a pathway or create bias. Do not describe an adjusted estimate as the total DiD effect without assumptions that justify that target and adjustment.

Error 4: Reporting an Interaction Without Its Scale

An interaction is a departure from additivity or multiplicativity on a specified scale. For a continuous outcome in the linear model above, \(\beta_3\) is a difference in differences of means, in the outcome's units. For a binary outcome, a linear probability model expresses an interaction as a difference in risk changes, in percentage points. A logistic model instead expresses its interaction on the log-odds scale. Those are different quantities and need not agree.

Worked Example: Risk-Scale and Odds-Scale Interactions

Suppose a hypothetical binary outcome has the following risks: treated group, 15% before and 35% after; comparison group, 10% before and 20% after. The difference-in-differences on the risk scale is

$$ (0.35-0.15)-(0.20-0.10)=0.20-0.10=0.10. $$

This is a 10-percentage-point difference in risk changes. In a saturated logistic regression, the interaction coefficient is instead the log of the ratio of the groups' pre-to-post odds ratios. The treated group's post-to-pre odds ratio is \((0.35/0.65)/(0.15/0.85)=3.051\). The comparison group's is \((0.20/0.80)/(0.10/0.90)=2.250\). Thus,

\hat{\theta} = \log\left(\frac{3.051}{2.250}\right) = \log(1.356) \approx 0.3046.

The value 0.3046 is an interaction on the log-odds scale. Exponentiating it gives 1.356, the ratio of the two groups' odds ratios. It is not a 30.46-percentage-point effect, nor is it the 10-percentage-point risk-scale DiD. A causal interpretation on either scale still requires suitable DiD assumptions.

A Short Derivation of the Interaction Contrast

The saturated regression makes the cell-mean relationship explicit. At \(G=0,P=0\), the fitted mean is \(\beta_0\); at \(G=1,P=0\), it is \(\beta_0+\beta_1\); at \(G=0,P=1\), it is \(\beta_0+\beta_2\); and at \(G=1,P=1\), it is \(\beta_0+\beta_1+\beta_2+\beta_3\). Substituting these four means into the difference in changes gives

\begin{aligned} &[(\beta_0+\beta_1+\beta_2+\beta_3)-(\beta_0+\beta_1)]\\ &\quad-[(\beta_0+\beta_2)-\beta_0]\\ &=(\beta_2+\beta_3)-\beta_2\\ &=\beta_3. \end{aligned}

This derivation also shows why omitting the interaction is consequential: setting \(\beta_3=0\) forces the two fitted changes to be equal. It does not show that the observed groups would have had parallel untreated trends; that is a substantive identifying assumption, not an algebraic consequence of the model.

Common Mistakes to Check Before Interpreting a Result

Calling the post-period gap a DiD. The correction is to subtract the pre-period gap, or directly calculate the difference between the two groups' changes.
Using only group and period indicators. Without the interaction, the model imposes a common change. Check that the interaction needed for the DiD contrast is present.
Adjusting for every available predictor. Check when each variable was measured and whether treatment could affect it. Post-treatment adjustment may change the estimand or introduce bias.
Reporting an interaction without units or scale. State whether it is a difference in means, a difference in risks, a log-odds contrast, or another scale-specific quantity.
Treating a correctly computed coefficient as proof of causality. The regression calculation does not establish parallel trends or the other design assumptions. Precision and standard errors also require appropriate inference for the assignment and dependence structure.

Key takeaway: In a two-period DiD, the group-by-period interaction represents the difference in outcome changes. A defensible interpretation also requires the right outcome scale, a deliberate choice of covariates, and credible identifying assumptions.

Check Your Understanding

For each question, distinguish what the model computes from what would be needed to interpret it causally.

A treated group changes from 40 to 48, while a comparison group changes from 35 to 39. What is the DiD, and what is the post-period gap?
In the two-group, two-period linear model, what assumption about fitted group changes is imposed when the interaction is omitted?
A variable measured after treatment is strongly associated with the outcome. Why is that fact alone insufficient to justify adjusting for it?
In a logistic DiD model, what scale does the group-by-period interaction coefficient use?
Does including the interaction in a regression establish parallel trends? Explain briefly.