Why Composition Matters for DiD
A difference-in-differences (DiD) comparison concerns changes in outcomes for a treated group and a comparison group. But a group observed in two periods need not contain the same mix of people, households, hospitals, or other units in each period. Even if each type of unit follows a stable outcome pattern, a change in the mix can alter the group’s average outcome. The resulting aggregate trend may not represent the trend that would have occurred for a fixed population.
This issue is distinct from whether baseline outcome levels match: DiD does not require those levels to be equal. The concern is whether changes in the groups’ composition, or in which units are observed, make their outcome trends incomparable. A composition check therefore helps assess the plausibility and meaning of a trend comparison; it cannot establish parallel trends by itself.
For example, if the treated group contains a larger share of older people in the post-period, its average outcome may change even if the outcome trend within each age group is unchanged. The size and direction of this effect depend on the outcome trends within age groups. A change in composition is not necessarily a problem: if outcome trends are the same across all relevant types, changing their proportions will not create a different aggregate trend. But when trends vary across types, shifts in proportions can matter.
Separating Within-Group Trends From Composition
Suppose \(X\) takes a finite set of values, such as “younger” and “older.” Write \(p_{gt}(x)\) for the share of group \(g\) with characteristic value \(x\) in period \(t\), and \(m_{gt}(x)\) for the mean outcome among those units. The group-period mean is a weighted average of the means within categories:
The change in a group’s mean can be decomposed exactly by adding and subtracting the period-1 means weighted by the period-0 composition:
The first term is the change in category-specific means, averaged using the period-0 composition. The second is the composition component: it captures how the period-1 mean changes when the category shares shift, holding the period-1 category means fixed. The decomposition is an accounting identity, not a causal model. Its particular reference weights matter for how the two components are described, but their sum is the observed change.
Worked Example: Composition Creates a False Aggregate Trend Difference
Consider a hypothetical setting with two types of units, low-trend and high-trend. Assume there is no treatment effect. The untreated mean outcome for low-trend units rises from 10 to 11 units, while the mean for high-trend units rises from 20 to 24 units. The comparison group is 80% low-trend and 20% high-trend in both periods. The treated group has the same pre-period shares but is 20% low-trend and 80% high-trend in the post-period.
The treated group’s pre-period mean is
Its post-period mean, using its changed composition, is
Thus the treated group’s aggregate change is \(21.4-12=9.4\) units. For the comparison group, the shares remain 80% and 20%, so
The aggregate DiD contrast is \(9.4-1.6=7.8\) units, despite the stipulated absence of a treatment effect. To see the source, hold the treated group’s shares at their pre-period values: its post-period mean would be \(0.8(11)+0.2(24)=13.6\), a 1.6-unit increase. The extra 7.8 units in its observed change come from its shift toward the type with the larger outcome trend. The calculation shows why a composition change can undermine an aggregate trend comparison when category-specific trends differ.
What to Check in the Data
A practical diagnostic begins by making the group and period explicit. For each covariate that could be related to outcome trends, compare its distribution in the treated and comparison groups over time. Depending on the covariate, this can mean comparing category shares, means, quantiles, or a fuller distribution. Also report the number of observed units in each group-period cell. A single pooled comparison of treated and comparison groups can conceal a change that occurs only after the intervention.
The following hypothetical table illustrates a simple check of the share of older participants. It does not show that age caused any outcome trend; it shows that the composition differs over time in a way worth investigating.
| Group | Pre-period older participants | Post-period older participants | Change in share |
|---|---|---|---|
| Treated | 30/100 = 0.30 | 60/100 = 0.60 | 0.30 |
| Comparison | 30/100 = 0.30 | 32/100 = 0.32 | 0.02 |
Worked Example: Calculating a Differential Composition Change
Suppose the counts in the table come from a hypothetical repeated cross-section. The treated group’s older-participant share changes by \(0.60-0.30=0.30\), or 30 percentage points. The comparison group’s share changes by \(0.32-0.30=0.02\), or 2 percentage points. The difference between these composition changes is
The treated group therefore has a 28-percentage-point larger increase in its older-participant share than the comparison group. This is a diagnostic for a differential change in age composition, not a DiD estimate of an outcome effect and not proof that the outcome trends are biased. To judge its relevance, investigate whether age predicts the outcome’s untreated trend and whether results depend on comparing a common covariate distribution. Also verify that the group-period counts and denominators represent the intended populations.
When covariate-specific trends are plausibly different, a useful follow-up is to compare like with like: examine outcomes within relevant covariate strata or use a common reference distribution when summarizing stratum-specific changes. This connects to covariate-conditional parallel trends and standardization. Such an analysis requires adequate overlap: each relevant covariate category must be represented in the groups and periods being compared. It also requires a defensible choice of target population and assumptions about trends within categories. Adjusting for measured covariates does not resolve changes in unmeasured characteristics or make an unsupported trend assumption true.
Sample Inclusion and Attrition
Composition can change because the population itself changes, as in repeated cross-sections, or because units enter or leave the observed sample. Examples include panel attrition, changing eligibility rules, incomplete records, or changes in who responds to a survey. If inclusion is related to outcomes or to their trends, the observed mean may no longer describe the same population across periods. A DiD calculation on those observed means can then combine outcome change with selection change.
A balanced panel restricts analysis to units observed in every period, which makes the set of included units constant by construction. That can clarify the population being analyzed, but it does not guarantee that the retained units represent the original population or that their untreated trends are parallel. In repeated cross-sections, by contrast, the individuals can differ by period even when the sampling frame is stable. In either design, describe who is observed, how inclusion is determined, and how inclusion rates vary by group and period.
Worked Example: Differential Attrition Alters the Observed DiD
Consider a hypothetical panel with 100 units in each group at baseline. The treated group’s baseline outcome mean is 50 units. Of those units, 80 are observed at follow-up; their baseline mean was 48. The other 20, who are not observed at follow-up, had a baseline mean of 58. These values are consistent with the full baseline mean because \(0.8(48)+0.2(58)=38.4+11.6=50\).
Suppose, for illustration, that without treatment both retained and non-retained treated units would have outcomes 2 units higher at follow-up than at baseline. The retained units’ follow-up mean would then be \(48+2=50\), and the non-retained units’ would be \(58+2=60\). The full treated-group follow-up mean under this no-effect scenario would be
Assume the comparison group is fully observed, with a mean of 50 units at baseline and 52 units at follow-up. The full-population trend is 2 units in each group, so the population DiD is \(2-2=0\). But if the observed treated baseline mean uses all 100 units while the follow-up mean uses only the 80 retained units, the observed treated change is \(50-50=0\). The observed comparison change is \(52-50=2\), giving an observed DiD of \(0-2=-2\) units. The apparent contrast is caused by comparing different treated-group samples across periods, not by a treatment effect. The example’s unobserved outcomes are stipulated to show the mechanism; in real data, those outcomes are unknown, so the effect of attrition requires substantive assumptions or additional analysis.
Interpreting Diagnostics Carefully
A composition diagnostic is a warning signal, not a pass-fail test for DiD. A detected shift matters most when it concerns characteristics associated with different untreated outcome trends, differs between the groups, and changes the population represented by the group-period means. Conversely, stable measured covariates do not rule out changes in unmeasured characteristics, and apparent balance does not prove parallel trends.
Keep three questions separate: Did the distribution of observed characteristics change? Did the set of units included in the analysis change, and why? Could those changes alter the aggregate trend comparison or the population to which the estimate applies? Descriptive tables, plots, and inclusion counts help answer the first two. The third requires substantive reasoning about outcome trends, measurement, and the target population.
Report group-period counts, inclusion rates, and the covariate distributions relevant to the outcome and study setting.
Compare changes within each group, then assess whether the treated and comparison groups changed differently.
Ask whether the changing characteristics could have different untreated outcome trends, and whether comparisons within strata or at common covariate weights are supportable.
Explain what population the observed means represent and what assumptions remain about unmeasured composition and sample inclusion.
These checks matter because a DiD estimate is an interpretation of a contrast between group-level changes, not merely an arithmetic difference. If those changes refer to shifting populations, an aggregate contrast may not answer the intended causal question. Clear reporting makes it possible to distinguish an outcome trend from a change in who contributes to that trend.
Check Your Understanding
Use the definitions and decomposition above to answer the following questions.
- In the decomposition of a group mean change, what does the within-category component represent, and what does the composition component represent?
- Can a group’s covariate distribution change without changing its aggregate outcome trend? Give a condition under which the composition component is zero.
- A covariate share rises by 12 percentage points in the treated group and 5 percentage points in the comparison group. What is the differential change in that share?
- Why can comparing all baseline units with only follow-up respondents create an apparent DiD even when the full-population untreated trends are equal?
- Why does detecting a differential covariate shift neither prove DiD bias nor establish that the identifying assumption is plausible?