Choosing Comparison Groups in Multi-Period Designs

Why Comparison-Group Choice Matters

In a two-period design, it is natural to describe one group as treated and another as untreated. With several periods and different treatment start dates, that distinction must be made anew at each date. A group that is untreated today may begin treatment later; a group already treated cannot ordinarily represent the untreated outcome path for another group. The choice of comparison group therefore determines which outcome changes are used to construct a counterfactual.

That choice also needs careful interpretation. For a fixed treated cohort and follow-up date, the causal target can remain the same when the controls change: the average effect for that treated cohort at that date. But the estimated contrast, the parallel-trends assumption needed to interpret it, and the treated cohorts that can be included in an overall summary may change. Keeping these distinctions explicit prevents a convenient comparison from silently redefining the question.

Definition: Let $G_i$ denote unit $i$'s first treatment period. The treatment cohort $G_i=g$ consists of units first treated in period $g$. Units with $G_i=\infty$ are never treated during the observed period. Under absorbing treatment, a unit is untreated before $G_i$ and treated from $G_i$ onward.

For example, if treatment is first introduced in periods 3 and 5 for two cohorts, then a unit in cohort $G_i=5$ is not yet treated in period 4, but is treated in period 5. The distinction is about treatment status at the date being analyzed, not a permanent label attached to that cohort. A cohort already treated in period 3 is not an untreated comparison in period 5.

Eligible Comparisons at a Given Date

Suppose cohort $g$ is the treated cohort of interest, $t$ is a follow-up period with $t\geq g$, and $b<g$ is a pre-treatment baseline period. For a comparison to represent an untreated outcome change through period $t$, its units must remain untreated in both $b$ and $t$. With treatment starting at $G_i$, a common eligible comparison set is $G_i>t$, which includes not-yet-treated cohorts and, if present, never-treated units. Never-treated units can also be used alone. The appropriate set depends on the design and the credibility of its identifying assumptions.

A cohort that starts treatment after the baseline but by period $t$ is not eligible as an untreated control for that follow-up contrast: its observed change includes treatment. Likewise, an already-treated cohort is not untreated at either date. Including such cohorts requires a different design and assumptions; they should not be added to the ordinary untreated comparison set just because their treatment began at a different time.

Worked Example: Checking Eligibility Across Dates

Suppose a panel contains three cohorts: $G_i=3$, $G_i=5$, and $G_i=\infty$. Consider cohort $G_i=3$, using period $b=2$ as its baseline.

At follow-up $t=4$, the $G_i=5$ cohort has not started treatment, and the $G_i=\infty$ cohort is never treated. Both satisfy $G_i>4$, so either may be considered as a comparison, subject to the assumptions discussed below. The $G_i=3$ units are treated by period 4 and are the focal cohort, not controls.

At follow-up $t=6$, the $G_i=5$ cohort has also started treatment. It no longer qualifies as an untreated comparison. Only the never-treated cohort satisfies $G_i>6$. If there were no never-treated units, this design would have no untreated comparison cohort for this later contrast.

The eligibility rule changes with $t$, even though the focal treated cohort remains the same. Thus, estimates at different follow-up dates may rely on different comparison populations and different parallel-trends assumptions.

The Cohort-by-Time Effect and Its Identification

To state the target precisely, let $Y_{it}(g)$ be unit $i$'s outcome in period $t$ under a treatment path that begins in period $g$, and let $Y_{it}(\infty)$ be its outcome if it remains untreated. For cohort $g$ at date $t\geq g$, the cohort-time average treatment effect is the mean difference between those outcomes for units in that cohort.

Definition: The cohort-time average treatment effect is $$ ATT(g,t)=E\!\left[Y_{it}(g)-Y_{it}(\infty)\mid G_i=g\right], \qquad t\geq g. $$ It is the average effect at date $t$ for units first treated in period $g$, relative to those same units remaining untreated.

Only one of the two potential outcomes is observed for each unit. Difference-in-differences identifies the missing untreated outcome change by comparing the treated cohort's change with an eligible comparison group's change. The required condition is parallel trends in untreated outcomes between the focal cohort and the selected comparison group, for the specified baseline and follow-up dates. In an unconditional version, this says that their expected changes would have been equal if both groups had remained untreated.

Identification condition: For a comparison set $C$ that is untreated through $t$, parallel trends requires $$ E\!\left[Y_{it}(\infty)-Y_{ib}(\infty)\mid G_i=g\right] = E\!\left[Y_{it}(\infty)-Y_{ib}(\infty)\mid i\in C\right]. $$ This condition concerns untreated potential outcomes, not necessarily the observed outcomes after some units receive treatment.

The usual contrast also relies on treatment status being correctly defined, no anticipation in the baseline outcome, and observed outcomes agreeing with the potential outcome under the treatment actually received. In particular, no anticipation means that treatment scheduled to begin at $g$ does not affect the focal cohort's outcome at $b<g$. The comparison group must actually remain untreated through $t$. Given these conditions, a short derivation shows why the contrast identifies the target.

Deriving the Comparison

Write the observed mean change for the focal cohort and subtract the observed mean change for the eligible controls. Before treatment, the focal cohort's observed baseline outcome is its untreated outcome; at $t$, its observed outcome is under adoption at $g$. Controls are untreated at both dates. Adding and subtracting the focal cohort's mean untreated change isolates the treatment effect:

\begin{aligned} & E[Y_{it}\mid G_i=g]-E[Y_{ib}\mid G_i=g] -\left\{E[Y_{it}\mid i\in C]-E[Y_{ib}\mid i\in C]\right\} \\ ={}& E[Y_{it}(g)-Y_{ib}(\infty)\mid G_i=g] -E[Y_{it}(\infty)-Y_{ib}(\infty)\mid i\in C] \\ ={}& E[Y_{it}(g)-Y_{it}(\infty)\mid G_i=g] \\ &\quad + E[Y_{it}(\infty)-Y_{ib}(\infty)\mid G_i=g] -E[Y_{it}(\infty)-Y_{ib}(\infty)\mid i\in C] \\ ={}& ATT(g,t). \end{aligned}

The last equality uses parallel trends: the two untreated mean changes cancel. The derivation also shows the danger of an unsuitable comparison group. If its untreated change differs from the focal cohort's untreated change, the last two terms do not cancel; the observed difference-in-differences then combines the treatment effect with a difference in untreated trends.

Worked Comparisons and What They Target

Worked Example: Pooling Not-Yet-Treated and Never-Treated Controls

Suppose a treated cohort has a mean outcome of 10 dollars per month at baseline and 18 dollars per month at follow-up. At that follow-up, two equal-sized eligible control cohorts remain untreated. The not-yet-treated cohort's mean rises from 20 to 25 dollars per month, while the never-treated cohort's mean rises from 30 to 32 dollars per month. Assume the focal cohort and the pooled controls satisfy parallel trends in untreated outcomes.

The treated cohort's observed change is $$ 18-10=8\text{ dollars per month}. $$ The two control cohorts have changes of $25-20=5$ and $32-30=2$ dollars per month. Because they are equal in size, the pooled control change is their equally weighted average: $$ \frac{1}{2}(5)+\frac{1}{2}(2)=3.5\text{ dollars per month}. $$

The difference-in-differences estimate is therefore $$ (18-10)-3.5=8-3.5=4.5\text{ dollars per month}. $$ Under the stated assumptions, this estimates the treated cohort's average treatment effect at follow-up: an increase of 4.5 dollars per month relative to its untreated outcome.

If the never-treated cohort alone were used, the contrast would instead be $8-2=6$ dollars per month. This difference does not, by itself, show which estimate is correct. Each comparison requires parallel trends between the focal cohort and the selected controls. If the pooled comparison satisfies that condition, the estimate using the never-treated cohort alone will generally require its own condition and need not identify the same causal effect. The cohort-time target is unchanged when the focal cohort and date are fixed; what changes is the comparison-based estimate and the assumption supporting its interpretation.

The example highlights a distinction between an estimand and an estimator. $ATT(g,t)$ defines the causal quantity for a specified treated cohort and date. A selected control group supplies observed changes used to estimate it. If the chosen controls are ineligible or their untreated trends do not match, the resulting contrast is not justified as an estimate of that target merely because it can be calculated.

Worked Example: A Changing Comparison Set Across Follow-Up Dates

Return to a design with cohorts $G_i=3$, $G_i=5$, and $G_i=\infty$, and focus on $G_i=3$. Suppose the baseline is period 2. At period 4, both later-treated and never-treated units are eligible controls. At period 6, only never-treated units are eligible. Consequently, an estimate at period 4 can use the change among both eligible cohorts, while an estimate at period 6 must use the never-treated cohort if the analysis requires controls untreated through each follow-up date.

This change has two implications. First, parallel trends must hold between the focal cohort and the period-4 comparison set for the first contrast, and between the focal cohort and the never-treated cohort for the later contrast. The condition for one set does not imply the condition for the other. Second, if the estimates are combined into a summary over dates, the availability of controls can determine which cohort-date effects are estimable and included. Such a summary should state which effects it combines; it is not automatically an effect for every treated cohort at every date.

The comparison set alone does not redefine $ATT(3,4)$ or $ATT(3,6)$. It changes which evidence is used to identify those quantities and may restrict which cohort-date effects can be reported. A summary's target changes when its included treated cohorts, dates, or weights change.

How Aggregation Changes the Target

A multi-period study often has several cohort-time effects. To report one number, the analyst must specify how those effects are combined. For example, an equal-cohort average gives the same weight to each included treatment cohort, while a unit-weighted average gives more weight to cohorts with more treated units. These summaries answer different questions when effects differ across cohorts.

Worked Example: Equal-Cohort and Unit-Weighted Effects

Suppose two treated cohorts have estimated effects at a common follow-up: 4 dollars per month for an early cohort of 80 units and 10 dollars per month for a later cohort of 20 units. If the target is the equal-cohort average, it is $$ \frac{1}{2}(4)+\frac{1}{2}(10)=7\text{ dollars per month}. $$

If the target is instead the average effect per treated unit across these two cohorts, the cohort weights are $80/100=0.8$ and $20/100=0.2$, giving $$ 0.8(4)+0.2(10)=3.2+2=5.2\text{ dollars per month}. $$ The two values differ because the cohort effects differ and the summaries apply different weights. This is a change in the aggregate estimand, not merely a change in the control group used to estimate one fixed cohort-time effect.

Comparison-group eligibility can indirectly affect aggregation: if no credible untreated comparison is available for a cohort at a date, that cohort-time effect may be omitted or require a different strategy. The resulting summary then represents a different collection or weighting of effects unless the target is explicitly held fixed. Report the treated cohorts and dates included, the weights used, and the comparison rule. This is especially important when treatment effects vary across adoption cohorts or over time.

Common Pitfall

Wrong reasoning: “The cohort treated earlier is a useful control for the later-treated cohort because it was observed before the later cohort adopted.”
Correction: At a follow-up when the earlier cohort is already treated, its observed change may include its own treatment effect. It does not supply an untreated outcome change for the later-treated cohort. Use units still untreated through that follow-up, or justify a different method with its own assumptions.
Another common confusion: “Changing controls necessarily changes the causal target.” For a fixed treated cohort and date, the target $ATT(g,t)$ remains the same. The comparison changes the contrast and the identifying condition. The target changes when the treated population, follow-up dates, or aggregation weights change.

Key takeaway: At each follow-up date, identify controls that remain untreated through that date. State the parallel-trends condition for that particular comparison. Keep the cohort-time causal target distinct from the comparison-based contrast, and specify cohort and date weights whenever effects are aggregated.

Check Your Understanding

Answer each question by naming the relevant cohort, date, comparison rule, or target.

A design has cohorts $G_i=2$, $G_i=4$, and $G_i=\infty$. For cohort $G_i=2$ at follow-up $t=3$, which cohorts can be untreated comparisons under the rule $G_i>t$?
In the same design, which cohorts satisfy that rule at $t=5$? Why is the $G_i=4$ cohort no longer eligible?
State in words what parallel trends requires for a treated cohort and a selected comparison group.
If two eligible comparison groups produce different difference-in-differences estimates for the same treated cohort and date, what can and cannot be concluded from that difference alone?
Two cohort effects are 3 and 9 units. Explain why an equal-cohort average can differ from a unit-weighted average, and what choice determines the aggregate target.