Effect Modification and Aggregate Treatment Effects

Why an Aggregate Effect Depends on Who Is in the Population

A treatment can have different effects in different covariate-defined groups. For example, a treatment may reduce an outcome more among patients with severe baseline disease than among those with mild disease. Even when the effect within each group stays the same, the overall average effect changes if the target population contains different proportions of the groups. This distinction matters when interpreting a matched estimate: its aggregate can reflect both treatment-effect variation and the composition of the matched target.

The discussion below uses additive effects on the outcome scale. A difference of 5 units has the same numerical meaning wherever it is calculated, so group effects can be averaged using population proportions. Effect modification is scale-dependent: variation in an additive effect does not automatically imply variation on every other effect scale.

The group effect is an average, not necessarily the effect for every individual in the group. Individual treatment effects may still vary within a group. Therefore, a partition into a few groups captures only the variation represented by those group definitions; unmeasured or finer-grained heterogeneity can remain.

How Group Effects Form an Aggregate

Suppose the groups \(g=1,\ldots,K\) form a mutually exclusive and exhaustive partition of the target population. Write \(q_{Tg}=P_T(G=g)\), the proportion of the target population in group \(g\). The aggregate effect is the group-effect average weighted by those proportions:

This identity follows from the law of total expectation: each group contributes its mean effect multiplied by the fraction of the target population in that group. It does not require effects to be constant within groups. It does require the group effects and group proportions to refer to the same target population.

For a treated-unit target, the corresponding decomposition uses the treated population's group proportions and treated-unit group effects. Define \(q_{1g}=P(G=g\mid A=1)\) and \(\tau_{1g}=E[Y(1)-Y(0)\mid A=1,G=g]\). Then \(\mathrm{ATT}=\sum_g q_{1g}\tau_{1g}\). In general, \(\tau_{1g}\) need not equal the full-population group effect \(\tau_{Tg}\): treated and other units within the same broad group may differ in their treatment-effect distribution.

Effect Variation Is Not the Same as a Composition Shift

Two questions should be kept separate. First, do the mean treatment effects differ across groups? That is a question about effect modification. Second, how much of each group is in the target population? That is a question about composition. An aggregate can differ between populations because their group proportions differ, because their within-group effects differ, or because both differ.

This distinction is directly relevant to matching. A matching procedure can change which units contribute to the estimate and therefore change the group mix of its target. If group effects vary, a different mix can change the aggregate even when the within-group effects are unchanged. Conversely, an aggregate estimate alone cannot reveal whether effects are similar across groups or whether opposing group effects have averaged out.

Assumptions for Learning Group Effects

Potential outcomes are not both observed for the same unit. Identifying group effects therefore requires assumptions about how observed treatment groups represent the relevant potential outcomes. In an observational study, a standard sufficient set is consistency, no interference, conditional exchangeability given measured covariates \(X\), and treatment positivity. Consistency links the observed outcome to the potential outcome under the treatment actually received; no interference rules out one unit's treatment changing another unit's outcome; conditional exchangeability states that, given \(X\), treatment assignment carries no further information about potential outcomes; positivity requires a positive probability of each treatment level at relevant covariate values.

For a group defined by \(G\), these assumptions must support the group effect for the population being discussed. In particular, it is not enough to have both treatment arms somewhere in the study if some relevant covariate profiles within a group have no treated or no control units. Sparse or absent treatment arms can make group effects unidentified or too imprecise to estimate reliably. Matching does not by itself guarantee that these conditions hold.

In a randomized study, random assignment, consistency, no interference, and positive probability of assignment to both arms identify group effects for the randomized study population, provided the groups are defined and represented there. Randomization does not automatically identify effects for a different target population. Transporting group effects to a separate target requires additional transportability assumptions, such as appropriate comparability of effects across study and target populations conditional on relevant covariates, as well as support for the target profiles in the study. The target's group proportions must also be known or estimated for its aggregate effect.

Diagnostics, Alternatives, and Common Pitfalls

When group effects are estimated rather than known, inspect the evidence within each group. Report group sizes and treatment-arm sizes, group-specific estimates with uncertainty, and whether both arms are represented over the covariate profiles relevant to each group. A small subgroup can produce a highly uncertain effect estimate; unstable estimates should not be treated as precise just because the overall sample is large. In a matched analysis, an overall balance summary can also conceal imbalance within an important group, so group-specific comparisons may be informative.

An interaction model can summarize effect variation by including treatment-by-group terms, while direct subgroup estimation reports a contrast separately within each group. These are alternative ways to represent the same broad question, not substitutes for identification assumptions or adequate data. A single pooled contrast is simpler but hides group differences; subgroup estimates are more revealing but can be noisy, especially when many groups are examined. Pre-specifying meaningful groups and interpreting uncertainty helps limit overconfident conclusions from chance patterns.

• Mistake: “The overall effect is zero, so treatment has no effect.” Opposing group effects can cancel in the weighted average. Examine the group effects and their uncertainty.
• Mistake: “A randomized trial identifies the effect in any target population.” Randomization identifies study-population effects. A separate target requires transportability assumptions and appropriate covariate support.
• Mistake: “Different aggregate estimates prove effect modification.” The aggregates may differ only because the populations have different group proportions. Compare group-specific effects as well as composition.
• Mistake: “No statistically detectable interaction proves equal effects.” An imprecise interaction estimate may not detect meaningful variation. Report uncertainty rather than interpreting a non-significant result as proof of homogeneity.