Simpson’s Paradox in Subgroup Treatment Effects

Why the Overall Comparison Can Reverse

A treatment may be associated with a lower event risk in every subgroup, yet with a higher risk in the combined data. This apparently contradictory pattern is called Simpson’s paradox. It arises because an overall risk is a weighted average of subgroup risks, and the weights can differ between treatment groups. The aggregate comparison therefore mixes differences in risks within subgroups with differences in who is represented in each treatment group.

For causal interpretation, keep two questions separate. First, are the treatment effects different across subgroups? Second, do the treatment groups have different subgroup compositions? The first concerns effect heterogeneity; the second concerns composition. A reversal in crude comparisons does not, by itself, answer the first question.

Definition: For a binary outcome \(Y\), treatment \(A\in\{0,1\}\), and baseline subgroup \(S=s\), let \(r_a(s)=P(Y=1\mid A=a,S=s)\) denote the observed event risk. The subgroup risk difference is \(r_1(s)-r_0(s)\). Simpson’s paradox in this setting describes a reversal in the direction of an overall, aggregated comparison relative to subgroup-specific comparisons. Whether any of these contrasts is causal requires additional assumptions.

A risk difference below zero indicates a lower event risk under treatment than under control. A positive risk difference indicates a higher event risk. We will use this scale throughout, so the direction of each comparison is explicit.

A Numerical Reversal

Consider a hypothetical comparison of a treatment and a control among patients in two baseline risk subgroups. The outcome is an adverse event. The table reports event counts and group sizes; all values are hypothetical.

Subgroup	Treatment: events / total	Treatment risk	Control: events / total	Control risk
Lower baseline risk	5 / 100	0.05	90 / 900	0.10
Higher baseline risk	675 / 900	0.75	80 / 100	0.80
All participants	680 / 1,000	0.68	170 / 1,000	0.17

Worked Example: Finding the Direction of the Reversal

Calculate the treatment-minus-control risk difference within each subgroup and in the combined data. Then compare their directions.

In the lower-risk subgroup, the treatment risk is \(5/100=0.05\), and the control risk is \(90/900=0.10\). Thus the subgroup risk difference is

$$ 0.05-0.10=-0.05. $$

Treatment is associated with a 5-percentage-point lower event risk in this subgroup. In the higher-risk subgroup, the corresponding calculation is

\frac{675}{900}-\frac{80}{100} =0.75-0.80 =-0.05.

Here, too, the treatment risk is 5 percentage points lower. But the crude risks are \(680/1{,}000=0.68\) for treatment and \(170/1{,}000=0.17\) for control, giving

$$ 0.68-0.17=0.51. $$

The crude comparison indicates a 51-percentage-point higher event risk in the treatment group, opposite to both subgroup comparisons. The subgroup mixes differ sharply: 900 of 1,000 treated participants are in the higher-risk subgroup, whereas only 100 of 1,000 control participants are. The overall risks answer a comparison between groups with very different subgroup compositions.

Separating Within-Subgroup Differences From Composition

The arithmetic behind the reversal can be made explicit. Let \(q_a(s)=P(S=s\mid A=a)\) be the proportion in subgroup \(s\) among people with treatment status \(a\). The crude observed risk in group \(a\) is the weighted average

R_a=\sum_s q_a(s)r_a(s).

If the weights \(q_1(s)\) and \(q_0(s)\) differ, the crude risk difference \(R_1-R_0\) is not simply a common weighted average of the subgroup risk differences. One exact decomposition, using the control risks as the reference for the composition component, is

R_1-R_0 = \sum_s q_1(s)\{r_1(s)-r_0(s)\} + \sum_s\{q_1(s)-q_0(s)\}r_0(s).

The first term averages within-subgroup risk differences using the treatment group’s subgroup weights. The second term captures how the crude contrast changes because the subgroup weights differ, with control risks held as the reference. This is an algebraic decomposition, not a unique causal decomposition: choosing a different reference distribution produces a different allocation of the total between components.

Deriving the Decomposition

The identity follows by adding and subtracting \(\sum_s q_1(s)r_0(s)\). Starting from the definitions of the crude risks,

\begin{aligned} R_1-R_0 &=\sum_s q_1(s)r_1(s)-\sum_s q_0(s)r_0(s)\\ &=\sum_s q_1(s)\{r_1(s)-r_0(s)\} +\sum_s q_1(s)r_0(s)-\sum_s q_0(s)r_0(s)\\ &=\sum_s q_1(s)\{r_1(s)-r_0(s)\} +\sum_s\{q_1(s)-q_0(s)\}r_0(s). \end{aligned}

The first line is the crude difference. The second line adds and subtracts the same quantity, \(\sum_s q_1(s)r_0(s)\), so the equality is exact. The final line groups the remaining terms by subgroup.

Worked Example: Checking the Two Components

Use the hypothetical counts above to calculate both components of the decomposition. The treatment group has subgroup proportions \(q_1(\text{lower})=100/1{,}000=0.10\) and \(q_1(\text{higher})=0.90\). The control group has \(q_0(\text{lower})=0.90\) and \(q_0(\text{higher})=0.10\).

The within-subgroup component, weighted by the treatment composition, is

$$ 0.10(-0.05)+0.90(-0.05)=-0.05. $$

For the composition component, use the control risks of 0.10 in the lower-risk subgroup and 0.80 in the higher-risk subgroup:

\begin{aligned} &(0.10-0.90)(0.10)+(0.90-0.10)(0.80)\\ &=(-0.80)(0.10)+(0.80)(0.80)\\ &=-0.08+0.64=0.56. \end{aligned}

Adding the components gives \(-0.05+0.56=0.51\), exactly the crude risk difference. In this example the within-subgroup component favors treatment, while the composition component is sufficiently large in the opposite direction to produce the crude reversal.

Composition Is Not the Same as Causal Effect Heterogeneity

To define subgroup effects causally, let \(Y(1)\) and \(Y(0)\) be the potential outcomes under treatment and control. For a baseline subgroup \(S=s\), the causal risk difference is

\tau(s)=E\{Y(1)-Y(0)\mid S=s\}.

With a binary outcome, this is the difference between the subgroup’s potential-outcome event risks under treatment and control. Causal effect heterogeneity means that \(\tau(s)\) varies across subgroups. By contrast, different values of \(q_1(s)\) and \(q_0(s)\) describe different subgroup composition in the observed treatment groups; they do not themselves show that causal effects vary.

For example, in the first table the observed subgroup risk differences are both \(-0.05\), so they show no variation in risk difference across these two subgroups. The crude contrast is nevertheless \(0.51\), because the groups have different subgroup compositions. If the subgroup comparisons identify causal effects, this pattern is consistent with a constant subgroup causal risk difference alongside a misleading crude comparison.

Identification matters. In a randomized trial, treatment assignment is independent of baseline potential outcomes by design, although finite samples can still have unequal subgroup proportions. With consistency, positivity, and no interference, subgroup comparisons can identify subgroup causal effects under randomization within the subgroups. In an observational study, identifying those effects instead requires, at minimum, conditional exchangeability given the relevant baseline covariates, positivity, and consistency. Crude observed subgroup risks are not automatically causal merely because they are reported separately.

Worked Example: Standardizing to a Common Target Population

Now consider a second hypothetical data set. Suppose its subgroup-specific treatment and control risks identify causal risks under the required design or adjustment assumptions. The lower-risk subgroup has risks 0.18 under treatment and 0.20 under control; the higher-risk subgroup has risks 0.55 and 0.60, respectively. The subgroup causal risk differences are therefore \(-0.02\) and \(-0.05\). The observed treatment group is 10% lower-risk and 90% higher-risk; the control group is 90% lower-risk and 10% higher-risk.

The crude risks are

R_1=0.10(0.18)+0.90(0.55)=0.513, \qquad R_0=0.90(0.20)+0.10(0.60)=0.240.

Thus, the crude treatment-minus-control difference is \(0.513-0.240=0.273\), even though both subgroup differences favor treatment. To compare risks in a common target population with equal subgroup weights, use the same weights, \(q(\text{lower})=q(\text{higher})=0.50\), under both treatment strategies:

\begin{aligned} R_1^{q}&=0.50(0.18)+0.50(0.55)=0.365,\\ R_0^{q}&=0.50(0.20)+0.50(0.60)=0.400,\\ R_1^{q}-R_0^{q}&=0.365-0.400=-0.035. \end{aligned}

The standardized contrast is a 3.5-percentage-point lower event risk under treatment for this equal-weight target population. It differs from the crude contrast because both treatment strategies are evaluated using the same subgroup distribution. The chosen target weights are part of the estimand: a different target population can have a different marginal causal effect when subgroup effects vary.

Worked Example: How the Target Population Changes the Marginal Effect

Continue with the subgroup causal risk differences from the second data set: \(-0.02\) in the lower-risk subgroup and \(-0.05\) in the higher-risk subgroup. For a target population that is 80% lower-risk and 20% higher-risk, the standardized effect is

$$ 0.80(-0.02)+0.20(-0.05) =-0.016-0.010 =-0.026. $$

For a target population that is 20% lower-risk and 80% higher-risk, it is

$$ 0.20(-0.02)+0.80(-0.05) =-0.004-0.040 =-0.044. $$

These are different marginal causal effects: a 2.6-percentage-point reduction and a 4.4-percentage-point reduction, respectively. The subgroup effects have not changed; the target population has. This is why a marginal treatment effect should be accompanied by a clearly defined target population when subgroup effects differ.

What a Reversal Does—and Does Not—Tell You

A reversal is a warning to inspect the subgroup distribution and the estimand, not proof that treatment effects are heterogeneous. Conversely, observing similar subgroup estimates does not establish that effects are identical: estimates may be imprecise, especially in small subgroups. The relevant causal question is how subgroup-specific potential-outcome contrasts vary, not whether a crude observational contrast agrees with each subgroup contrast.

When treatment groups differ in composition, a crude contrast compares risks under two different mixtures of subgroups. Standardization instead specifies a common set of target weights and averages subgroup-specific causal risks or effects using those weights. This connects the subgroup results to a population-level question without confusing changes in the mix of people with changes in treatment response.

Common Mistake

Wrong reasoning: “The treatment effect reverses after stratifying, so the treatment must work differently in the subgroups.”
Correction: A crude-versus-subgroup reversal can arise from different subgroup weights even when subgroup effects are constant. First calculate the within-subgroup contrasts and compare their causal interpretations; then examine the weights and state the target population for any marginal contrast.
Another mistake: Treating every subgroup-specific association as a causal effect. In observational data, stratifying on one subgroup variable does not guarantee control of other confounding. State the assumptions needed to interpret the comparisons causally.

Key takeaway: An aggregate treatment contrast combines subgroup-specific risks with subgroup composition. Simpson’s paradox is a pattern of reversals in comparisons; causal effect heterogeneity is variation in subgroup causal effects. Distinguish them by defining the causal estimand, assessing identification assumptions, and standardizing to an explicit target population when appropriate.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In the first table, which subgroup has the higher event risk in the treatment group, and how does that affect the crude treatment risk?
Explain in words what each term represents in the decomposition of the crude risk difference.
Why does a crude reversal not by itself establish causal effect heterogeneity?
For the second data set, what would the standardized effect be in a target population that is 60% lower-risk and 40% higher-risk?
Name assumptions that are needed to interpret observational subgroup risk differences causally.