Simpson's Paradox in Grouped Data

When the Overall Pattern Reverses

An overall comparison can tell a different story from comparisons made separately within relevant groups. For example, an outcome may be higher at one predictor level in each group, yet lower at that predictor level after all groups are pooled. This is not an arithmetic contradiction. The pooled comparison combines observations using the group proportions present at each predictor level, and those proportions may differ sharply.

This pattern is called Simpson’s paradox. It is important in regression analysis because an overall association and an association adjusted for a group variable need not have the same direction. The reversal is a feature of how the data are distributed across groups; by itself, it does not show that one comparison is correct or that either comparison describes a causal effect.

To make the comparison precise, let \(X\) be a binary predictor with values 0 and 1, let \(Y\) be a quantitative outcome, and let \(G\) identify a group. For a group \(g\), the within-group difference is \(\bar{y}_{1g}-\bar{y}_{0g}\), where \(\bar{y}_{1g}\) and \(\bar{y}_{0g}\) are the sample mean outcomes at \(X=1\) and \(X=0\) in that group. The pooled difference is \(\bar{y}_{1}-\bar{y}_{0}\), calculated without separating the groups.

When \(X\) is coded 0 and 1, the slope from a least-squares regression of \(Y\) on \(X\), with an intercept, equals the difference in the two sample means: \(\hat{\beta}_1=\bar{y}_{1}-\bar{y}_{0}\). The fitted value at \(X=0\) is the mean for observations with \(X=0\), and the fitted value at \(X=1\) is the mean for observations with \(X=1\); their difference is the slope. Thus, in this setting, comparing pooled means is also comparing the direction of the unadjusted regression slope.

A Worked Reversal in Pooled Means

Why the Group Proportions Matter

A pooled mean is a weighted average of the group means. The weights are the proportions of observations from each group at that predictor level. If those weights differ between \(X=0\) and \(X=1\), the two pooled means mix group outcomes differently. As a result, the pooled difference includes both changes within groups and changes in group composition.

The following identity makes that statement exact. Write \(w_{xg}\) for the proportion of observations at predictor level \(x\) that belong to group \(g\). The weights at each level add to 1, and the pooled mean at that level is \(\bar{y}_x=\sum_g w_{xg}\bar{y}_{xg}\). Provided each group is observed at both predictor levels, add and subtract \(\sum_g w_{1g}\bar{y}_{0g}\) from the pooled difference:

The first sum is a weighted average of the within-group differences, using the group proportions at \(X=1\) as weights. The second sum is a composition term: it reflects how the group proportions change between \(X=0\) and \(X=1\), combined with the groups’ mean outcomes at \(X=0\). Either term can influence the pooled result. Even if every within-group difference is positive, a sufficiently negative composition term can make the pooled difference negative.

In the score example, the lower- and higher-preparation proportions are \(0.10\) and \(0.90\) at \(X=0\), but \(0.90\) and \(0.10\) at \(X=1\). The weighted within-group component is:

The composition component, using the \(X=0\) means, is:

Adding the two components gives \(2+(-16)=-14\) points, exactly the pooled difference. The positive within-group pattern is more than offset by the fact that the \(X=1\) observations are concentrated in the group with lower scores.

Worked Example: Reversal in Success Rates

A Reversal Is Possible, Not Inevitable

Different group proportions can produce a reversal, but they do not guarantee one. To see the distinction, retain the two preparation groups and the four mean scores from the first example, but suppose each group contributes 50 students at each study-plan level. The group means remain 10 and 12 points for the lower-preparation group and 30 and 32 points for the higher-preparation group.

How to Read a Pooled Association Carefully

A reversal is a signal to inspect how the data are distributed, not a reason to discard one summary automatically. The pooled association describes the observed combined groups. The within-group associations describe comparisons among observations sharing the specified group category. These answer different descriptive questions because they use different comparisons and, in general, different weights.

In regression analysis, including a group variable as a predictor can express comparisons that hold group membership fixed, whereas a regression using only \(X\) describes the pooled relationship. The precise adjusted comparison depends on the model and data; the arithmetic examples here do not establish that every possible adjustment will produce a particular single summary. In particular, a group-specific pattern should not be assumed to be constant in size merely because its direction agrees across groups.

A group variable may be important for understanding the observed association, but a reversal alone does not establish why the predictor and group are related, whether the group is an appropriate adjustment variable, or whether any association is causal. Those conclusions require information and assumptions beyond the pooled and grouped means or rates. Describe what the data show and identify the group composition that contributes to the discrepancy.

Common Mistake

• Wrong reasoning: “The pooled mean is lower at \(X=1\), so the outcome must also be lower at \(X=1\) within the groups.” This treats a pooled average as though it used the same group proportions as each within-group comparison.
• Correction: Calculate the group-specific means or rates and compare the group proportions at each predictor level. The overall comparison can reverse when it combines groups in different proportions.
• Another overreach: “Because the direction changes after grouping, the group variable must be the cause of the reversal.” A composition difference can explain the arithmetic pattern, but a descriptive reversal by itself does not establish a causal explanation.