Why a Pooled Comparison Needs a Check
A pooled proportion combines observations from several subgroups. That single number can give a different impression from the comparisons made separately within those subgroups. To diagnose a possible Simpson reversal, compare the groups within every stratum, then compare those results with the pooled proportions. The subgroup sizes matter because they determine how much each stratum contributes to each pooled result.
For example, suppose two treatments are compared across patients in two risk categories. A pooled success rate reflects both the treatments' success rates within each category and the proportion of each treatment group that falls into each category. If the groups have very different compositions, the pooled rates may reverse the within-category comparison. This is a feature of how the table is aggregated; by itself, it does not establish why the groups differ or support a causal conclusion.
For a comparison group \(G\) and stratum \(j\), let \(x_{Gj}\) be the number with the outcome and \(n_{Gj}\) the group total in that stratum. The within-stratum outcome proportion is \(\hat p_{Gj}=x_{Gj}/n_{Gj}\). The pooled proportion for group \(G\) is the total number with the outcome divided by the group's total across all strata. These proportions are defined only when the relevant group totals are positive.
A Simpson reversal occurs when the within-stratum comparisons all point in one direction, but the pooled comparison points in the opposite direction. For instance, group A has a higher outcome proportion than group B in every stratum, while group A has a lower pooled outcome proportion. A change in the size of a difference is not, on its own, a reversal.
A Systematic Table Diagnosis
Use the same outcome and comparison groups in every stratum. For each subgroup, choose the correct denominator: the group total within that stratum. Then calculate the pooled proportions using the corresponding totals across all strata. Reading the outcome counts alone is not enough, because a larger count may simply reflect a larger group.
For each stratum and comparison group, divide the outcome count by that group's total in the stratum.
State which group has the higher proportion in each stratum. Check whether all strata agree.
Add outcome counts across strata within each group, and divide by that group's total across strata.
A genuine Simpson reversal requires a consistent within-stratum direction that opposes the pooled direction. Compare the stratum weights for the groups to see whether their compositions differ.
The weight of stratum \(j\) in group \(G\)'s pooled proportion is \(n_{Gj}/n_G\), where \(n_G\) is that group's total across strata. Thus the pooled proportion is a weighted average of the within-stratum proportions. The weights can differ between groups; that difference is central to diagnosing a reversal.
Worked Example: The Within-Stratum Directions Do Not Agree
Consider this hypothetical table of outcomes in two groups and two strata. The outcome and no-outcome columns contain participant counts.
| Stratum | Group | Outcome | No outcome | Total |
|---|---|---|---|---|
| Lower | A | 80 | 120 | 200 |
| Lower | B | 6 | 14 | 20 |
| Higher | A | 15 | 5 | 20 |
| Higher | B | 160 | 40 | 200 |
In the lower stratum, the outcome proportions are \(80/200=0.400\) for A and \(6/20=0.300\) for B, so A's proportion is higher by 0.100, or 10.0 percentage points. In the higher stratum, they are \(15/20=0.750\) for A and \(160/200=0.800\) for B, so B's proportion is higher by 0.0500, or 5.00 percentage points.
The pooled totals for A are 95 with the outcome and \(200+20=220\) participants. For B, they are 166 with the outcome and \(20+200=220\) participants. Therefore,
The pooled proportion is higher for B. But the within-stratum directions disagree: A is higher in the lower stratum, while B is higher in the higher stratum. Therefore, this table does not show a strict Simpson reversal, which requires the within-stratum comparisons to point consistently in the opposite direction from the pooled comparison.
The group compositions differ sharply. In A, \(200/220=0.909\) of participants are in the lower stratum; in B, only \(20/220=0.091\) are. This composition helps describe why pooling produces the displayed overall rates, but it does not turn the mixed within-stratum pattern into a Simpson reversal.
A Reversal Explained by Different Stratum Weights
The next hypothetical table has the defining pattern: A has a higher outcome proportion in each stratum, yet its pooled proportion is lower. Here the strata are called lower risk and higher risk.
Worked Example: A Genuine Simpson Reversal
| Stratum | Group | Outcome | No outcome | Total |
|---|---|---|---|---|
| Lower risk | A | 9 | 1 | 10 |
| Lower risk | B | 80 | 20 | 100 |
| Higher risk | A | 20 | 80 | 100 |
| Higher risk | B | 1 | 9 | 10 |
Within the lower-risk stratum, A's proportion is \(9/10=0.900\), while B's is \(80/100=0.800\). A is higher by 0.100, or 10.0 percentage points. Within the higher-risk stratum, A's proportion is \(20/100=0.200\), while B's is \(1/10=0.100\). Again, A is higher by 0.100, or 10.0 percentage points. The within-stratum direction is consistent: A has the higher proportion in both strata.
Now pool the strata separately within each group. A has \(9+20=29\) outcomes among \(10+100=110\) observations; B has \(80+1=81\) outcomes among \(100+10=110\) observations. Thus,
The pooled proportion is higher for B, even though A is higher in both strata. This is a Simpson reversal. The subgroup sizes help explain the arithmetic: 100 of A's 110 observations are in the higher-risk stratum, where both groups have lower outcome proportions. By contrast, 100 of B's 110 observations are in the lower-risk stratum, where both groups have higher proportions.
Equivalently, the lower-risk stratum makes up \(10/110=0.091\) of A but \(100/110=0.909\) of B. The higher-risk stratum makes up \(0.909\) of A and \(0.091\) of B. The pooled figures give much more weight to the lower within-stratum rate for A and to the higher within-stratum rate for B.
Why Different Weights Can Reverse the Pooled Direction
The weighted-average formula makes the diagnosis precise. For group \(G\), each stratum's weight is its share of that group's total. Since the stratum totals add to the group total, the weights sum to one. The pooled outcome proportion is the sum of each within-stratum proportion multiplied by its group-specific weight:
For the reversal example, A's weights are \(10/110=0.091\) for lower risk and \(100/110=0.909\) for higher risk. Its pooled rate is therefore
B's weights are \(100/110=0.909\) for lower risk and \(10/110=0.091\) for higher risk. Its pooled rate is
The within-stratum advantage for A is not contradicted by these calculations. The pooled comparison answers a different descriptive question because its weights differ between A and B.
A useful special case clarifies the role of composition. If the two groups have the same stratum weights \(w_j\), then their pooled difference is
If A's proportion is higher in every stratum, then each difference \(\hat p_{Aj}-\hat p_{Bj}\) is positive. A weighted sum of positive differences with nonnegative weights that sum to one is positive, so the pooled proportion for A must also be higher. With identical stratum weights, a consistent within-stratum direction cannot reverse. Different weights allow the pooled direction to differ.
When Unequal Composition Does Not Cause a Reversal
Different subgroup sizes make a reversal possible, but they do not guarantee one. The within-stratum rates and the size of the weight differences also matter. The next table illustrates unequal composition without a reversal.
Worked Example: Different Composition, Same Direction
| Stratum | Group | Outcome | No outcome | Total |
|---|---|---|---|---|
| Lower risk | A | 81 | 9 | 90 |
| Lower risk | B | 8 | 2 | 10 |
| Higher risk | A | 2 | 8 | 10 |
| Higher risk | B | 9 | 81 | 90 |
In the lower-risk stratum, A's proportion is \(81/90=0.900\) and B's is \(8/10=0.800\). In the higher-risk stratum, A's proportion is \(2/10=0.200\) and B's is \(9/90=0.100\). A is higher in both.
The pooled proportions are \(83/100=0.830\) for A and \(17/100=0.170\) for B. Here A has more observations in the lower-risk stratum: \(90/100=0.900\), compared with \(10/100=0.100\) for B. This composition favors A in the pooled comparison, so unequal weights reinforce rather than reverse the within-stratum direction. The table has unequal composition, but it is not a Simpson reversal.
Common Mistakes and Practical Limits
- Calling every change in the pooled comparison a Simpson reversal. A reversal requires one consistent direction within every stratum and the opposite pooled direction. If subgroup directions disagree, describe that mixed pattern instead.
- Comparing outcome counts instead of proportions. Counts depend on group size. Use the outcome count divided by the relevant group total for each comparison.
- Checking only one stratum. A reversal claim requires examining every listed stratum with defined proportions. A direction in one subgroup cannot establish a consistent pattern.
- Assuming different sizes must produce a reversal. Unequal weights can affect the pooled comparison, but as the third example shows, they may reinforce the within-stratum direction.
- Treating the descriptive pattern as a causal explanation. The table shows how pooled and subgroup proportions compare. It does not, by itself, explain why the groups have different compositions or establish that group membership caused the outcome.
When a table suggests a reversal, report both levels of comparison: the within-stratum proportions and directions, and the pooled proportions and direction. Also state the subgroup sizes or weights. This makes it possible to distinguish a genuine reversal from a mixed subgroup pattern, and to see how much each stratum contributes to the pooled figures.
Check Your Understanding
Use the proportions and group totals—not outcome counts alone—to answer these questions.
- Group A has outcome proportions of 0.60 and 0.20 in two strata; group B has proportions of 0.50 and 0.10. What within-stratum pattern is present, and what additional comparison is needed to diagnose a Simpson reversal?
- What must be true about the within-stratum directions and pooled direction for a comparison to qualify as a Simpson reversal?
- In the genuine reversal example, what fraction of group A is in the higher-risk stratum, and what fraction of group B is in that stratum?
- If the two groups have the same stratum weights and A's outcome proportion is higher in every stratum, can the pooled proportion for A be lower? Explain using the weighted-difference formula.
- Why is a table with conflicting within-stratum directions not a strict Simpson reversal, even if its pooled direction differs from one of its subgroup directions?