Why a Pooled Comparison Can Point the Wrong Way
When observations come from different clinics, schools, or other clusters, a pooled comparison combines two features: the outcome rates within each cluster and the proportions of each comparison group drawn from each cluster. If those proportions differ substantially, the pooled contrast can have the opposite direction from every cluster-specific contrast. This is a Simpson reversal.
The reversal is an arithmetic property of aggregation, not a special kind of outcome or model. It is important in clustered studies because clusters can differ in baseline outcome rates, while the groups being compared can be represented in those clusters in very different proportions. A pooled rate then answers a different weighted question from a comparison that gives clusters common weights.
For cluster \(k\) and group \(a\), let \(n_{ak}\) be the number of observations and \(y_{ak}\) the number with the outcome of interest. Define the observed rate \(r_{ak}=y_{ak}/n_{ak}\), and let \(w_{ak}=n_{ak}/\sum_k n_{ak}\) be that group’s share from cluster \(k\). The pooled rate for group \(a\) is a weighted average:
The key is that the weights \(w_{ak}\) may differ between groups. Thus, even if \(r_{1k}<r_{0k}\) for every cluster, it does not follow that \(R_1<R_0\). A pooled rate weights each cluster according to its representation in that group, not according to a common cluster distribution.
A Reversal From Unequal Clinic Composition
Worked Example: Lower Recovery Within Both Clinics, Higher Recovery Overall
Suppose, hypothetically, two clinics compare recovery after an intervention with recovery under usual care. The outcome is recovery by a fixed follow-up date. The table gives the number recovered and the number observed in each group.
| Clinic | Intervention recovered / total | Usual care recovered / total |
|---|---|---|
| A | 80 / 100 | 9 / 10 |
| B | 18 / 90 | 27 / 90 |
| Total | 98 / 190 | 36 / 100 |
Compare within each clinic. In clinic A, the intervention recovery rate is \(80/100=0.80\), or 80%, compared with \(9/10=0.90\), or 90%, under usual care. The within-clinic difference is \(0.80-0.90=-0.10\), a difference of −10 percentage points. In clinic B, the rates are \(18/90=0.20\) and \(27/90=0.30\), respectively, again a difference of −10 percentage points. The intervention group has a lower observed recovery rate in each clinic.
Now pool the observations. The intervention rate is \(98/190=0.5158\), or about 51.6%. The usual-care rate is \(36/100=0.36\), or 36.0%. The pooled difference is \(0.5158-0.3600=0.1558\), or about 15.6 percentage points in favor of the intervention group. The direction has reversed.
Locate the composition difference. Clinic A contributes \(100/190=52.6\%\) of the intervention group, but only \(10/100=10.0\%\) of the usual-care group. Clinic B contributes the remaining 47.4% of the intervention group and 90.0% of the usual-care group. Clinic A has much higher recovery rates than clinic B for both groups, and the intervention group contains far more observations from clinic A. The pooled comparison therefore combines the group contrast with a large difference in clinic composition.
Why the Weights Matter
The role of composition can be made explicit by adding and subtracting a weighted rate. One valid decomposition of the pooled difference uses group 1’s weights for the within-cluster differences and group 0’s rates for the composition component:
This identity follows by expanding the right-hand side. The first sum becomes \(\sum_k w_{1k}r_{1k}-\sum_k w_{1k}r_{0k}\); the second is \(\sum_k w_{1k}r_{0k}-\sum_k w_{0k}r_{0k}\). The middle terms cancel, leaving \(R_1-R_0\). The first component reflects within-cluster rate differences under group 1’s weights. The second reflects the change in weights, evaluated using group 0’s rates. Other decompositions are possible, but each makes clear that pooled contrasts combine within-cluster rates and cluster weights.
Worked Example: Decomposing the Reversed Difference
Use the clinic data above. For the intervention group, the weights are \(w_{1A}=100/190=0.5263\) and \(w_{1B}=90/190=0.4737\). For usual care, they are \(w_{0A}=10/100=0.1000\) and \(w_{0B}=90/100=0.9000\). The within-clinic differences are both −0.10.
The first component of the decomposition is \(0.5263(-0.10)+0.4737(-0.10)=-0.1000\). The composition component is \((0.5263-0.1000)(0.90)+(0.4737-0.9000)(0.30)\) \(=(0.4263)(0.90)+(-0.4263)(0.30)=0.2558\).
Adding the components gives \(-0.1000+0.2558=0.1558\), the pooled difference of about 15.6 percentage points. In this decomposition, the negative within-clinic component is outweighed by the positive component associated with the groups’ different clinic weights. The calculation describes the source of the arithmetic reversal; it does not establish why the groups had different clinic compositions.
Common Weights Answer a Different Question
A common-weight comparison assigns both groups the same distribution across clusters. It removes the difference in cluster composition from the descriptive contrast, while retaining the observed rate in each group within each cluster. The choice of common weights defines the target population for that standardized comparison: equal weights target an equally weighted average of the included clusters, while weights based on a specified population target that population’s cluster distribution.
Worked Example: Standardizing to Equal Clinic Weights
For these two clinics, assign each clinic weight \(0.50\) in both groups. The standardized intervention rate is \(0.50(0.80)+0.50(0.20)=0.50\), or 50.0%. The standardized usual-care rate is \(0.50(0.90)+0.50(0.30)=0.60\), or 60.0%. The standardized difference is \(0.50-0.60=-0.10\), or −10 percentage points.
This agrees with the two within-clinic differences because those differences are both −10 percentage points. It differs from the crude pooled difference of +15.6 percentage points because the crude rates use different clinic weights for the two groups. The standardized result is not automatically the uniquely correct summary: its meaning depends on whether equal clinic weights match the question of interest.
As another possible target, use the distribution of all 290 observations: clinic A has weight \(110/290=0.3793\), and clinic B has weight \(180/290=0.6207\). The standardized intervention rate is \(0.3793(0.80)+0.6207(0.20)=0.4276\). The standardized usual-care rate is \(0.3793(0.90)+0.6207(0.30)=0.5276\). Their difference is again −0.1000. Here the result is the same because the clinic-specific differences are equal; with varying differences, the choice of common weights can change the standardized contrast.
What a Reversal Does—and Does Not—Show
A Simpson reversal is a warning to report the level of comparison and the weighting scheme. It does not, by itself, say that the pooled comparison is erroneous or that the within-cluster comparison is the correct answer. Those summaries answer different questions. The crude pooled rate describes the observed group-specific mixtures. A standardized contrast describes a comparison after both groups have been assigned a common cluster distribution. The earlier material on effect modification and population averaging is relevant: when cluster-specific contrasts vary, the population-averaged contrast depends on how clusters are weighted.
Nor does the reversal alone establish a causal effect. A within-cluster comparison has a causal interpretation only under appropriate design and assumptions, such as adequate control of relevant differences between groups within clusters, a well-defined intervention, and suitable outcome ascertainment. If assignment was not randomized, unmeasured differences can remain within clinics. A crude association, a standardized association, and a causal effect should not be treated as interchangeable.
Clustered observations may also be correlated within a clinic. That dependence affects the uncertainty calculation, not the arithmetic identity \(R_a=\sum_k w_{ak}r_{ak}\). A standard error that treats all observations as independent can therefore give misleading inference. GEE or a GLMM can represent clustering under their respective assumptions, but choosing a model does not remove the need to define the target contrast and its weights. A conditional model-based comparison and a population-averaged comparison need not target the same quantity.
Diagnostics and Common Failure Modes
- Compare directions before pooling. Tabulate the group-specific outcome rates within clusters. If every within-cluster difference points one way but the pooled difference points the other, the reversal is directly visible.
- Inspect denominators and weights. Report \(n_{ak}\) or the percentages of each group from each cluster, not only the rates. A reversal is plausible when cluster composition differs and cluster outcome rates differ.
- Check for overlap. A within-cluster comparison requires observations from both groups in the cluster. If a cluster has only one group, its group contrast is not observed there; a model-based estimate would require additional assumptions rather than direct within-cluster evidence.
- State the target population. A common-weight estimate is incomplete without saying how the common weights were chosen. Different target distributions can yield different averages when cluster-specific effects vary.
- Do not equate a small cluster count with negligible influence. A cluster’s contribution depends on its rate and weight, not simply on the number of clusters. Conversely, a very large cluster may dominate an observation-weighted pooled rate.
Worked Example: No Within-Clinic Comparison Is Available
Suppose, hypothetically, clinic A reports 70 recoveries among 100 intervention patients and has no usual-care patients. Clinic B reports 40 recoveries among 100 usual-care patients and has no intervention patients. The crude rates are \(70/100=70\%\) for intervention and \(40/100=40\%\) for usual care, a difference of \(70\%-40\%=30\) percentage points.
But clinic A contributes 100% of the intervention group and clinic B contributes 100% of the usual-care group. There is no clinic in which both groups are observed, so neither within-clinic difference can be calculated from these data. The pooled difference cannot distinguish an intervention association from differences between the clinics. This is a lack of within-cluster overlap, not evidence that a Simpson reversal has or has not occurred.
Choosing What to Report
The main alternatives are not merely competing calculations; they correspond to different summaries. The pooled comparison uses each group’s observed cluster mix. A stratified presentation reports the cluster-specific rates and contrasts directly. Standardization uses a shared set of weights to summarize those cluster-specific rates for a defined target mix. A GLMM or GEE may provide model-based estimates that account for clustering, but its reported contrast should be interpreted according to the model and estimand, not assumed to equal every crude or standardized quantity.
When cluster-specific effects differ, one number can conceal important variation even if no reversal occurs. When cluster effects are similar, the choice of common weights may have little effect on a standardized difference, but crude group rates can still differ if group composition is unequal and baseline rates vary. The useful practice is to show the cluster-specific evidence, state the weighting behind any summary, and use an inferential method that respects dependence and the study design.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- If group 1 has a lower rate than group 0 in every cluster, must its pooled rate be lower? Why or why not?
- In the clinic example, what were the pooled intervention and usual-care recovery rates, rounded to one decimal place?
- Why did equal clinic weights produce a different contrast from pooling the observed groups?
- What can be concluded about within-clinic differences if one group is absent from every clinic in which the other group is observed?
- Does a Simpson reversal by itself establish that one group caused a better outcome? Explain.
Solutions. (1) No. The groups can have different cluster weights, and clusters can have different baseline rates, so the weighted averages can reverse the within-cluster direction. (2) \(98/190=51.6\%\) and \(36/100=36.0\%\). (3) Equal weighting assigned both groups the same clinic distribution, whereas the crude rates used the groups’ different observed clinic compositions. (4) The within-clinic differences cannot be calculated from those data; a pooled contrast mixes group and clinic differences. (5) No. A reversal is descriptive arithmetic and does not establish causal identification or control of confounding.