Clusters and Pooled Correlations

Why Grouping Can Change a Correlation

A scatterplot may contain several clusters, such as measurements from different locations, schools, or treatment groups. Looking at all the points together can produce a correlation that is quite different from the correlations suggested by the separate clusters. The reason is that a pooled correlation reflects both the patterns within groups and the positions of the groups relative to one another.

That distinction matters because a single correlation can conceal important structure. Groups with little linear association individually may form a strong pooled pattern if their centers are separated in the same direction. Conversely, a strong pattern within each group may look weak overall—or even point in the opposite direction—if the group centers are arranged differently.

Here, a within-group correlation is the Pearson correlation calculated using the paired observations in one group. A pooled correlation is the Pearson correlation calculated from all paired observations together, without separating them by group. Both describe linear association, but they answer different descriptive questions.

These are descriptive calculations: no probability model or random-sampling assumption is needed to calculate a correlation for the observed data. For a correlation to be defined, the pooled data must have variation in both variables; a within-group correlation is defined only if that group has variation in both variables. As in any scatterplot analysis, the correlation summarizes linear pattern, so the plots should also be examined.

How Within-Group and Between-Group Variation Combine

The decomposition below makes the pooling effect precise. For group \(g\), let \(n_g\) be its number of observations, \(\bar{x}_g\) and \(\bar{y}_g\) its means, and let \(\bar{x}\) and \(\bar{y}\) be the means across all observations. Define the within-group cross-product sum and squared-deviation sums by

The corresponding pooled quantities use deviations from the overall means. Each can be split into a within-group part and a between-group part:

For example, the cross-product decomposition follows by writing each observation’s deviation from the overall mean as the sum of its deviation from its group mean and its group mean’s deviation from the overall mean:

Multiplying these expressions and summing within group \(g\) gives the within-group cross-product, the between-group term, and two cross terms. The cross terms are zero because deviations from a group mean sum to zero:

Thus the group’s contribution to the pooled cross-product is \(S_{xy,g}+n_g(\bar{x}_g-\bar{x})(\bar{y}_g-\bar{y})\). Summing over groups gives the stated result. The squared-deviation formulas follow by the same expansion with \(x\) paired with itself or \(y\) paired with itself.

The pooled correlation is \(r=S_{xy}/\sqrt{S_{xx}S_{yy}}\). The between-group term in \(S_{xy}\) is positive when group means for \(x\) and \(y\) tend to be high together, and negative when high group means for one tend to accompany low group means for the other. Meanwhile, between-group differences also add to \(S_{xx}\) and \(S_{yy}\), changing the denominator. Therefore, pooled correlation is not generally a simple average of the group correlations.

Reading Grouped Scatterplots Carefully

When a scatterplot shows distinct clusters, inspect both the points within each cluster and the locations of the cluster centers. A useful sequence is to identify the within-group directions, note whether the group means move together or in opposite directions, and then compare those features with the pooled pattern. A pooled correlation alone cannot tell you which component is responsible.

The central practical lesson is not that one calculation is always preferable. The pooled correlation describes the full set of observations, while within-group correlations describe patterns among observations in each group. Which description is useful depends on the question being asked. If group identity is important to understanding the data, report or discuss it alongside the pooled result.

Common Mistake: Treating the Pooled Correlation as a Group-Wise Summary

A common error is to say, “The pooled correlation is negative, so the relationship is negative in every group.” The second worked example shows why that reasoning fails: the pooled correlation is about \(-0.8881\), but both groups have correlation \(1\). The pooled sign reflects the combined within-group and between-group contributions, not necessarily the direction in any one group.

The correction is to keep the levels of description separate. State whether a value is pooled or within-group, inspect the grouped scatterplot, and avoid using the pooled value to make claims about patterns that were not calculated within the groups. Similarly, do not assume that similar within-group correlations guarantee a similar pooled correlation: group locations can substantially change the pooled result.