Lurking Variables Behind Correlation

When One Scatterplot Hides More Than It Shows

A scatterplot of two quantitative variables can show a clear overall pattern, yet tell an incomplete story. Suppose two groups have different typical values for both variables. Their separation in the plot may create a pooled association even if little or no association appears within either group. The reverse can happen too: a pattern that is clear within each group can weaken or reverse when the groups are combined.

The third variable that helps reveal this structure might be group membership, age category, season, or another characteristic. Looking at the relationship separately within levels of that variable is called stratifying the data. This does not automatically explain why the groups differ, but it can show whether a pooled correlation represents a similar pattern within groups or mainly reflects differences between them.

Definition: A lurking variable is a variable not included in the initial analysis of two measured variables that may be related to one or both of them and may help explain, alter, or conceal their observed association. To stratify data is to divide observations into groups based on a third variable and examine the two-variable relationship separately within each group.

A third variable can play different roles. It may be associated with both measured variables, so that group differences contribute to the pooled pattern. Or the association between the measured variables may genuinely differ across groups. A stratified display helps distinguish these possibilities descriptively; it does not by itself establish a causal explanation.

How Group Differences Affect Correlation

Correlation summarizes the linear association across all paired observations included in its calculation. If those observations come from distinct groups, the overall calculation combines two sources of variation: variation among observations within each group, and variation among the group averages. The second source can reinforce, weaken, or oppose the first.

The decomposition below makes this idea precise. Let group \(g\) contain \(n_g\) observations, with group means \(\bar{x}_g\) and \(\bar{y}_g\). Let \(\bar{x}\) and \(\bar{y}\) be the means across all observations. The total cross-product sum is

\sum_g\sum_i (x_{ig}-\bar{x})(y_{ig}-\bar{y}) = \sum_g\sum_i (x_{ig}-\bar{x}_g)(y_{ig}-\bar{y}_g) + \sum_g n_g(\bar{x}_g-\bar{x})(\bar{y}_g-\bar{y})

The first term on the right measures the combined within-group cross-products. The second measures how the group means vary together. To see why the decomposition holds, write each deviation as \(x_{ig}-\bar{x}=(x_{ig}-\bar{x}_g)+(\bar{x}_g-\bar{x})\), and use the corresponding expression for \(y\). On multiplying, the terms involving one within-group deviation and one group-mean deviation sum to zero: within each group, \(\sum_i(x_{ig}-\bar{x}_g)=0\) and \(\sum_i(y_{ig}-\bar{y}_g)=0\). The remaining terms are exactly the within-group and between-group terms shown above.

Correlation divides the total cross-product sum by the product of the overall sums of squares’ square roots. Those sums of squares also include within-group and between-group variation. Therefore, the decomposition explains how the numerator can change direction, but the pooled correlation is not simply a sum or average of the group correlations. It depends on the spread in both variables as well.

Worked Example: A Positive Pooled Correlation Despite Negative Group Patterns

Consider hypothetical measurements of two quantities, \(X\) and \(Y\), in two groups. The points in Group A are \((1,8),(2,7),(3,6),(4,5)\); those in Group B are \((6,13),(7,12),(8,11),(9,10)\). What happens to the correlation when the groups are combined?

Within Group A, every one-unit increase in \(X\) accompanies a one-unit decrease in \(Y\), so its correlation is \(r=-1\). The same is true within Group B: its correlation is also \(r=-1\). For each group, the means are \((\bar{x}_g,\bar{y}_g)=(2.5,6.5)\) in A and \((7.5,11.5)\) in B. The overall means are \(\bar{x}=5\) and \(\bar{y}=9\).

For each group, the within-group sums of squares are \(5\) for \(X\) and \(5\) for \(Y\), and the within-group cross-product sum is \(-5\). Across both groups, those within-group totals are \(S_{xx,\mathrm{within}}=10\), \(S_{yy,\mathrm{within}}=10\), and \(S_{xy,\mathrm{within}}=-10\). The two group means move together: the group with larger mean \(X\) also has larger mean \(Y\). Their between-group cross-product contribution is

S_{xy,\mathrm{between}} =4(2.5-5)(6.5-9)+4(7.5-5)(11.5-9) =50.

The between-group sums of squares are \(50\) for each variable. Thus \(S_{xy}= -10+50=40\), while \(S_{xx}=10+50=60\) and \(S_{yy}=10+50=60\). The pooled correlation is

r=\frac{S_{xy}}{\sqrt{S_{xx}S_{yy}}} =\frac{40}{\sqrt{60(60)}} =\frac{40}{60} \approx 0.667.

So the combined data have a positive correlation even though the relationship is perfectly negative within each group. The separation between the group averages overwhelms the negative within-group pattern in the pooled calculation. A pooled scatterplot alone could invite a misleading summary of the pattern.

When Pooling Conceals or Reverses a Pattern

Group differences can also work against the within-group pattern. If groups with higher average \(X\) have lower average \(Y\), the between-group contribution to the cross-product sum is negative. It can weaken a positive within-group association, cancel it, or make the pooled correlation negative. This kind of reversal after combining groups is commonly called Simpson’s paradox. The name describes a possible feature of aggregated data, not a guarantee that the group variable is causal or that one particular analysis is always the right one.

Worked Example: Positive Within-Group Patterns, Negative Pooled Correlation

Suppose two hypothetical groups have the following paired values. Group A has \((6,1),(7,2),(8,3),(9,4)\); Group B has \((1,6),(2,7),(3,8),(4,9)\). Within either group, the values increase together, so \(r=1\) in each. Does that imply a positive pooled correlation?

The group means are \((7.5,2.5)\) and \((2.5,7.5)\), and the overall means are \((5,5)\). In each group, \(S_{xx}=5\), \(S_{yy}=5\), and \(S_{xy}=5\). The within-group totals are therefore \(10\), \(10\), and \(10\), respectively.

The group means move in opposite directions: Group A has higher mean \(X\) but lower mean \(Y\). The between-group cross-product contribution is

S_{xy,\mathrm{between}} =4(7.5-5)(2.5-5)+4(2.5-5)(7.5-5) =-50.

The between-group sums of squares are \(50\) for each variable. Hence \(S_{xy}=10-50=-40\), and \(S_{xx}=S_{yy}=10+50=60\). The pooled correlation is

r=\frac{-40}{\sqrt{60(60)}}\approx -0.667.

Thus, both groups have a perfectly positive pattern, but the pooled data have a negative correlation. The opposing group averages dominate the within-group pattern. In another data set, between-group and within-group contributions could cancel, yielding a pooled correlation near zero even when each group has a meaningful pattern.

Worked Example: A Pooled Association Created by Group Separation

In a hypothetical class dataset, \(X\) is weekly study hours and \(Y\) is a quiz score in points. Students belong to either a lower-placement or higher-placement class. The observations are:

Lower-placement class: \((1,49),(1,51),(3,49),(3,51)\). Higher-placement class: \((5,79),(5,81),(7,79),(7,81)\). The example is deliberately constructed to show how class groups can shape a pooled pattern.

In each class, the mean deviations of \(X\) are \((-1,-1,1,1)\), while those of \(Y\) are \((-1,1,-1,1)\). Their cross-products sum to \(1-1-1+1=0\), so the within-class correlation is \(0\) in both groups. Each class has \(S_{xx}=4\) and \(S_{yy}=4\).

Across all eight students, \(\bar{x}=4\) hours and \(\bar{y}=65\) points. The class means are \((2,50)\) and \((6,80)\). The between-class cross-product sum is

S_{xy,\mathrm{between}} =4(2-4)(50-65)+4(6-4)(80-65)=240.

The within-class cross-product sum is zero, so \(S_{xy}=240\). The total sums of squares are \(S_{xx}=8+32=40\) and \(S_{yy}=8+1800=1808\). Therefore,

r=\frac{240}{\sqrt{40(1808)}}\approx 0.892.

The pooled data have a strong positive correlation, even though neither class shows a linear association in these constructed observations. Most of the pooled pattern comes from the difference between class averages. It would be inaccurate to describe this pooled correlation as evidence that students who study more within either class tend to score higher.

Reading Stratified Scatterplots Carefully

A useful display places the same two quantitative variables on the axes and distinguishes groups using separate panels or clear symbols. Compare the pooled view with the within-group views. Ask whether the groups overlap, whether their centers differ, whether the within-group directions resemble one another, and whether one group has much more variability than another.

The comparison helps organize interpretation:

If the pooled pattern is strong but within-group patterns are weak, group separation may account for much of the overall association.
If within-group patterns point in the same direction but the pooled association is weak, between-group differences may be masking them.
If group-specific patterns differ in direction or strength, a single pooled correlation may not describe any group well.
If pooled and within-group patterns are similar, the third variable may be less important to the descriptive summary, though this does not rule out other lurking variables.

These are diagnostic descriptions, not mechanical rules. A few observations, unequal group sizes, or very different spreads can make the visual comparison uncertain. Correlation also summarizes linear association, so a curved within-group pattern may not be captured by a group’s correlation. Always look at the scatterplots themselves rather than relying only on one number.

What Stratification Can and Cannot Establish

A lurking variable is not necessarily measured, and even when a third variable is available, stratifying by it does not prove that it is the complete explanation. For example, age group might help describe an association between two health measurements, but differences between age groups could reflect multiple factors. A stratified display can show that the pooled association changes after separating by age; it cannot, by itself, show what would happen if age were changed or controlled experimentally.

The distinction matters because a correlation is a description of how paired values vary in the data at hand. A third-variable analysis can reveal that the description depends on which observations are combined. It does not turn a correlation into a causal effect. The appropriate conclusion is about the observed data: for instance, “the pooled association is positive, while the within-group patterns are weak,” rather than “group membership caused the association.”

Common Mistake

Wrong reasoning: “The pooled correlation is positive, so the two variables are positively associated for individuals in every group.”
Correction: A pooled correlation combines within-group patterns with differences among group averages. Inspect stratified scatterplots and describe both levels separately; the pooled sign need not match every group’s pattern.
Another wrong reasoning: “Once a third variable is identified, it must be the cause of the original correlation.”
Correction: Stratifying reveals how the observed association varies across groups. It does not establish a causal mechanism or exclude other variables.

Key takeaway: A third variable can create, strengthen, weaken, reverse, or conceal a pooled correlation through the way observations are grouped. Compare the overall scatterplot with the within-group scatterplots before treating one correlation as a summary of the full pattern.

Check Your Understanding

Answer each question by referring to the distinction between within-group and between-group patterns.

In the first worked example, why is the pooled correlation positive even though both within-group correlations are negative?
What does it mean to stratify a scatterplot by a third variable?
Can two groups each have a positive correlation while the pooled correlation is negative? Explain how group means can make this possible.
A pooled correlation is near zero, but separate group plots show positive trends in both groups. Give one way group differences could contribute to this result.
Why does finding a lurking variable not, by itself, establish that the variable caused the observed association?