Correlation Can Hide Distinct Scatterplot Patterns

Why a Correlation Needs a Scatterplot

Pearson correlation is a compact numerical summary of the direction and strength of linear association between two quantitative variables. That compactness is useful: one number can summarize a large set of paired observations. But it also means that different point patterns can receive the same correlation. Curvature, separate clusters, and influential observations may be hard to detect from the number alone.

In predictive analytics, this matters because the shape of the relationship can affect whether a model is appropriate and how its predictions should be interpreted. A correlation value is not a substitute for looking at the paired data. The scatterplot supplies information about form and structure that the summary does not retain.

Definition: For $n$ paired observations $(x_i,y_i)$, the sample Pearson correlation is $$ r=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})} {\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2\sum_{i=1}^{n}(y_i-\bar{y})^2}}, $$ where $\bar{x}$ and $\bar{y}$ are the sample means. It is defined when both variables have nonzero sample variation. Its value lies between $-1$ and $1$; the sign describes the direction of linear association, while the magnitude describes how closely the points follow a straight-line pattern.

This is a descriptive statistic calculated from paired observations; it does not require a particular probability distribution. It is not, by itself, a measure of every kind of association. A value near zero says that the observations have little linear association as summarized by this formula. It does not establish that the variables are unrelated. In particular, a strong curved relationship can have correlation zero.

Same Correlation, Different Curves

A direct way to see what the summary leaves out is to compare two data sets with exactly the same correlation but different shapes. The following example uses small, hypothetical data sets so each calculation can be checked by hand.

Worked Example: Two Curves With Zero Correlation

Two hypothetical portfolios are compared using a predictor $x$ and a measured response $y$. For each portfolio, the five predictor values are $-2,-1,0,1,2$. Portfolio A has response values $4,1,0,1,4$; Portfolio B has response values $0,3,4,3,0$. Calculate each correlation and describe what the number misses.

Method. Apply the Pearson formula to each set of paired values, using the same predictor values in both. The common mistake to avoid is interpreting a zero correlation as evidence that there is no relationship.

For both sets, $\bar{x}=(-2-1+0+1+2)/5=0$. In Portfolio A, $\bar{y}=(4+1+0+1+4)/5=2$. The centered response values are $2,-1,-2,-1,2$. Thus the sum of cross-products is $ (-2)(2)+(-1)(-1)+(0)(-2)+(1)(-1)+(2)(2)=-4+1+0-1+4=0$.

The sum of squared centered predictor values is $4+1+0+1+4=10$. The sum of squared centered response values is $4+1+4+1+4=14$. Therefore $r_A=0/\sqrt{10(14)}=0.0000$.

For Portfolio B, $\bar{y}=(0+3+4+3+0)/5=2$, so the centered response values are $-2,1,2,1,-2$. The cross-product sum is $ (-2)(-2)+(-1)(1)+(0)(2)+(1)(1)+(2)(-2)=4-1+0+1-4=0$. The squared-deviation sums are again 10 for $x$ and 14 for $y$, so $r_B=0/\sqrt{10(14)}=0.0000$.

Check. In both cases, the cross-products cancel and the denominator is positive, so the calculated correlation is zero. But Portfolio A has a U-shaped pattern: $y$ is low near $x=0$ and higher toward either end. Portfolio B has the reverse, an inverted-U pattern. The same value of $r$ describes neither curve adequately.

Key takeaway: Pearson correlation summarizes straight-line association. When the scatterplot is curved, the value of $r$ can be small even though knowing $x$ helps describe $y$.

How Grouping Can Change the Story

A pooled correlation can also hide different relationships within groups. If observations form distinct clusters, their separation may dominate the overall calculation. A large pooled correlation can then suggest an overall direction that does not describe the pattern inside either group. Grouped symbols on a scatterplot can help reveal this structure.

Worked Example: A Strong Pooled Correlation Hides Within-Group Patterns

A hypothetical analyst combines measurements from two regions. The paired observations are $(-6,-4),(-4,-6)$ in Region 1 and $(4,6),(6,4)$ in Region 2. Calculate the pooled Pearson correlation and compare it with the direction within each region.

Method. First calculate the pooled correlation from all four observations. Then inspect the pairs within each region. The common mistake to avoid is assuming that a strong pooled correlation must describe the pattern within every subgroup.

The $x$-values sum to $-6-4+4+6=0$, and the $y$-values sum to $-4-6+6+4=0$. Hence $\bar{x}=\bar{y}=0$, so the observations are already centered. The cross-product sum is $ (-6)(-4)+(-4)(-6)+(4)(6)+(6)(4)=24+24+24+24=96$.

The sum of squared $x$-values is $36+16+16+36=104$. The sum of squared $y$-values is $16+36+36+16=104$. Therefore $r=96/\sqrt{104(104)}=96/104=12/13\approx0.9231$.

Check. The positive pooled correlation is large. Within Region 1, $x$ increases from $-6$ to $-4$ while $y$ decreases from $-4$ to $-6$. Within Region 2, $x$ increases from $4$ to $6$ while $y$ decreases from $6$ to $4$. Each region has a negative within-group direction, while the pooled association is strongly positive. The separation between regions drives the pooled result.

In a scatterplot, the points appear as two separated groups, each sloping downward. If the region labels are not shown, the pooled correlation does not disclose that structure. The analyst should not discard the pooled calculation; rather, they should describe what it measures and also examine the groups.

This example illustrates why context and grouping matter when interpreting an association. A correlation calculated across all observations is a summary for those observations as combined. It does not automatically describe a subgroup, establish a causal effect, or indicate that the same pattern applies throughout the predictor range.

Why the Formula Measures Linear Association

The numerator of Pearson correlation is the sum of products of centered deviations. When observations tend to be above their $x$-mean at the same time as they are above their $y$-mean, these products tend to be positive. When one variable tends to be above its mean while the other is below, products tend to be negative. Positive and negative products can also cancel, as they did in the curved examples.

The denominator scales the cross-product sum by the variation in each variable. This makes the measure independent of the units in which $x$ and $y$ are recorded. The cancellation of the sample-size factors can be seen from the covariance and standard deviations.

Justification: Let $s_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})/(n-1)$, $s_x=\sqrt{\sum (x_i-\bar{x})^2/(n-1)}$, and $s_y=\sqrt{\sum (y_i-\bar{y})^2/(n-1)}$. The standardized covariance is $$ \frac{s_{xy}}{s_xs_y} = \frac{\sum (x_i-\bar{x})(y_i-\bar{y})/(n-1)} {\sqrt{\sum (x_i-\bar{x})^2/(n-1)}\sqrt{\sum (y_i-\bar{y})^2/(n-1)}} = \frac{\sum (x_i-\bar{x})(y_i-\bar{y})} {\sqrt{\sum (x_i-\bar{x})^2\sum (y_i-\bar{y})^2}} =r. $$ The factors $1/(n-1)$ cancel. Thus $r$ is the covariance after scaling both variables to their standard-deviation units. This explains why changing measurement units does not change $r$, but it does not make $r$ a complete description of the scatterplot.

Influential Observations Can Change the Correlation

Correlation is also sensitive to individual observations, especially points far from the center of the predictor values. Such a point can substantially alter the sums in both the numerator and denominator. A high correlation calculated with the point included does not prove that the point is valid; nor does a changed correlation prove that it is an error. The observation should be checked in context before any decision about its treatment.

Worked Example: One Point Reverses the Correlation

Three hypothetical branches have predictor-response pairs $(1,1),(2,2),(3,3)$. A fourth branch, with pair $(10,0)$, is added. Calculate the correlation before and after adding the fourth branch, and explain why the change merits inspection.

Method. Use the Pearson formula on the first three pairs, then recalculate it using all four. The common mistake to avoid is deleting the fourth observation solely because it changes the result.

For the first three pairs, $y=x$, so each point lies exactly on a rising straight line. To verify the correlation directly, $\bar{x}=\bar{y}=2$. Both squared-deviation sums are $(1-2)^2+(2-2)^2+(3-2)^2=2$, and the cross-product sum is also 2. Thus $r=2/\sqrt{2(2)}=1.0000$.

With the fourth pair included, $\bar{x}=(1+2+3+10)/4=4$ and $\bar{y}=(1+2+3+0)/4=1.5$. The centered $x$-values are $-3,-2,-1,6$; the centered $y$-values are $-0.5,0.5,1.5,-1.5$. The cross-product sum is $1.5-1-1.5-9=-10$. The squared-deviation sums are $9+4+1+36=50$ for $x$, and $0.25+0.25+2.25+2.25=5$ for $y$. Therefore $r=-10/\sqrt{50(5)}=-10/\sqrt{250}\approx-0.6325$.

Check. The result changes from $1.0000$ to approximately $-0.6325$, including a reversal of sign. The fourth point has an unusually large $x$-value relative to the first three and a low $y$-value, so it strongly affects the calculation. The appropriate next step is to inspect the observation and its context, then report any modeling decision transparently—not to remove it automatically.

A Practical Visual Check

A scatterplot and its correlation answer different questions. The number summarizes linear association; the plot reveals the arrangement of observations. When interpreting a correlation, check whether the plot suggests a curve, separated clusters, a point far from the main body, or a range with little data. These features can change what a pooled summary means and whether a simple linear description is plausible.

In R, cor(x, y) calculates Pearson correlation for two numeric vectors when the observations are paired in the same order. A scatterplot can be drawn with plot(x, y). The correlation should be interpreted alongside that plot, with the same set of paired observations used for both. If values are missing, be deliberate about which pairs are included; comparing a plot and a correlation based on different rows can lead to inconsistent conclusions.

The correlation is not a test of whether a relationship is causal, and a value near zero does not show that no predictive pattern exists. Conversely, a large magnitude does not guarantee that a linear model gives an adequate account of every group or region. These are descriptive conclusions: further modeling decisions require attention to the shape and context of the data.

Exam-ready habit: State what the correlation summarizes, then describe what the scatterplot reveals that the number does not. In particular, check for curvature, clusters, and influential points before treating a single value of $r$ as the full story.

Check Your Understanding

For each problem, calculate or interpret the correlation as requested, then consider what the scatterplot would add.

For $x=(-1,0,1)$ and $y=(3,2,1)$, calculate Pearson correlation. What direction does it describe? (Answer: $r=-1.0000$, a perfectly decreasing linear pattern.)
For $x=(-2,-1,0,1,2)$ and $y=(4,1,0,1,4)$, calculate the correlation and name the visible pattern. (Answer: $r=0.0000$; the points form a U-shaped pattern.)
Using the same $x$-values as in Question 2, let $y=(0,3,4,3,0)$. What is the correlation, and how does the pattern differ from Question 2? (Answer: $r=0.0000$; it is an inverted-U pattern.)
Four paired observations are $(-6,-4),(-4,-6),(4,6),(6,4)$, with the first two assigned to Group A and the last two to Group B. The pooled correlation is $12/13$. What direction appears within each group, and why can it differ from the pooled direction? (Answer: Both groups have a negative within-group direction; separation of the groups drives the strongly positive pooled correlation.)
For $(1,1),(2,2),(3,3),(10,0)$, calculate the correlation to four decimals. Should the last observation be removed just because the result changes? (Answer: $r=-0.6325$; no—inspect and validate the observation before deciding how to handle it.)

What You'll Learn