Why Pearson Correlation Is Useful—and Limited
When two numeric predictors contain much the same information, including both in a model can create redundancy. Pearson correlation provides a concise measure of their linear association: values near \(1\) indicate a strong positive linear pattern, values near \(-1\) a strong negative linear pattern, and values near \(0\) little linear association. It is useful as an initial summary when screening numeric predictors, but it is not a complete description of their relationship.
The same correlation can arise from very different scatterplots. A single unusual observation can substantially alter it, and a clear curved pattern can have a correlation close to zero. For that reason, interpret Pearson correlation alongside a scatterplot of the paired observations. Correlation describes association, not causation, and a large absolute correlation does not by itself establish that one predictor should be removed.
The formula uses paired values: each \(x_i\) must be matched to the \(y_i\) observed on the same record. If a predictor has no variation, its sum of squared deviations is zero, and \(r\) is undefined. Missing values also require care: the calculation must use pairs from the same observations, rather than combining separate sets of available \(x\) and \(y\) values.
How to Calculate and Interpret \(r\)
A reliable calculation starts by finding both means, then computing each observation’s deviations from those means. The numerator adds the products of paired deviations. The two sums of squared deviations in the denominator scale that numerator, making the result unit-free and bounded between \(-1\) and \(1\).
Confirm that each row is a matched pair of numeric observations and that both variables vary.
Calculate \(\bar{x}\) and \(\bar{y}\), then find \(x_i-\bar{x}\) and \(y_i-\bar{y}\) for each row.
Find the sum of paired deviation products and the sum of squared deviations for each variable.
Calculate \(r\), then state its direction and the strength of the linear association in context. Check the scatterplot before drawing a broader conclusion.
The sample covariance is the numerator divided by \(n-1\); each sample standard deviation divides its corresponding sum of squared deviations by \(n-1\) before taking the square root. In their ratio, the factors of \(n-1\) cancel, giving the displayed formula. This also explains why \(r\) is not measured in the units of either variable.
Worked Example: Comparing Two Numeric Predictors
In a hypothetical portfolio, an analyst compares a numeric building-age score \(x\) with a numeric maintenance-spend score \(y\) for five properties. The paired observations are \(x=(1,2,3,4,5)\) and \(y=(2,4,5,4,5)\). Calculate Pearson correlation and describe its screening implication.
Method. Use the centered-deviation formula. The means are \(\bar{x}=(1+2+3+4+5)/5=3\) and \(\bar{y}=(2+4+5+4+5)/5=20/5=4\).
The deviations are \(x_i-\bar{x}=(-2,-1,0,1,2)\) and \(y_i-\bar{y}=(-2,0,1,0,1)\). Their paired products are \(4,0,0,0,2\), so the numerator is \(4+0+0+0+2=6\). The squared deviations sum to \(4+1+0+1+4=10\) for \(x\), and \(4+0+1+0+1=6\) for \(y\).
Rounded to four decimal places, \(r=0.7746\). The observed association is positive and fairly strong in a linear sense. This is a screening signal that the predictors may share information; it is not proof that either is unnecessary. The scatterplot should be checked for curvature, clusters, or influential observations before using this summary. The method avoids treating a correlation as a model-selection decision by itself.
Why Units Do Not Affect Correlation
Changing units or shifting the origin does not change the correlation when the transformation is a positive linear rescaling. For example, converting a measurement from metres to centimetres multiplies every value by \(100\). Its deviations and standard deviation are also multiplied by \(100\), so the multiplier cancels from the correlation formula. Adding a constant changes the values but not their deviations from the mean.
More generally, replacing \(x\) with \(a+bx\) and \(y\) with \(c+dy\), where \(b\) and \(d\) are nonzero, leaves the absolute value of correlation unchanged. Its sign is reversed if exactly one of \(b,d\) is negative; if both are negative or both are positive, the sign is unchanged. This property makes correlation useful for comparing linear association across differently scaled measurements. It does not mean correlation is unaffected by unusual observations or by nonlinear transformations.
Scatterplots Reveal What a Single Number Hides
A scatterplot places one predictor on each axis and displays the paired observations as points. Before interpreting \(r\), look for the overall direction and shape, isolated points far from the main cloud, and separate clusters. A linear summary is most informative when the plotted relationship is reasonably straight and no point dominates the pattern.
An outlier can change the sample means, the deviation products, and the sums of squares at the same time. Consequently, its effect on \(r\) can be substantial; it is not safe to assume that an outlier merely makes correlation larger or merely makes it smaller. Investigate whether the observation reflects a recording or data-processing error. If it is a valid observation, do not discard it simply because it changes the result; report how it affects the interpretation.
Worked Example: Checking the Effect of an Extreme Observation
A hypothetical operations team records \(x\), a numeric workload score, and \(y\), a numeric processing score. The first five pairs are \((1,1),(2,2),(3,3),(4,4),(5,5)\). A sixth record, \((6,0)\), is added. Calculate and compare the correlations before and after adding that record.
Before adding the sixth record: the first five points lie on \(y=x\), so their means are both \(3\). The deviation products sum to \(4+1+0+1+4=10\), and each variable’s squared deviations sum to \(10\). Therefore \(r=10/\sqrt{10\cdot10}=1\).
After adding the sixth record: the new means are \(\bar{x}=21/6=3.5\) and \(\bar{y}=15/6=2.5\). The \(x\)-deviations are \((-2.5,-1.5,-0.5,0.5,1.5,2.5)\), and the \(y\)-deviations are \((-1.5,-0.5,0.5,1.5,2.5,-2.5)\). The paired products are \(3.75,0.75,-0.25,0.75,3.75,-6.25\), which sum to \(2.5\).
The squared \(x\)-deviations sum to \(6.25+2.25+0.25+0.25+2.25+6.25=17.5\). The squared \(y\)-deviations also sum to \(2.25+0.25+0.25+2.25+6.25+6.25=17.5\). Thus \(r=2.5/\sqrt{17.5\cdot17.5}=2.5/17.5\approx0.1429\).
Adding one point changed the correlation from \(1\) to approximately \(0.1429\). The calculation does not establish whether the sixth record is erroneous or should be excluded. A scatterplot would show that it is far from the first five points’ straight-line pattern and would make its influence visible. The method avoids assuming that a high correlation is stable simply because most observations follow a line.
A different limitation occurs when the relationship is strong but curved. Pearson correlation summarizes a straight-line pattern, not every kind of association. In particular, symmetry can make positive and negative deviation products cancel.
Worked Example: A Strong Curved Pattern With Zero Correlation
For five hypothetical sites, a predictor \(x\) takes values \((-2,-1,0,1,2)\), and a numeric response-like predictor \(y\) takes values \((4,1,0,1,4)\). Calculate \(r\) and explain why it does not imply that the variables have no relationship.
Method. First calculate the means: \(\bar{x}=0\) and \(\bar{y}=(4+1+0+1+4)/5=2\). The \(x\)-deviations are \((-2,-1,0,1,2)\); the \(y\)-deviations are \((2,-1,-2,-1,2)\).
The paired products are \(-4,1,0,-1,4\), which sum to \(0\). The squared \(x\)-deviations sum to \(4+1+0+1+4=10\), and the squared \(y\)-deviations sum to \(4+1+4+1+4=14\). Therefore \(r=0/\sqrt{10\cdot14}=0\).
The scatterplot would show the exact curved pattern \(y=x^2\), not a cloud with no structure. The correlation is zero because the negative products on one side of \(x=0\) cancel the positive products on the other side. A conclusion of “no relationship” would be wrong; the correct conclusion is that this example has no linear association as summarized by Pearson correlation, despite a clear nonlinear relationship.
Common Calculation Trap: Check the Numerator
When the data are centered, the numerator is the sum of all paired products. Missing a product or using a single product instead of the sum can produce an incorrect correlation, even when the denominator is right.
Worked Example: Auditing a Correlation Calculation
For \(x=(2,4,6)\) and \(y=(9,6,3)\), calculate \(r\) and identify the common arithmetic error that would give the wrong answer.
The means are \(\bar{x}=12/3=4\) and \(\bar{y}=18/3=6\). The deviations are \(x_i-\bar{x}=(-2,0,2)\) and \(y_i-\bar{y}=(3,0,-3)\). Their paired products are \(-6,0,-6\), so the numerator is \(-6+0-6=-12\), not \(-6\).
The squared deviations sum to \(4+0+4=8\) for \(x\), and \(9+0+9=18\) for \(y\). Hence $$ r=\frac{-12}{\sqrt{8\cdot18}}=\frac{-12}{\sqrt{144}}=-1. $$
The three points lie exactly on the decreasing line \(y=12-1.5x\), so a correlation of \(-1\) is consistent with the data. A common incorrect calculation uses only one of the two nonzero products, giving \(-6/\sqrt{8\cdot18}=-0.5\). The correction is to sum every paired product before dividing. This example also illustrates why checking the plot or the exact linear pattern is a useful arithmetic check.
Using Correlation Responsibly in Predictor Screening
For numeric predictors, a correlation matrix can quickly indicate pairs with strong linear redundancy. A large positive or negative value means that the observed values tend to move together in a straight-line pattern, in the same or opposite direction. It does not say that the predictors are identical, that the association will persist in new data, or that either predictor has a particular relationship with the outcome.
A correlation computed for a screening step is also dependent on the observations used. A small sample, a restricted range of predictor values, a subgroup structure, or influential records can change the apparent pattern. State which observations the calculation represents and inspect the scatterplot in the same data subset. Keep the conclusion appropriately narrow: “these predictors show a strong positive linear association in this sample” is more accurate than “these predictors are redundant in every setting.”
Pearson correlation is aligned with the SOA Exam PA (Predictive Analytics) syllabus because it connects a numerical summary to a practical screening question and requires careful interpretation. The exam-relevant discipline is to calculate \(r\) correctly, identify what it does measure, and recognize from a scatterplot when that summary is insufficient. The central trap is treating a value near zero as proof of no association, or a large absolute value as a complete diagnosis of redundancy.
Check Your Understanding
For each problem, calculate or interpret the correlation using the paired data given. Treat all examples as hypothetical.
- For \(x=(0,1,2)\) and \(y=(5,3,1)\), calculate Pearson correlation and state its direction. (Answer: \(r=-1.0000\); perfectly negative linear association.)
- For \(x=(1,2,3,4)\) and \(y=(2,2,4,4)\), calculate \(r\) to four decimal places. (Answer: \(r=0.8944\).)
- For \(x=(1,2,3,4)\) and \(y=(1,4,2,5)\), calculate \(r\) to four decimal places. (Answer: \(r=0.7071\).)
- A scatterplot shows a precise U-shaped pattern, with lower \(y\) values near the middle of the \(x\)-range and higher \(y\) values at both ends. If the plot is symmetric around the middle, could Pearson correlation be zero even though \(x\) and \(y\) are clearly related? (Answer: Yes. Symmetry can make the paired deviation products cancel, giving \(r=0\) despite a clear nonlinear relationship.)
- Five paired observations follow \(y=x\) exactly, and a sixth observation lies far from that line. Is it valid to assume that the six-point correlation remains \(1\), and what should be checked? (Answer: No. Recalculate using all six pairs and inspect the scatterplot; determine whether the unusual point is a data error or a valid influential observation before deciding how to handle it.)