Why a Correlation Coefficient Needs a Picture
Pearson correlation summarizes the strength and direction of linear association between two variables. That is useful when a linear pattern is a reasonable description of the paired observations. It can be misleading when one observation has unusual influence or when the relationship is curved. In risk work, a coefficient by itself may conceal the pattern of paired losses that matters for modeling.
A scatterplot puts each paired observation \((x_i,y_i)\) on the same set of axes. It can reveal a cluster, a curved shape, or an unusual point that a single summary number cannot show. The plot and the coefficient answer different questions: the coefficient quantifies linear association, while the plot helps assess whether that summary describes the data adequately.
The paired observations must be kept together: each \(x_i\) is matched with its corresponding \(y_i\). The formula does not require a particular distribution to calculate \(r\). However, treating the coefficient as a summary of a broader population pattern requires care: a few unusual observations or a nonlinear pattern can make a linear summary unrepresentative. A scatterplot is therefore an essential diagnostic, not a decoration added after the calculation.
How an Unusual Observation Can Change the Summary
An observation can be influential because it is far from the other points in the \(x\)-direction, the \(y\)-direction, or both. A point far out in the \(x\)-direction is often described as having high leverage: it can strongly affect the direction of a fitted linear pattern. Whether a point should be excluded depends on the data-generating context and the analysis objective. An observation is not invalid merely because it changes the answer.
Worked Example: One High-Leverage Loss Pair Reverses the Correlation
A hypothetical analyst pairs a portfolio's quarterly claim count \(x\) with its corresponding quarterly claim amount \(y\), measured in thousands of dollars. Five quarters have pairs \((1,2),(2,4),(3,6),(4,8),(5,10)\). A sixth quarter has \((20,0)\). Calculate the sample correlation with all six quarters, compare it with the first five, and interpret the change.
Method. Calculate the centered sums from the raw sums, then apply the sample Pearson formula. For all six observations, the sums are \(\sum x_i=35\), \(\sum y_i=30\), \(\sum x_i^2=455\), \(\sum y_i^2=220\), and \(\sum x_i y_i=110\). The means are \(\bar{x}=35/6\) and \(\bar{y}=5\).
The centered sums are
Thus
For the first five quarters, \(y_i=2x_i\) for every observation, so their sample correlation is exactly \(1\). The sixth pair changes the correlation from a perfect positive linear association to a moderate negative one. It is far from the other \(x\)-values and has substantial influence on the result. The all-observation coefficient does not establish that the underlying relationship became negative; it shows how sensitive this sample's linear summary is to that point. The calculation is checked by the fact that the numerator \(S_{xy}\) is negative, so the reported correlation must be negative.
Common mistake avoided: Reporting “moderate negative association” as a complete description without inspecting the points. The correct interpretation notes both the sign and magnitude of \(r\) and the influential observation that drives the change. Do not remove the quarter automatically: first determine whether it is a recording error, a valid unusual outcome, or evidence that one linear summary is inadequate.
Zero Correlation Does Not Mean Independence
Correlation describes linear association, not every possible form of dependence. In particular, a variable can be determined exactly by another variable and still have zero Pearson correlation. A symmetric curved pattern is a standard way this occurs: positive and negative contributions to the centered cross-product cancel.
Worked Example: A Deterministic Curve With Zero Pearson Correlation
For five hypothetical paired loss indicators, let \(x=-2,-1,0,1,2\), and let the paired severity measure be \(y=x^2\). Find the sample Pearson correlation and decide whether the variables are independent.
The paired \(y\)-values are \(4,1,0,1,4\). Therefore \(\bar{x}=0\) and \(\bar{y}=10/5=2\). The required sums are \(\sum x_i^2=10\), \(\sum y_i^2=16+1+0+1+16=34\), and \[ \sum x_i y_i=(-2)(4)+(-1)(1)+(0)(0)+(1)(1)+(2)(4)=0. \] Using centered sums gives
Hence \(r=0/\sqrt{10(14)}=0\). But \(y=x^2\) for every pair, so knowing \(x\) determines \(y\) exactly. The variables are dependent, despite having zero sample linear correlation. The zero numerator is checked directly: the negative-\(x\) products cancel the corresponding positive-\(x\) products.
Common mistake avoided: Concluding that zero Pearson correlation means there is no relationship. It means the centered cross-products have no net linear association in this sample; it does not rule out a curved relationship or other dependence.
Curvature Can Also Produce a High Correlation
A curved pattern does not always produce a small Pearson correlation. If the observed \(x\)-values cover only one side of a curve, or are not symmetrically distributed around its turning point, the points may follow a clear increasing or decreasing trend. Pearson correlation can then be large even though a straight line does not describe the relationship's shape adequately.
Worked Example: Strong Correlation Along a Curved Pattern
A hypothetical set of five risk scores and associated modeled severities follows \(y=x^2\), with \(x=0,1,2,3,4\). Calculate the sample correlation and check whether its size alone supports a linear description.
The paired values are \((0,0),(1,1),(2,4),(3,9),(4,16)\). The means are \(\bar{x}=10/5=2\) and \(\bar{y}=30/5=6\). The raw sums are \(\sum x_i^2=30\), \(\sum y_i^2=0+1+16+81+256=354\), and \(\sum x_i y_i=0+1+8+27+64=100\). Consequently,
The sample correlation is
This is a strong positive linear association by the usual scale of \(r\), but the observations lie on a quadratic curve, not a straight line. To check the shape, the least-squares line has slope \(S_{xy}/S_{xx}=40/10=4\) and intercept \(\bar{y}-4\bar{x}=6-8=-2\). Its fitted values at \(x=0,1,2,3,4\) are \(-2,2,6,10,14\); the observed-minus-fitted residuals are \(2,-1,-2,-1,2\). Their curved pattern, rather than random scatter around zero, warns that the line misses systematic structure.
The arithmetic is internally consistent: the positive centered cross-product gives positive \(r\), and the coefficient is less than one because the data do not lie on a straight line. The high value summarizes the strong upward trend over these observed scores; it does not eliminate the evidence of curvature.
Common mistake avoided: Treating a large \(r\) as proof that a linear model adequately describes the relationship. Check the scatterplot and, when using a fitted line, inspect whether residuals show a systematic shape.
A Short Justification for the Calculation
The correlation formula is the centered cross-product standardized by the variation in each variable. The sample covariance is \(S_{xy}/(n-1)\), and the sample standard deviations are \(\sqrt{S_{xx}/(n-1)}\) and \(\sqrt{S_{yy}/(n-1)}\). Dividing covariance by the product of those standard deviations cancels the common factor \(n-1\):
This also explains why the result has no measurement units: the units in the cross-product are canceled by the units in the two standard deviations. Standardization makes correlation comparable across different scales, but it does not make it robust to influential observations or sensitive to every form of dependence.
Practical Interpretation and Exam Traps
For paired risk data, a careful interpretation distinguishes what the coefficient establishes from what it does not. The coefficient describes the direction and strength of linear association in the observed pairs. It does not by itself establish causation, independence, a suitable distribution for the pair, or an adequate linear model.
- Inspect the pairs, not just the number. A scatterplot can show whether a point is isolated, whether the trend is curved, and whether a single observation may be driving the coefficient.
- Do not discard a valid extreme observation solely to improve the fit. Verify the data and explain any exclusion; if the observation is genuine, consider whether the chosen summary remains informative.
- Do not infer independence from zero correlation. The symmetric quadratic example has zero correlation while one variable is fully determined by the other.
- Do not infer linearity from a large correlation. The increasing quadratic example has a high \(r\), yet its residuals reveal curvature.
- Keep the interpretation tied to the observed range and context. A coefficient computed for one set of paired observations need not describe another range or another population.
For actuarial analysis, these checks matter because a dependence summary may inform the way risks are represented together. A Pearson coefficient remains useful when linear association is the question and the data support that summary. When outliers or curvature are present, report the limitation clearly rather than asking one coefficient to represent the full dependence structure. These diagnostic principles are aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.
Check Your Understanding
For each problem, calculate or interpret the linear summary as requested, then consider what the paired pattern allows you to conclude.
- A hypothetical dataset contains \((1,3),(2,6),(3,9),(4,12)\). What is its sample Pearson correlation, and what feature of the pairs supports your answer? (Answer: \(r=1\); every pair lies exactly on the increasing line \(y=3x\).)
- For \(x=-1,0,1\) and \(y=1,0,1\), calculate the sample correlation and state whether \(y\) is independent of \(x\). (Answer: \(r=0\); they are dependent because \(y=x^2\) for every pair.)
- A valid additional observation is far to the right of the other points in a scatterplot. What feature of the point should be checked, and why might the sample correlation change substantially? (Answer: Check its high leverage and its position relative to the trend; a far-right point can strongly alter the centered cross-product and hence \(r\).)
- A sample has \(S_{xx}=16\), \(S_{yy}=25\), and \(S_{xy}=-10\). Calculate \(r\) and state its linear direction. (Answer: \(r=-10/\sqrt{16(25)}=-0.5000\); the linear association is negative.)
- For the pairs \((0,0),(1,1),(2,4),(3,9)\), the observed relationship is \(y=x^2\). Is a high positive correlation sufficient to establish that a straight line describes the pattern? (Answer: No; the relationship is curved, so inspect the scatterplot or residual pattern even if \(r\) is high.)