Tutorials › CAS Exam 9 - Risk Management for Actuaries › Sample Pearson Correlation and Calculation

Home  /  Tutorials  /  CAS Exam 9 / Correlation and Tail Dependence

CAS Exam 9 · Correlation and Tail Dependence · Tutorial 425 of 995

Sample Pearson Correlation and Calculation

Calculate and interpret sample Pearson correlation from paired observations, and see why the sample denominator matters for covariance but cancels from the correlation formula.

Advanced 10 min read

What You'll Learn

  • Identify the paired-observation conditions needed to calculate sample Pearson correlation.
  • Calculate the sample correlation using centered deviations.
  • Use computational sums to obtain centered sums of squares and cross-products.
  • Explain the role of \(n-1\) in sample covariance and why it cancels from correlation.
  • Recognize denominator mismatches and precision loss as calculation traps.

Why Calculate Sample Correlation?

Actuarial analysis often begins with paired observations: for each period, portfolio, or simulated scenario, two risk measures are recorded together. Sample Pearson correlation summarizes the direction and strength of their linear association in those observed pairs. It can be used to describe data and to estimate a population relationship, but it does not by itself establish causation or capture every form of dependence.

The calculation is straightforward once the pairs are kept intact. The main choices are whether to work from deviations around the sample means or from computational sums, and how to use the sample denominator consistently. Neither method changes the result when applied correctly.

Definition: For \(n\) paired observations \((x_i,y_i)\), where \(x_i\) and \(y_i\) are the two measurements on the same observational unit, the sample means are \(\bar{x}=\sum_{i=1}^n x_i/n\) and \(\bar{y}=\sum_{i=1}^n y_i/n\). Define the centered sums \(C_{XX}=\sum_{i=1}^n(x_i-\bar{x})^2\), \(C_{YY}=\sum_{i=1}^n(y_i-\bar{y})^2\), and \(C_{XY}=\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})\). If both variables vary in the sample, the sample Pearson correlation is \(r=C_{XY}/\sqrt{C_{XX}C_{YY}}\).

The observations must be correctly paired: the \(i\)th \(x\)-value and \(i\)th \(y\)-value must refer to the same unit. The formula requires \(C_{XX}>0\) and \(C_{YY}>0\); if either variable is constant in the sample, its sample standard deviation is zero and the correlation is undefined. The calculation itself does not require the observations to be independent or identically distributed. Those assumptions matter for particular statistical inferences, not for computing \(r\) from a given dataset.

Centered-Deviation Calculation

The centered method follows the definition directly: find each sample mean, subtract it from every observation, and calculate the sums of squared deviations and paired deviation products. Keeping a table of the paired deviations helps prevent mismatching observations.

Worked Example: Comparing Two Hypothetical Quarterly Loss Measures

A hypothetical insurer records two loss measures, in thousands of dollars, over four quarters: \(x=(2,4,6,8)\) and \(y=(1,3,4,6)\). Calculate the sample Pearson correlation using centered deviations. Name the method and identify a common mistake it avoids.

Method. Use centered deviations so every product is formed from observations belonging to the same quarter. The sample means are \(\bar{x}=(2+4+6+8)/4=5\) and \(\bar{y}=(1+3+4+6)/4=3.5\).

The paired deviations are \((-3,-2.5),(-1,-0.5),(1,0.5),(3,2.5)\). Thus \(C_{XX}=(-3)^2+(-1)^2+1^2+3^2=9+1+1+9=20\). Similarly, \(C_{YY}=(-2.5)^2+(-0.5)^2+0.5^2+2.5^2=6.25+0.25+0.25+6.25=13\).

The cross-products sum to \(C_{XY}=(-3)(-2.5)+(-1)(-0.5)+(1)(0.5)+(3)(2.5)=7.5+0.5+0.5+7.5=16\). Therefore,

$$ r=\frac{16}{\sqrt{20\cdot 13}}=\frac{16}{\sqrt{260}}\approx 0.9923. $$

The result is a strong positive linear association in these four observed pairs. The common mistake avoided here is sorting or rearranging one variable separately: that would destroy the pairing and change the cross-products. As a check, \(r\) is positive because the larger \(x\)-values generally occur with larger \(y\)-values, and it lies between \(-1\) and \(1\).

Computational Formula

When the observations are numerous, or when a table of deviations is inconvenient, the same centered sums can be obtained from raw totals. Expanding the products gives the computational formulas below.

$$ C_{XX}=\sum_{i=1}^n x_i^2-\frac{(\sum_{i=1}^n x_i)^2}{n}, \qquad C_{YY}=\sum_{i=1}^n y_i^2-\frac{(\sum_{i=1}^n y_i)^2}{n}, $$
$$ C_{XY}=\sum_{i=1}^n x_i y_i-\frac{(\sum_{i=1}^n x_i)(\sum_{i=1}^n y_i)}{n}, \qquad r=\frac{C_{XY}}{\sqrt{C_{XX}C_{YY}}}. $$

These formulas are algebraically equivalent to calculating deviations from the means. They can be convenient, but if the observations have large magnitudes and little variation, the terms being subtracted may be very close. Rounding those large terms too early can cause a substantial relative error in a small centered sum. In such cases, centered deviations or adequate computational precision are safer.

Worked Example: Using Computational Sums with Large Offsets

For five hypothetical paired observations, let \(x=(1001,1002,1003,1004,1005)\) and \(y=(10,13,9,12,11)\). Calculate \(r\) using the computational formula. The values of \(x\) are close together relative to their size, so retain exact integer sums until the centered quantities are obtained.

Method. First find the totals: \(\sum x_i=5015\), \(\sum y_i=55\), and \(\sum y_i^2=100+169+81+144+121=615\). For \(x\), the sum of squares is \(1001^2+1002^2+1003^2+1004^2+1005^2=1{,}002{,}001+1{,}004{,}004+1{,}006{,}009+1{,}008{,}016+1{,}010{,}025=5{,}030{,}055\).

The paired products sum to \(\sum x_i y_i=1001(10)+1002(13)+1003(9)+1004(12)+1005(11)=10{,}010+13{,}026+9{,}027+12{,}048+11{,}055=55{,}166\). Apply the formulas:

$$ C_{XX}=5{,}030{,}055-\frac{5015^2}{5} =5{,}030{,}055-5{,}030{,}045=10, $$
$$ C_{YY}=615-\frac{55^2}{5}=615-605=10, \qquad C_{XY}=55{,}166-\frac{5015\cdot55}{5} =55{,}166-55{,}165=1. $$

Therefore \(r=1/\sqrt{10\cdot10}=0.1000\). As a check, the means are \(1003\) and \(11\); the \(x\)-deviations are \((-2,-1,0,1,2)\), and the \(y\)-deviations are \((-1,2,-2,1,0)\). Their cross-products sum to \(2-2+0+1+0=1\), consistent with \(C_{XY}\). The common mistake is to miscalculate a large sum of squares or its correction term: either error can be concealed if both are wrong by the same amount. Check the raw sum and correction separately.

Why the Sample Denominator Is \(n-1\)

The sample covariance and sample variances use \(n-1\), rather than \(n\), as their denominator. The reason is that the sample means are estimated from the same observations, so the deviations from those means sum to zero and are not all free to vary independently. For independent, identically distributed observations with finite variances, dividing the centered sums by \(n-1\) makes the sample covariance and sample variances unbiased estimators of their population counterparts.

1
Expand the centered cross-products.
\(\sum (x_i-\bar{x})(y_i-\bar{y})=\sum x_i y_i-\bar{x}\sum y_i-\bar{y}\sum x_i+n\bar{x}\bar{y}\).
2
Substitute the sample means.
Since \(\sum x_i=n\bar{x}\) and \(\sum y_i=n\bar{y}\), the last three terms combine to \(-n\bar{x}\bar{y}=-(\sum x_i)(\sum y_i)/n\). This gives the computational formula for \(C_{XY}\).
3
Use the sample covariance consistently.
Define \(s_{XY}=C_{XY}/(n-1)\), \(s_X^2=C_{XX}/(n-1)\), and \(s_Y^2=C_{YY}/(n-1)\). Then \(\sqrt{s_X^2s_Y^2}=\sqrt{C_{XX}C_{YY}}/(n-1)\), so dividing \(s_{XY}\) by this quantity cancels \(n-1\) and gives \(r=C_{XY}/\sqrt{C_{XX}C_{YY}}\).

The cancellation explains why the correlation formula contains no explicit \(n-1\), even though it is built from sample covariance and sample standard deviations. The choice between dividing all three centered sums by \(n-1\) or all three by \(n\) does not change the resulting correlation. It does change the covariance and variance estimates themselves.

Worked Example: Checking the Denominator in a Sample Correlation

For four hypothetical paired risk scores, suppose the centered sums calculated from the data are \(C_{XX}=10\), \(C_{YY}=10\), and \(C_{XY}=-9\). Calculate the sample covariance, sample variances, and correlation. Then show the error from mixing sample and \(n\)-denominator quantities.

Method. With \(n=4\), the sample denominator is \(n-1=3\). Thus \(s_{XY}=-9/3=-3\), \(s_X^2=10/3\), and \(s_Y^2=10/3\). The correlation is

$$ r=\frac{-3}{\sqrt{(10/3)(10/3)}}=\frac{-3}{10/3}=-0.9000. $$

Equivalently, \(r=-9/\sqrt{10\cdot10}=-0.9000\). A common mistake is to divide the covariance by \(n-1\) but calculate the standard deviations using variances divided by \(n\). That incorrect mixed calculation gives \((-9/3)/\sqrt{(10/4)(10/4)}=-3/2.5=-1.2\), which is impossible for a Pearson correlation. Using a consistent denominator gives the valid value \(-0.9000\), which is within the required bounds.

Common Calculation Traps

  • Breaking the pairs. Each cross-product must use the two observations from the same unit. Reordering one column changes the statistic.
  • Mixing denominators. Use either the sample covariance and sample standard deviations, all based on \(n-1\), or the centered-sum formula. Do not mix sample and \(n\)-denominator quantities.
  • Using the wrong correction term. In \(C_{XY}\), the correction is \((\sum x_i)(\sum y_i)/n\), not \((\sum x_i)^2/n\). The latter belongs in \(C_{XX}\).
  • Rounding too early. Keep exact sums or extra precision through the centered quantities; round the final correlation only.
  • Overinterpreting the result. A sample correlation describes linear association in the observed pairs. It is not proof of causation, independence, or the absence of other dependence patterns.
Key takeaway: Calculate \(C_{XX}\), \(C_{YY}\), and \(C_{XY}\) from correctly paired observations, then compute \(r=C_{XY}/\sqrt{C_{XX}C_{YY}}\). The \(n-1\) denominator is used for sample covariance and variance, but cancels when they are combined consistently into correlation. These calculation skills are aligned with the CAS Exam 9 (Risk Management for Actuaries) syllabus.

Check Your Understanding

For each problem, calculate the requested sample quantities from the paired observations. Round correlations to four decimal places where needed.

  1. For \(x=(1,2,3)\) and \(y=(2,4,5)\), calculate \(C_{XX}\), \(C_{YY}\), \(C_{XY}\), and \(r\) using centered deviations. (Answer: \(C_{XX}=2\), \(C_{YY}=14/3\), \(C_{XY}=3\), \(r\approx0.9820\))
  2. For \(x=(2,3,5,6)\) and \(y=(9,7,6,4)\), calculate the sample means and Pearson correlation using centered deviations. (Answer: \(\bar{x}=4\), \(\bar{y}=6.5\), \(C_{XX}=10\), \(C_{YY}=13\), \(C_{XY}=-11\), \(r\approx-0.9648\))
  3. For \(x=(10,20,30,40)\) and \(y=(3,5,4,8)\), use raw sums to calculate the three centered sums and \(r\). (Answer: \(C_{XX}=500\), \(C_{YY}=14\), \(C_{XY}=70\), \(r=70/\sqrt{7000}\approx0.8367\))
  4. For \(x=(5,7,9,11)\) and \(y=(2,6,5,7)\), calculate the sample covariance and sample variances using \(n-1\), then verify the correlation using the centered-sum formula. (Answer: \(s_{XY}=14/3\), \(s_X^2=20/3\), \(s_Y^2=14/3\), \(r=14/\sqrt{280}\approx0.8367\))
  5. For \(x=(1,1,3,3)\) and \(y=(2,4,6,8)\), calculate the means, centered sums, and sample correlation. State whether the correlation is defined. (Answer: \(\bar{x}=2\), \(\bar{y}=5\), \(C_{XX}=4\), \(C_{YY}=20\), \(C_{XY}=8\), \(r=8/\sqrt{80}\approx0.8944\); it is defined because both variables vary.)

Previous

Next

Back to Tutorials
Clinical Biostats