Why Rank Correlation Is Useful
Pearson correlation measures linear association: it summarizes how closely paired values follow a straight-line pattern. But predictors can have a strong increasing or decreasing relationship without that relationship being linear. For example, a predictor may rise steadily as another predictor increases, while the size of each increase changes substantially. In that setting, the order of the values may carry a clearer signal than their distances.
Spearman rank correlation measures the direction and strength of a monotone association by replacing the observed values with their ranks and calculating the Pearson correlation of those ranks. It is useful when screening numeric or ordinal predictors for relationships that Pearson correlation may summarize poorly. It is not a general test for every kind of dependence, and it does not establish a causal relationship.
The rank transformation preserves order but discards the original spacing between values. A strictly increasing transformation of one predictor leaves its ranks unchanged; a strictly decreasing transformation reverses its ranks. Consequently, the sign of \(r_s\) indicates whether higher values of one variable tend to accompany higher or lower values of the other. Its magnitude describes how consistently the paired ranks move together.
Calculating Spearman Correlation Without Ties
When there are no ties in either variable, rank each variable from 1 to \(n\). For observation \(i\), let \(d_i=R_{Xi}-R_{Yi}\), where \(R_{Xi}\) and \(R_{Yi}\) are its ranks in the two variables. The rank-difference shortcut is then
The formula assumes that both rank lists contain exactly the integers 1 through \(n\), which is why it is appropriate when there are no tied values. When ties occur, use the Pearson correlation of average ranks instead; the shortcut above is not generally valid.
Worked Example: Screening Two Predictors With No Ties
A hypothetical portfolio dataset contains six records with two numeric predictors: an exposure score \(X\) and a service-duration measure \(Y\). The ordered \(X\)-values are 10, 20, 30, 40, 50, 60. The corresponding \(Y\)-values, in that same record order, are 12, 18, 35, 33, 50, 70. Calculate and interpret \(r_s\). No distributional model is assumed; the target is a descriptive association across these six paired records.
Step 1: Rank both variables. The \(X\)-ranks are \(1,2,3,4,5,6\). Sorting the \(Y\)-values gives \(12,18,33,35,50,70\), so their ranks in the original record order are \(1,2,4,3,5,6\).
Step 2: Find rank differences and squares. Subtract the \(Y\)-rank from the \(X\)-rank for each record:
Step 3: Apply the no-ties formula. Here \(n=6\), so the denominator is \(6(6^2-1)=6(35)=210\):
The ranks have a strong positive association: records with higher \(X\) generally have higher \(Y\). The value is not exactly 1 because the third and fourth records reverse order. A common mistake is to rank the \(Y\)-values after sorting them but then fail to restore those ranks to the original paired records; keeping the pairs aligned is essential.
Ties: Use Average Ranks
Ties are common when predictors are rounded, recorded in categories with an order, or take only a small number of distinct values. If two observations tie for ranks 2 and 3, assign each the average rank \((2+3)/2=2.5\). Continue to compute the Pearson correlation of the two rank vectors. This procedure gives each tied observation the same rank and preserves the average of the ranks that the tied observations jointly occupy.
Worked Example: Calculating Correlation With a Tie
For five hypothetical policy records, predictor \(X\) has values \(1,2,2,4,5\), while predictor \(Y\), paired in that order, has values \(10,20,15,30,25\). Calculate Spearman correlation. No distributional model is assumed.
Step 1: Assign ranks, averaging the tie. The two \(X\)-values equal to 2 occupy ranks 2 and 3, so each receives rank 2.5. Thus \(R_X=(1,2.5,2.5,4,5)\). The \(Y\)-ranks in paired order are \(R_Y=(1,3,2,5,4)\).
Step 2: Center the ranks. Each rank vector has mean rank \((1+2+3+4+5)/5=3\). The centered vectors are
Step 3: Calculate the rank covariance numerator and sums of squares. The sum of cross-products is \(4+0+0.5+2+2=8.5\), not 8: the third product is \((-0.5)(-1)=0.5\), and the fifth product is \(2(1)=2\). Therefore the cross-product sum is \(8.5\). The rank sums of squares are \(4+0.25+0.25+1+4=9.5\) for \(X\), and \(4+0+1+4+1=10\) for \(Y\).
Step 4: Calculate the Pearson correlation of the ranks.
The records show a positive rank association. The method avoids a common tie-related error: using the no-ties shortcut on average ranks. With ties, calculate the Pearson correlation of the rank vectors directly. The intermediate cross-product sum is shown explicitly so that the tied-rank arithmetic can be checked.
Why the Rank-Difference Formula Holds
The shortcut follows directly from the Pearson correlation of ranks when there are no ties. Let \(R_X\) and \(R_Y\) denote the rank vectors. Both have mean \((n+1)/2\), and each has the same sum of squared deviations:
To obtain this value, use \(\sum_{k=1}^{n} k=n(n+1)/2\) and \(\sum_{k=1}^{n} k^2=n(n+1)(2n+1)/6\) in the expansion of \(\sum_{k=1}^{n}(k-(n+1)/2)^2\). Since ranks are centered at the same mean, subtracting the rank vectors gives
where \(\bar R=(n+1)/2\). The Pearson correlation of the ranks is the cross-product sum divided by \(Q\). Rearranging the preceding identity therefore gives
This derivation also explains the limitation of the shortcut: it relies on each rank vector having the same fixed sum of squares \(Q\), which is not true in general when average ranks replace the full set of distinct integer ranks.
Monotone Does Not Mean Linear, and Correlation Does Not Mean Dependence Is Absent
Worked Example: A Monotone Pattern With Uneven Spacing
Consider five hypothetical records with \(X=(1,2,3,4,5)\) and \(Y=(1,2,3,4,100)\). Calculate Spearman correlation and compare it with Pearson correlation to see how the measures respond to spacing. There are no ties, and no distributional model is assumed.
Spearman calculation. Both variables have ranks \((1,2,3,4,5)\), so every rank difference is zero. Thus \(\sum d_i^2=0\), and
Pearson comparison. The means are \(\bar X=3\) and \(\bar Y=22\). The centered vectors are \((-2,-1,0,1,2)\) and \((-21,-20,-19,-18,78)\). Their cross-product sum is \(42+20+0-18+156=200\); their sums of squares are \(10\) and \(441+400+361+324+6084=7610\). Hence
The ordering is perfectly increasing, giving Spearman correlation 1, even though the uneven final gap makes Pearson correlation smaller. Spearman describes the consistency of ordering, not how evenly spaced the values are.
Worked Example: A Strong Relationship That Spearman Misses
A hypothetical predictor \(X\) takes values \(1,2,3,4,5,6,7\), and a second predictor \(Y\) is determined by \(Y=(9,4,1,0,1,4,9)\) in that order. Calculate Spearman correlation and explain what it says about dependence. This is a descriptive calculation; no distributional model is assumed.
Step 1: Rank both variables. The \(X\)-ranks are \((1,2,3,4,5,6,7)\). In \(Y\), the value 0 has rank 1; the two 1s receive average rank 2.5; the two 4s receive average rank 4.5; and the two 9s receive average rank 6.5. Therefore \(R_Y=(6.5,4.5,2.5,1,2.5,4.5,6.5)\).
Step 2: Calculate the rank correlation. Both rank means are 4. The centered \(X\)-ranks are \((-3,-2,-1,0,1,2,3)\), and the centered \(Y\)-ranks are \((2.5,0.5,-1.5,-3,-1.5,0.5,2.5)\). Their cross-product sum is
The rank vectors have nonzero sums of squares, so their Pearson correlation is \(r_s=0\). Yet \(Y\) is completely determined by \(X\) in this example: it decreases toward the middle and then increases. Spearman correlation misses this U-shaped, non-monotone dependence because the increasing and decreasing rank patterns cancel.
A value near zero therefore means little evidence of a monotone relationship, not proof that two predictors are unrelated. Inspecting a scatterplot remains important, particularly when a nonlinear pattern is plausible. Rank correlation is also not immune to every data issue: one changed observation can alter ranks, ties reduce the detail available in ordering, and neither a high nor low correlation establishes causation.
Practical Calculation and Interpretation
In R, the following calculates Spearman correlation for two numeric columns:
The option shown uses records for which both variables are observed. Confirm the number of paired complete records and use the same records for both rank vectors. In particular, do not rank each predictor on a different subset and then correlate those ranks. For a screening task, describe the direction, approximate strength, and limitation in context: for example, “the predictors have a strong positive monotone association in the observed records; this does not show that their relationship is linear or causal.”
| Finding | Appropriate interpretation | What it does not establish |
|---|---|---|
| Large positive \(r_s\) | Higher values tend to accompany higher values in rank | A linear relationship or causation |
| Large negative \(r_s\) | Higher values tend to accompany lower values in rank | A particular functional form |
| \(r_s\) near zero | Little overall monotone rank association | Absence of nonlinear dependence |
When comparing predictors, treat correlation as a screening summary rather than an automatic instruction to remove a variable. Check the observed pattern, data quality, and purpose of the model before acting on a correlation value. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, retain the paired-record order when ranking and round correlation values to four decimal places.
- A hypothetical sample has four paired records. \(X=(8,12,15,21)\) and \(Y=(3,9,6,14)\). Rank both variables, use the no-ties formula, and interpret the sign. (Answer: \(R_X=(1,2,3,4)\), \(R_Y=(1,3,2,4)\), \(r_s=0.8000\), a positive rank association.)
- For four hypothetical records, \(X=(2,4,4,8)\) and \(Y=(5,9,7,11)\). Assign average ranks and calculate the Pearson correlation of those ranks. (Answer: \(R_X=(1,2.5,2.5,4)\), \(R_Y=(1,3,2,4)\), \(r_s\approx 0.9487\).)
- A predictor has ranks \((1,2,3,4,5)\). A second predictor is a strictly decreasing transformation of the first, with no ties. What are the second ranks and Spearman correlation? State the interpretation. (Answer: second ranks \((5,4,3,2,1)\), \(r_s=-1.0000\); the ordering is perfectly decreasing.)
- For six records, \(X=(1,2,3,4,5,6)\) and the \(Y\)-ranks in paired order are \((2,5,3,6,4,1)\). Calculate \(\sum d_i^2\), then \(r_s\). (Answer: \(d=(-1,-3,0,-2,1,5)\), \(\sum d_i^2=40\), and \(r_s=1-240/210\approx -0.1429\).)
- A dataset has a clear U-shaped scatterplot, but the computed Spearman correlation is near zero. Explain whether this contradicts the observed pattern and identify what the correlation summarizes. (Answer: No contradiction; opposing rank trends can cancel. A value near zero indicates little overall monotone rank association, not absence of dependence.)