Sampling Uncertainty in Rank Correlations

Why Rank Correlations Need Uncertainty Estimates

A sample Spearman or Kendall correlation describes the observed ordering of paired losses. It does not, by itself, show how precisely that ordering estimates the dependence in the population. A different sample from the same population could produce a noticeably different rank correlation, particularly when the sample is small or contains many tied values.

Sampling uncertainty is the variation in a statistic across repeated samples drawn under the same sampling process. It is distinct from variation in the losses themselves. For actuarial risk analysis, that distinction matters: an estimated rank association may inform dependence assessment, but a small data set can leave substantial uncertainty about its magnitude.

Definition: The standard error of a rank-correlation estimator is the standard deviation of its sampling distribution under a specified population model. An estimated standard error approximates that quantity. A confidence interval combines an estimate and an uncertainty method to give a range for the population parameter; its interpretation depends on the assumptions and procedure used.

Throughout, assume initially that the observed pairs $(X_i,Y_i)$, $i=1,\ldots,n$, are independent and identically distributed. If observations are repeated in time, clustered by policyholder, or otherwise dependent across pairs, ordinary independent-pair standard errors and resampling can be misleading. The resampling method must reflect the actual sampling structure.

Large-Sample Standard Errors Under Independence

Simple standard errors are available for testing the null hypothesis that two variables are independent when there are no ties. They are useful for a large-sample test of no association, but they are not universal standard errors for an observed nonzero correlation and should not automatically be used to form a confidence interval around it.

Result: Under independence and continuous margins, with no ties, the null standard error of Spearman’s sample rank correlation $\hat{\rho}_S$ is $$ \widehat{\operatorname{SE}}_0(\hat{\rho}_S)=\frac{1}{\sqrt{n-1}}. $$ For Kendall’s sample tau, using the untied statistic $\hat{\tau}=(C-D)/\binom{n}{2}$, where $C$ and $D$ are the numbers of concordant and discordant pairs, the null variance and standard error are $$ \operatorname{Var}_0(\hat{\tau})=\frac{2(2n+5)}{9n(n-1)},\qquad \widehat{\operatorname{SE}}_0(\hat{\tau})=\sqrt{\frac{2(2n+5)}{9n(n-1)}}. $$

These formulas describe variability when the null model of independence holds. With a sufficiently large sample, a test statistic formed by dividing the observed correlation by its null standard error is often compared with the standard normal distribution. For small samples, or when a more accurate null distribution is needed, a permutation test can be preferable: keep one variable’s observed values fixed and randomly permute the other variable’s values to generate statistics under independence.

The Spearman formula has a simple justification. With no ties, each sample has ranks $1,\ldots,n$, and under independence one ranking is a random permutation relative to the other. Spearman’s correlation is the correlation of those ranks. The permutation variance of that correlation is $1/(n-1)$; taking its square root gives the stated null standard error.

Worked Example: Testing Kendall Association With Untied Data

A hypothetical insurer has $n=30$ independent paired observations of two continuous risk measurements, with no ties. The observed Kendall statistic is $\hat{\tau}=0.40$. Use the large-sample null standard error to assess evidence against independence.

Method. Use the untied Kendall null variance, then standardize the observed statistic. The assumptions are independent pairs, continuous measurements, no ties, and the null hypothesis of independence for the reference standard error.

First calculate the variance: $$ \widehat{\operatorname{Var}}_0(\hat{\tau}) =\frac{2(2(30)+5)}{9(30)(29)} =\frac{130}{7830} =0.0166028. $$

Thus the null standard error is $$ \widehat{\operatorname{SE}}_0(\hat{\tau}) =\sqrt{0.0166028} \approx 0.12885. $$

The standardized statistic is $$ z=\frac{0.40}{0.12885}\approx 3.10. $$

Using a standard normal approximation, the two-sided p-value is approximately $0.0019$. This is evidence against independence under the stated model. It is not a confidence interval for the population Kendall association: the standard error used here is specifically the null standard error.

Check. The statistic is positive, so the observed ordering is positively associated; the positive $z$-value has the same sign. The common mistake avoided is interpreting a null-based test standard error as a general uncertainty estimate around $0.40$.

Worked Example: Spearman Test and Its Interpretation

For a hypothetical portfolio study, $n=26$ independent pairs of continuous risk measurements have no ties. The sample Spearman correlation is $\hat{\rho}_S=0.32$. Calculate the null standard error and an approximate two-sided test statistic.

Method. Under independence and no ties, use the null standard error $1/\sqrt{n-1}$. The large-sample normal approximation is being used for the test.

The standard error is $$ \widehat{\operatorname{SE}}_0(\hat{\rho}_S) =\frac{1}{\sqrt{26-1}} =\frac{1}{5} =0.20. $$

The standardized statistic is $$ z=\frac{0.32}{0.20}=1.60. $$

The approximate two-sided p-value is $2P(Z\geq1.60)\approx0.1096$, where $Z$ has a standard normal distribution. At conventional significance levels, these data do not provide strong evidence against independence. This does not establish that the population association is zero.

Check. The estimate is positive and the test statistic is positive. The common mistake avoided is claiming that $0.32$ is precise merely because it is positive; the test result depends on its size relative to sampling variability.

Paired Bootstrap for Rank-Correlation Uncertainty

For uncertainty around an estimated rank correlation away from the independence null, a paired nonparametric bootstrap is a practical option. It does not require a simple closed-form standard error for the population association. Each bootstrap sample is formed by drawing $n$ observed pairs with replacement, so the pairing is preserved. Recalculate the rank statistic from scratch in each resample.

Record the observed estimate.
Calculate Spearman’s correlation or Kendall’s tau-b from the original paired data, using a consistent treatment of ties.

Resample pairs.
Draw $n$ pair-indices with replacement from the observed $n$ pairs. Do not resample the two variables separately, since that destroys their observed pairing.

Recalculate the statistic.
Recompute ranks and the selected rank correlation in the resample. Repeated selections can create additional ties, which the statistic must handle.

Summarize the replicates.
Use the replicate standard deviation as a bootstrap standard error, or use suitable empirical quantiles as a percentile interval. More replicates improve numerical stability but do not remedy unrepresentative data.

If $B$ bootstrap estimates are $r_1^*,\ldots,r_B^*$, their mean is $\bar r^*=B^{-1}\sum_{b=1}^B r_b^*$. The usual estimated bootstrap standard error is their sample standard deviation: $$ \widehat{\operatorname{SE}}_{\mathrm{boot}} =\sqrt{\frac{\sum_{b=1}^B(r_b^*-\bar r^*)^2}{B-1}}. $$ A percentile interval uses the appropriate empirical quantiles of the $B$ estimates; for example, a nominal 95% interval uses the 2.5th and 97.5th percentiles. This is an approximate interval, not a guarantee of exact 95% coverage.

Centering matters. The sample standard deviation of bootstrap replicates uses deviations from their own mean, not deviations from the original estimate. A squared-deviation sum centered at the original estimate can instead support a different bootstrap variance estimate if divided by $B$; it is not the sample variance of the replicate values.

Worked Example: Bootstrap Uncertainty With Tied Loss Categories

A hypothetical study records $n=40$ paired risk scores, some of which take repeated ordinal values. The original sample Kendall tau-b is $0.36$. A paired bootstrap with $B=1{,}000$ resamples produces a replicate mean of $0.34$, a sum of squared deviations from that replicate mean of $42.36$, and empirical 2.5th and 97.5th percentiles of $0.02$ and $0.66$. Calculate the bootstrap standard error and percentile interval.

Method. Use the sample standard deviation of the 1,000 bootstrap replicates and their percentile interval. Tau-b is recomputed in each resample to accommodate ties.

The centered sum of squares is given as $42.36$, so $$ \widehat{\operatorname{SE}}_{\mathrm{boot}} =\sqrt{\frac{42.36}{1000-1}} =\sqrt{\frac{42.36}{999}} =\sqrt{0.0424024} \approx 0.2059. $$

The 95% percentile interval is the supplied pair of quantiles, $[0.02,\,0.66]$. It is entirely positive, but its width indicates appreciable sampling uncertainty. Its interpretation depends on the paired observations being representative and independent under the sampling design; the interval does not account for model error or unobserved risks.

Centering check and common mistake. The replicate mean is $0.34$, whereas the original estimate is $0.36$. The sum of squared deviations from the original estimate is $$ \sum_{b=1}^{1000}(r_b^*-0.36)^2 =42.36+1000(0.34-0.36)^2 =42.36+0.40 =42.76. $$ Using $42.76/999$ as the sample variance would wrongly treat deviations from the original estimate as deviations from the replicate mean. The correct sample standard error uses $42.36/999$. A variance estimate centered at the original estimate would instead divide $42.76$ by $1000$, and would be a different quantity.

Ties, Small Samples, and Practical Limits

Ties change the sampling distribution. The no-tie null standard error for Spearman’s correlation and the displayed Kendall formula should not be applied mechanically to tied data. Spearman correlation with ties uses average ranks, while Kendall tau-b adjusts for ties in the two variables. Their variances depend on the tie pattern. For a null test, use a tie-aware method such as a permutation procedure that preserves the observed marginal values. For interval estimation, a paired bootstrap can be useful, but should be treated cautiously when the sample is small or the data contain extensive ties.

In a small sample, bootstrap resamples can contain only a limited set of distinct observations and may poorly represent the population’s possible rank patterns. Many ties can make this limitation more severe. Report the sample size, the tie handling, the uncertainty method, and the interval or standard error; do not present extra decimal places as if they remove sampling uncertainty.

Do not confuse a test with an interval. A null standard error evaluates variability under independence; it is not automatically an interval standard error around a nonzero estimate.
Preserve pairs when resampling. Resampling each variable independently erases the dependence being estimated.
Recompute ranks in every resample. Do not hold the original ranks fixed after resampling observations.
Respect the sampling design. If observations are dependent across time or clusters, an ordinary pair bootstrap may understate uncertainty; use a design-appropriate procedure.

Key takeaway: Use untied null standard errors for their stated purpose: testing independence under the corresponding assumptions. For uncertainty around an estimated rank association, a paired bootstrap can provide an approximate standard error or interval, but ties, small samples, and the sampling design must guide interpretation.

Check Your Understanding

For each problem, identify the assumptions before calculating or choosing an uncertainty method.

For $n=20$ independent continuous pairs with no ties, calculate the null standard error of Kendall’s tau using the stated large-sample formula. (Answer: $\sqrt{90/3420}\approx0.1622$)
For $n=17$ independent untied pairs, the sample Spearman correlation is $0.45$. Calculate its null standard error and standardized statistic. (Answer: standard error $0.2500$; $z=1.80$)
A paired bootstrap has $B=400$ replicates, with sum of squared deviations from their replicate mean equal to $15.96$. Find the sample standard error. (Answer: $\sqrt{15.96/399}=0.2000$)
A data set contains many tied ordinal scores. Should the untied Kendall null standard error be used without adjustment? State a suitable approach for a null test. (Answer: No; use a tie-aware procedure such as a permutation test preserving the observed marginal values.)
A time series of monthly paired losses is resampled as independent individual pairs to estimate uncertainty in Spearman correlation. What is the main concern? (Answer: Serial dependence is ignored; ordinary pair resampling may misstate uncertainty, so use a method that reflects the time-series sampling structure.)

What You'll Learn