Comparing Correlations from Independent Samples

Why Compare Correlations?

A correlation can describe how strongly two measurements vary together in one population, but many studies ask whether that association differs across groups. For example, a researcher may compare the correlation between a biomarker and disease severity in a treatment group with the corresponding correlation in a control group. The samples must be independent for the procedure developed here: the people in one group are not also contributing observations to the other.

The comparison concerns two population correlations, not merely the numerical difference between two sample correlations. Since sample correlations have a sampling distribution that depends on the underlying correlation, comparing them directly with a standard error designed for means is not appropriate. Fisher’s transformation provides a useful approximate normal scale for the comparison.

Fisher’s Transformation and Its Sampling Approximation

Definition: For a correlation \(r\) with \(-1<r<1\), its Fisher-transformed value is \(z_r=\operatorname{atanh}(r)=\frac{1}{2}\log\!\left(\frac{1+r}{1-r}\right)\). For a population correlation \(\rho\), write \(\zeta=\operatorname{atanh}(\rho)\). The inverse transformation is \(\rho=\tanh(\zeta)\).

Suppose group \(j\), for \(j=1,2\), consists of \(n_j\) independent observations of a pair of variables drawn from a bivariate normal population with population correlation \(\rho_j\). Let \(r_j\) be the usual sample Pearson correlation for that group. For samples large enough for the Fisher approximation to be useful, the transformed sample correlation is approximately normal:

\operatorname{atanh}(r_j)\ \approx\ N\!\left(\operatorname{atanh}(\rho_j),\frac{1}{n_j-3}\right).

The variance approximation \(1/(n_j-3)\) is the standard Fisher-\(z\) approximation. It is an approximation, not an exact finite-sample distribution. It is most useful when observations within each group are independent, the paired measurements are reasonably well described by a bivariate normal distribution, and the sample sizes are not very small. Correlations extremely close to \(1\) or \(-1\), severe departures from bivariate normality, and influential observations can make the approximation unreliable.

Because the two groups are independent, their transformed sample correlations are independent as well. This independence is what lets us add their variances when forming a difference. If the same people contribute to both correlations, this calculation does not apply.

Testing Equality of Two Population Correlations

The null hypothesis of interest is \(H_0:\rho_1=\rho_2\). Since the Fisher transformation is strictly increasing, this is equivalent to \(H_0:\operatorname{atanh}(\rho_1)=\operatorname{atanh}(\rho_2)\). The approximate standard error for the difference of transformed sample correlations is

SE_z=\sqrt{\frac{1}{n_1-3}+\frac{1}{n_2-3}}.

Test statistic: For testing \(H_0:\rho_1=\rho_2\), use \(Z=\frac{\operatorname{atanh}(r_1)-\operatorname{atanh}(r_2)} {\sqrt{1/(n_1-3)+1/(n_2-3)}}\). Under the null, \(Z\) is approximately standard normal. For a two-sided test, the approximate \(p\)-value is \(2\{1-\Phi(|Z|)\}\), where \(\Phi\) is the standard normal distribution function.

The test follows directly from the sampling approximation. Under the null, both transformed sample correlations have the same approximate mean, so their difference has approximate mean zero. Independence means that the variance of the difference is the sum of the two variances. Standardizing by the square root of that sum gives the stated statistic.

Worked Example: Testing Two Group Correlations

In an independent treatment group of \(n_1=30\), the sample correlation between a biomarker and symptom severity is \(r_1=0.60\). In an independent control group of \(n_2=35\), the corresponding sample correlation is \(r_2=0.30\). Test the two-sided null hypothesis that the population correlations are equal.

First transform both sample correlations:

\operatorname{atanh}(0.60)=\frac12\log(4)=0.693147,\qquad \operatorname{atanh}(0.30)=\frac12\log\!\left(\frac{1.30}{0.70}\right)=0.309520.

Their transformed difference is \(0.693147-0.309520=0.383627\). The standard error is

SE_z=\sqrt{\frac1{30-3}+\frac1{35-3}} =\sqrt{\frac1{27}+\frac1{32}} =\sqrt{0.068287}=0.261318.

Thus \(Z=0.383627/0.261318=1.468\), giving a two-sided approximate \(p\)-value of \(2\{1-\Phi(1.468)\}\approx0.142\). At the 0.05 level, the data do not provide sufficient evidence that the population correlations differ. This conclusion is not evidence that the correlations are exactly equal; the estimate is imprecise enough that equality remains plausible.

Confidence Intervals for the Difference in Raw Correlations

A confidence interval for \(\operatorname{atanh}(\rho_1)-\operatorname{atanh}(\rho_2)\) is not automatically a confidence interval for \(\rho_1-\rho_2\). The inverse transformation applies to one correlation at a time, and in general the difference of two transformed values cannot be converted back by applying \(\tanh\) to that difference.

For a confidence interval directly on the raw difference, use the delta method. Let \(\widehat{\zeta}_j=\operatorname{atanh}(r_j)\), so that \(r_j=\tanh(\widehat{\zeta}_j)\). The derivative of \(\tanh(\zeta)\) is \(1-\tanh^2(\zeta)\). Applying this derivative to each transformed estimate gives the following estimated variance for the raw difference:

\widehat{\operatorname{Var}}(r_1-r_2) \ \approx\ \frac{(1-r_1^2)^2}{n_1-3} +\frac{(1-r_2^2)^2}{n_2-3}.

The two terms add because the samples are independent. A two-sided approximate \(100(1-\alpha)\%\) confidence interval for \(\rho_1-\rho_2\) is therefore

(r_1-r_2)\ \pm\ z_{1-\alpha/2} \sqrt{\frac{(1-r_1^2)^2}{n_1-3} +\frac{(1-r_2^2)^2}{n_2-3}},

where \(z_{1-\alpha/2}\) is the standard normal quantile. This is a first-order, plug-in delta-method interval: its center and standard error estimate the raw difference and its sampling uncertainty. As with the test, its accuracy depends on the sampling approximation and is not exact in small samples.

Worked Example: A Confidence Interval for the Raw Difference

An independent sample of \(50\) patients has a sample correlation of \(r_1=0.70\) between two clinical measures. A second independent sample of \(60\) patients has \(r_2=0.40\). Construct an approximate 95% confidence interval for \(\rho_1-\rho_2\).

The estimated difference is \(r_1-r_2=0.70-0.40=0.30\). The estimated standard error from the delta method is

\begin{aligned} SE_{\mathrm{diff}} &=\sqrt{\frac{(1-0.70^2)^2}{50-3} +\frac{(1-0.40^2)^2}{60-3}}\\ &=\sqrt{\frac{0.51^2}{47}+\frac{0.84^2}{57}}\\ &=\sqrt{0.005534+0.012379} =\sqrt{0.017913}=0.13384. \end{aligned}

Using \(z_{0.975}=1.96\), the margin of error is \(1.96(0.13384)=0.2623\). The interval is

0.30\pm0.2623=[0.0377,\ 0.5623].

The interval is entirely above zero, suggesting a larger population correlation in the first group at the approximate 5% two-sided level. The interval is fairly wide, so the size of that difference remains uncertain.

Deriving the Confidence Interval

The confidence interval formula follows by applying the delta method to the transformed sampling distributions and then using a normal approximation for the difference. The derivation also makes clear which uncertainty is being estimated.

Start with the transformed estimates.
For each independent sample, \(\widehat{\zeta}_j=\operatorname{atanh}(r_j)\) has approximate variance \(1/(n_j-3)\).

Transform back locally.
Since \(r_j=\tanh(\widehat{\zeta}_j)\), the derivative at the estimate is \(1-r_j^2\). The delta method therefore gives \(\widehat{\operatorname{Var}}(r_j)\approx(1-r_j^2)^2/(n_j-3)\).

Form the difference.
The estimates are independent across groups, so \(\widehat{\operatorname{Var}}(r_1-r_2)\) is the sum of their estimated variances. Standardizing the estimated difference gives an approximately standard normal statistic.

Invert the standardized statement.
An approximately \(1-\alpha\) probability statement is \(-z_{1-\alpha/2}\le ((r_1-r_2)-(\rho_1-\rho_2))/SE_{\mathrm{diff}}\le z_{1-\alpha/2}\). Rearranging for \(\rho_1-\rho_2\) gives the stated interval.

The derivation is approximate at two points: the Fisher-\(z\) sampling distribution is approximated as normal, and the delta method uses a local linear approximation to \(\tanh\). These approximations explain why the interval should not be interpreted as an exact small-sample guarantee.

Worked Example: A Significant Difference in Larger Samples

Suppose independent groups of \(n_1=80\) and \(n_2=75\) have sample correlations \(r_1=0.55\) and \(r_2=0.25\), respectively. Test equality with a two-sided alternative.

The transformed values are \(\operatorname{atanh}(0.55)=0.618381\) and \(\operatorname{atanh}(0.25)=0.255413\). Their difference is \(0.362968\). The standard error is

SE_z=\sqrt{\frac1{77}+\frac1{72}} =\sqrt{0.012987+0.013889} =\sqrt{0.026876}=0.16394.

Therefore \(Z=0.362968/0.16394=2.214\), with approximate two-sided \(p\)-value \(2\{1-\Phi(2.214)\}\approx0.0268\). At the 0.05 level, this provides evidence that the population correlations differ. The test establishes evidence of a difference, not that one particular group causes the other group’s association.

Common Mistakes and Practical Interpretation

Comparing the correlations without accounting for sample size. A difference such as \(0.10\) can be more or less informative depending on the two sample sizes. Use the standard error, not the raw difference alone.
Using the raw difference with the Fisher-\(z\) standard error. The standard error \(\sqrt{1/(n_1-3)+1/(n_2-3)}\) belongs to the transformed difference. For an interval on \(\rho_1-\rho_2\), use the delta-method standard error that includes the factors \((1-r_j^2)^2\).
Treating a transformed-difference interval as a raw-difference interval. A confidence interval for \(\operatorname{atanh}(\rho_1)-\operatorname{atanh}(\rho_2)\) describes a different parameter. State the scale of the interval explicitly.
Using this independent-samples formula for overlapping samples. If the two correlations share participants or a measured variable, their estimates are dependent. Adding the two variances as if they were independent gives the wrong uncertainty.
Interpreting a non-significant test as proof of equality. A failure to reject may reflect low precision. A confidence interval helps show which differences remain compatible with the data.

In reporting results, identify the two groups, give both sample correlations and sample sizes, state the estimated raw difference and its confidence interval, and report the test statistic and approximate \(p\)-value when testing equality. This keeps the inferential scale and the practical interpretation visible.

Check Your Understanding

Use the assumptions and formulas from this tutorial to answer the following.

Why is the standard error for the Fisher-transformed difference the square root of a sum of two variance terms?
What parameter does a confidence interval for \(\operatorname{atanh}(\rho_1)-\operatorname{atanh}(\rho_2)\) describe, and why is it not automatically an interval for \(\rho_1-\rho_2\)?
For \(r_1=0.50\), \(n_1=40\), \(r_2=0.20\), and \(n_2=45\), write the Fisher-\(z\) test statistic for equality. You may leave the final arithmetic unevaluated.
Which assumptions make the independent-samples comparison inappropriate when the same participants contribute to both correlations?
What does a two-sided test with a non-significant result allow you to conclude—and what does it not establish?