Invariance of Partial and Multiple Correlations

Why Coordinate Changes Need Not Change Association

A set of controls can be recorded in different coordinates without changing the information in their linear span. For example, replacing two controls by their sum and difference is an invertible change of coordinates: neither control direction is lost. Partial and multiple correlations depend on what can be linearly explained by the controls, so they should not depend on which such coordinates are used. Target variables behave differently: reversing the direction of one target can reverse a signed correlation.

The distinctions are important. A nonsingular transformation of controls preserves the control space; a transformation of a target changes the scale or orientation of the quantity whose association is being summarized. We establish these facts for centered variables with finite second moments.

Definitions and Conditions

Definition: Let \(X\) and \(Y\) be centered random variables and let \(Z=(Z_1,\ldots,Z_q)^\mathsf{T}\) be a centered control vector with positive-definite covariance matrix \(\Sigma_Z=\operatorname{Cov}(Z)\). The linear projection residuals are \(e_X=X-\operatorname{Cov}(X,Z)\Sigma_Z^{-1}Z\) and \(e_Y=Y-\operatorname{Cov}(Y,Z)\Sigma_Z^{-1}Z\). If both residual variances are positive, the partial correlation of \(X\) and \(Y\) given the controls \(Z\) is \(\rho_{XY\cdot Z}=\operatorname{Cov}(e_X,e_Y)/\sqrt{\operatorname{Var}(e_X)\operatorname{Var}(e_Y)}\).

Centering is convenient because an intercept is then unnecessary. With uncentered data, the same definitions apply after including an intercept or centering each variable. Positive definiteness of \(\Sigma_Z\) makes the displayed projection coefficients unique. Positive residual variances are needed for the partial correlation to be defined.

Definition: For a centered scalar target \(Y\) with positive variance and controls \(Z\) as above, the multiple correlation of \(Y\) with \(Z\) is \(R_{Y\cdot Z}=\sqrt{\operatorname{Var}(\widehat Y)/\operatorname{Var}(Y)}\), where \(\widehat Y=\operatorname{Cov}(Y,Z)\Sigma_Z^{-1}Z\) is the linear projection of \(Y\) on the controls. Thus \(R_{Y\cdot Z}\) is nonnegative.

The multiple correlation measures the length of the projection relative to the total standard deviation of the target. It is not a signed correlation with a particular control. A useful equivalent expression follows by writing \(c=\operatorname{Cov}(Z,Y)\):

R_{Y\cdot Z}^{\,2} =\frac{c^\mathsf{T}\Sigma_Z^{-1}c}{\operatorname{Var}(Y)}.

Invariance Under Nonsingular Control Transformations

Let \(W=AZ\), where \(A\) is a nonsingular \(q\times q\) matrix. Since \(A\) is invertible, \(Z=A^{-1}W\), so \(Z\) and \(W\) generate exactly the same linear combinations. The projection itself, not just its length, therefore remains unchanged.

Theorem: Suppose \(X,Y,Z\) satisfy the conditions above and \(W=AZ\) for a nonsingular matrix \(A\). Then the residuals of \(X\) and \(Y\) after projection on \(W\) equal their residuals after projection on \(Z\). Consequently \(\rho_{XY\cdot W}=\rho_{XY\cdot Z}\). For any centered scalar target \(Y\) with positive variance, \(R_{Y\cdot W}=R_{Y\cdot Z}\).

Proof for partial correlation. Write \(b_X=\operatorname{Cov}(Z,X)\). The covariance matrix and target covariance for \(W\) are \(\operatorname{Cov}(W)=A\Sigma_ZA^\mathsf{T}\) and \(\operatorname{Cov}(W,X)=A b_X\). The projection of \(X\) on \(W\) is therefore

\operatorname{Cov}(X,W)\operatorname{Cov}(W)^{-1}W =b_X^\mathsf{T}A^\mathsf{T}(A\Sigma_ZA^\mathsf{T})^{-1}AZ =b_X^\mathsf{T}\Sigma_Z^{-1}Z.

Here \((A\Sigma_ZA^\mathsf{T})^{-1}=(A^\mathsf{T})^{-1}\Sigma_Z^{-1}A^{-1}\), so the middle expression reduces as shown. It is exactly the projection of \(X\) on \(Z\). Repeating the calculation with \(Y\) gives the same equality of its projections. Both residuals, and hence their covariance and variances, are unchanged. The partial correlations are equal. \(\square\)

Proof for multiple correlation. Put \(c=\operatorname{Cov}(Z,Y)\). The covariance of \(W\) with \(Y\) is \(Ac\), so the squared multiple correlation using \(W\) is

\frac{(Ac)^\mathsf{T}(A\Sigma_ZA^\mathsf{T})^{-1}(Ac)} {\operatorname{Var}(Y)} = \frac{c^\mathsf{T}\Sigma_Z^{-1}c}{\operatorname{Var}(Y)} =R_{Y\cdot Z}^{\,2}.

Both multiple correlations are defined to be nonnegative, so equality of their squares implies equality of the correlations. \(\square\)

Worked Example: Partial Correlation After Replacing Controls by Sums and Differences

Let \(Z_1,Z_2\) be independent, centered variables with variance 1. Let \(U,V\) be centered variables independent of \(Z_1,Z_2\), with \(\operatorname{Var}(U)=1\), \(\operatorname{Var}(V)=4\), and \(\operatorname{Cov}(U,V)=1\). Define \(X=Z_1+2Z_2+U\) and \(Y=2Z_1-Z_2+V\). The residuals after controlling for \(Z=(Z_1,Z_2)^\mathsf{T}\) are \(U\) and \(V\), so

\rho_{XY\cdot Z} =\frac{1}{\sqrt{1\cdot4}}=\frac12.

Now use the transformed controls \(W_1=Z_1+Z_2\) and \(W_2=Z_1-Z_2\). Their transformation matrix is \(A=\begin{pmatrix}1&1\\1&-1\end{pmatrix}\), whose determinant is \(-2\), so it is nonsingular. The original controls can be recovered as \(Z_1=(W_1+W_2)/2\) and \(Z_2=(W_1-W_2)/2\). Thus they span the same control space, and the residuals remain \(U\) and \(V\). The partial correlation is still \(1/2\).

Worked Example: Multiple Correlation in New Control Coordinates

Let \(Z_1,Z_2\) be independent \(N(0,1)\) variables, and let \(\varepsilon\sim N(0,3)\) be independent of them. Set \(Y=Z_1+2Z_2+\varepsilon\). Then \(\operatorname{Var}(Y)=1+4+3=8\), while the projection of \(Y\) on \(Z\) is \(Z_1+2Z_2\), with variance \(1+4=5\). Hence \(R_{Y\cdot Z}=\sqrt{5/8}\).

Transform the controls to \(W_1=2Z_1+Z_2\), \(W_2=Z_1-Z_2\). The matrix \(A=\begin{pmatrix}2&1\\1&-1\end{pmatrix}\) has determinant \(-3\), so this is nonsingular. Direct substitution gives \(Z_1+2Z_2=W_1-W_2\). The projection is still the same random variable, now expressed in the new coordinates; its variance remains 5. The target variance remains 8, so \(R_{Y\cdot W}=\sqrt{5/8}\), as required.

Transforming Targets: Signs and Scale

For a scalar target, multiplication by a nonzero constant changes its orientation if that constant is negative. Adding a constant does not affect centered covariances or variances. Let \(X^*=aX+d\) and \(Y^*=bY+g\), with \(a,b\ne0\), while leaving the controls unchanged. Their projection residuals are \(a e_X\) and \(b e_Y\): linear projection reproduces the constant through the intercept convention, or equivalently the centered variables are scaled by \(a\) and \(b\). Thus

\rho_{X^*Y^*\cdot Z} =\frac{ab\,\operatorname{Cov}(e_X,e_Y)} {\sqrt{a^2\operatorname{Var}(e_X)b^2\operatorname{Var}(e_Y)}} =\operatorname{sign}(ab)\rho_{XY\cdot Z}.

Consequently, reversing exactly one target reverses the sign of the partial correlation; reversing both leaves it unchanged. Positive changes of scale do not change it. If only the scalar target \(Y\) is transformed to \(Y^*=bY+g\), its explained variance and total variance are both multiplied by \(b^2\). Therefore the nonnegative multiple correlation is unchanged: \(R_{Y^*\cdot Z}=R_{Y\cdot Z}\). The fitted regression coefficients do change: multiplying the target by a negative number reverses their signs. That coefficient sign change does not make the multiple correlation negative.

Worked Example: Reversing a Target Changes a Partial Correlation’s Sign

Use the variables \(X,Y,Z_1,Z_2,U,V\) from the sums-and-differences example. Their partial correlation is \(1/2\). Define \(X^*=-3X+4\) and \(Y^*=2Y-1\). The centered residuals are \(e_{X^*}=-3U\) and \(e_{Y^*}=2V\). Their covariance is \((-3)(2)(1)=-6\), and their variances are \(9\) and \(16\). Hence

\rho_{X^*Y^*\cdot Z} =\frac{-6}{\sqrt{9\cdot16}} =-\frac12.

The sign changed because exactly one of the two scale factors was negative. In contrast, for the target model \(Y=Z_1+2Z_2+\varepsilon\) above, replacing \(Y\) by \(-2Y+7\) multiplies its explained variance \(5\) and total variance \(8\) by 4. The new multiple correlation is \(\sqrt{20/32}=\sqrt{5/8}\), unchanged, even though the projection coefficients are now \(-2\) and \(-4\).

Common Pitfall and Practical Interpretation

A common mistake is to treat every change in a control variable as an innocuous coordinate change. The invariance theorem requires the transformed controls to span exactly the original control space. A nonsingular transformation of the full control vector meets this condition. Dropping a control, duplicating a direction in a rank-deficient transformation, or adding a genuinely new control does not. In those cases the projection space may change, and so may the partial or multiple correlation.

Another mistake is to say that reversing the response makes the multiple correlation negative. By definition, \(R_{Y\cdot Z}\) is a nonnegative length ratio. Reversing the response reverses regression coefficients and signed correlations with individual controls, but not the multiple correlation. Partial correlation, by contrast, is signed and changes sign when exactly one of its target variables is reversed.

Key takeaway: Invertible changes of coordinates among the controls preserve their linear span, so they preserve both partial and multiple correlations. Target transformations can affect signs: a partial correlation changes sign when exactly one target is reversed, whereas a scalar-target multiple correlation remains nonnegative and unchanged by any nonzero rescaling.

Check Your Understanding

Use the projection and sign rules to answer the following questions.

Why does a nonsingular transformation of the controls leave each projection residual unchanged?
Suppose \(X\) and \(Y\) have partial correlation \(0.4\). What is the partial correlation of \(-X\) and \(3Y\) given the same controls?
If a scalar response is multiplied by \(-5\), what happens to its multiple correlation and its regression coefficients?
What property of the control transformation is necessary for the invariance result?
Why is it incorrect to describe the multiple correlation as negative after reversing the response?