Deriving Partial Correlation by Residual Projection

Why Project Both Variables?

A partial correlation describes the linear association between two variables after their linear relationships with a shared set of controls have been removed. The geometric idea is simple: project each variable onto the space spanned by the controls, then compare the two parts left over. The projection formulation makes precise why both variables must be residualized against the same controls, and it connects the resulting correlation to a coefficient in a multiple regression.

We work with observed vectors of length $n$. The results are algebraic and require no probability model. Assume the control design matrix includes an intercept, has full column rank, and that neither variable is completely explained by the controls. These conditions ensure that the formulas below are defined and that the residual vectors have mean zero.

Projection onto the Control Space

Definition: Let $Z$ be an $n\times q$ matrix of control-variable columns, with full column rank. Its column space $\mathcal{C}(Z)$ is the set of all vectors $Z\gamma$, where $\gamma\in\mathbb{R}^q$. The orthogonal projection matrix onto this space and the residual-maker matrix are $$ P_Z=Z(Z'Z)^{-1}Z',\qquad M_Z=I-P_Z. $$ For observed vectors $x,y\in\mathbb{R}^n$, the residuals after fitting the controls are $e_x=M_Zx$ and $e_y=M_Zy$.

Here $Z'$ denotes the transpose. Full column rank makes $Z'Z$ invertible. The vector $P_Zx$ is the least-squares fit of $x$ using the controls, while $M_Zx=x-P_Zx$ is the part left after that fit. Since the intercept is among the columns of $Z$, both residual vectors are orthogonal to the vector of ones and therefore have sample mean zero.

The sample partial correlation of $x$ and $y$ given these controls is the ordinary sample correlation of their two residual vectors. In the projection notation, this is

r_{xy\cdot Z} = \frac{e_x'e_y}{\sqrt{(e_x'e_x)(e_y'e_y)}} = \frac{x'M_Zy}{\sqrt{(x'M_Zx)(y'M_Zy)}}.

The denominator is positive exactly when neither variable lies entirely in the control space. Because the residuals have mean zero, their sample covariance and variances use a common divisor $n-1$, which cancels in the correlation. Thus the displayed expression is the usual Pearson correlation of the two residual vectors.

Worked Example: Projecting Two Variables off a Control

Take $n=5$, with an intercept and one control $z$. Define

z=(-2,-1,0,1,2)',\quad a=(1,-2,0,2,-1)',\quad c=(2,-1,-2,-1,2)'.

Let $x=2z+a$ and $y=-z+a+c$. The controls are the columns of $Z=[\mathbf{1},z]$, where $\mathbf{1}$ is the vector of five ones. First check the needed orthogonality: $\mathbf{1}'z=\mathbf{1}'a=\mathbf{1}'c=0$, $z'a=z'c=0$, and $a'c=0$. Also $z'z=10$, $a'a=10$, and $c'c=14$. Thus $a$ and $c$ are already orthogonal to both control columns.

The projection of $x$ onto the controls is $2z$, since its intercept coefficient is zero and $z'x/(z'z)=2$. The projection of $y$ is $-z$, since $z'y/(z'z)=-1$ and its intercept coefficient is zero. Therefore

e_x=a,\qquad e_y=a+c.

Their inner product is $a'(a+c)=a'a+a'c=10$. Their squared lengths are $e_x'e_x=10$ and $e_y'e_y=(a+c)'(a+c)=10+14=24$. Hence

r_{xy\cdot Z}=\frac{10}{\sqrt{10\cdot24}} =\frac{5}{\sqrt{60}} =\sqrt{\frac{5}{12}} \approx 0.6455.

The positive value reflects the shared $a$ component in the residuals. The portions involving $z$ do not contribute to the partial correlation because they are removed by projection onto the control space.

The Projection Formula for Partial Correlation

Theorem: Suppose $Z$ has full column rank and contains an intercept, and suppose $M_Zx\ne 0$ and $M_Zy\ne 0$. The sample partial correlation of $x$ and $y$ given the controls in $Z$ is $$ r_{xy\cdot Z} = \frac{x'M_Zy}{\sqrt{(x'M_Zx)(y'M_Zy)}}. $$ Equivalently, it is the cosine of the angle between the two residual vectors $M_Zx$ and $M_Zy$.

Proof. From its definition, $P_Z=Z(Z'Z)^{-1}Z'$ is symmetric. It is also idempotent:

P_Z^2 =Z(Z'Z)^{-1}Z'Z(Z'Z)^{-1}Z' =Z(Z'Z)^{-1}Z' =P_Z.

Thus $M_Z=I-P_Z$ is symmetric and idempotent as well: $M_Z'=M_Z$ and $M_Z^2=M_Z$. The residual inner product can therefore be written as

$$ e_x'e_y=(M_Zx)'(M_Zy) =x'M_Z'M_Zy =x'M_Z^2y =x'M_Zy. $$

The same calculation gives $e_x'e_x=x'M_Zx$ and $e_y'e_y=y'M_Zy$. Since the intercept vector lies in $\mathcal{C}(Z)$, $M_Z\mathbf{1}=0$. Symmetry then gives $\mathbf{1}'e_x=\mathbf{1}'M_Zx=(M_Z\mathbf{1})'x=0$, and likewise $\mathbf{1}'e_y=0$. The residual means are zero, so their Pearson correlation is their inner product divided by the product of their lengths. Substituting the three identities proves the formula. The cosine interpretation follows from the definition of the angle between nonzero vectors. $\square$

Worked Example: No Controls Beyond an Intercept

Consider $x=(1,2,4,5)'$ and $y=(2,1,3,6)'$, with the intercept as the only control. Projection onto the intercept subtracts each variable’s sample mean. The means are $\bar{x}=3$ and $\bar{y}=3$, so

e_x=(-2,-1,1,2)',\qquad e_y=(-1,-2,0,3)'.

The residual cross-product is $e_x'e_y=2+2+0+6=10$. The squared lengths are $e_x'e_x=4+1+1+4=10$ and $e_y'e_y=1+4+0+9=14$. Therefore the partial correlation given only an intercept is

r_{xy\cdot \mathbf{1}}=\frac{10}{\sqrt{10\cdot14}} =\sqrt{\frac{5}{7}} \approx 0.8452.

With no substantive controls, residualizing against the intercept simply centers each vector. This calculation is the ordinary sample correlation of $x$ and $y$.

Equivalence to the Frisch-Waugh-Lovell Construction

The Frisch-Waugh-Lovell theorem states that the coefficient of $x$ in a least-squares regression of $y$ on the controls and $x$ can be found by first removing the controls from both $y$ and $x$, then regressing the residualized $y$ on the residualized $x$. The following derivation establishes that statement for the present setup.

Theorem: Suppose $Z$ has full column rank and $x'M_Zx>0$. In the least-squares regression of $y$ on the columns of $Z$ and on $x$, the coefficient of $x$ is $$ \hat{\beta}_x=\frac{x'M_Zy}{x'M_Zx} =\frac{e_x'e_y}{e_x'e_x}. $$ This is also the least-squares slope in the regression of $e_y$ on $e_x$ with no intercept.

Proof. For any proposed coefficient $\beta$ on $x$, the remaining fitted vector must be $Z\gamma$ for some $\gamma$. The least-squares criterion is $\|y-x\beta-Z\gamma\|^2$. At fixed $\beta$, the best fit from $Z$ is the projection $P_Z(y-x\beta)$, leaving residual $M_Z(y-x\beta)$. Thus minimizing over both coefficients reduces to minimizing over $\beta$ the squared length

\|M_Z(y-x\beta)\|^2 =(e_y-\beta e_x)'(e_y-\beta e_x).

Expanding gives $e_y'e_y-2\beta e_x'e_y+\beta^2e_x'e_x$. Since $e_x'e_x=x'M_Zx>0$, this quadratic has a unique minimum. Differentiating with respect to $\beta$ and setting the derivative to zero yields

-2e_x'e_y+2\beta e_x'e_x=0, \qquad \hat{\beta}_x=\frac{e_x'e_y}{e_x'e_x}.

The projection identities proved above give $e_x'e_y=x'M_Zy$ and $e_x'e_x=x'M_Zx$, establishing the stated expression. The ordinary least-squares slope through the origin when regressing $e_y$ on $e_x$ minimizes the same quadratic and therefore has the same value. $\square$

Worked Example: Recovering the Multiple-Regression Coefficient

Use the vectors from the first worked example. There $e_x=a$ and $e_y=a+c$, with $e_x'e_y=10$ and $e_x'e_x=10$. The coefficient of $x$ in the regression of $y$ on an intercept, $z$, and $x$ is therefore

\hat{\beta}_x=\frac{e_x'e_y}{e_x'e_x} =\frac{10}{10}=1.

This can also be checked directly. The data were constructed so that $x=2z+a$ and $y=-z+a+c$. Since $a$ and $c$ are orthogonal to the controls and to each other, the multiple-regression fit uses coefficient $1$ for $x$, coefficient $-3$ for $z$, and intercept zero: $y=x-3z+c$. The residual $c$ is orthogonal to $x$ and $z$, so these coefficients satisfy the least-squares normal equations.

The partial correlation and the regression coefficient are related but not identical. In this example the residual-regression slope is $1$, while the partial correlation is $\sqrt{5/12}$. The slope retains the measurement scales of $x$ and $y$; the correlation standardizes both residual vectors to unit length.

Conditions and a Common Pitfall

The projection formulation requires care about rank and nonzero residual variation. If $Z$ is rank deficient, the inverse in the displayed formula for $P_Z$ does not exist; one can instead define the orthogonal projection onto its column space, but the full-rank formulas given here assume the stated condition. If $M_Zx=0$ or $M_Zy=0$, the corresponding variable is exactly in the control space, so its residual length is zero and the partial correlation is undefined.

A common mistake is to residualize only $y$, then correlate $x$ with that residual. That does not give the partial correlation: the original $x$ still contains its component explained by the controls. The correction is to use $M_Zx$ and $M_Zy$ together. The Frisch-Waugh-Lovell theorem makes the same point for regression: the coefficient of $x$ is obtained by residualizing both the response and the predictor against the controls before fitting the residual regression.

Key takeaway: With a common full-rank control design that includes an intercept, partial correlation is the normalized inner product of the two projected residuals. The coefficient of the predictor in the full multiple regression is the unstandardized slope from regressing one residual on the other.

Check Your Understanding

Use the projection and residual-regression results to answer these questions.

What property of $M_Z$ allows $e_x'e_y$ to be written as $x'M_Zy$?
Why does including an intercept among the controls ensure that the residual vectors have mean zero?
What additional condition is needed for the Frisch-Waugh-Lovell coefficient formula to have a nonzero denominator?
Explain why residualizing only $y$ does not produce the partial correlation of $x$ and $y$ given $Z$.
How does the coefficient from the residual regression differ in interpretation from the partial correlation?