Partial Correlation in Three-Variable Gaussian Models

Why Correlation Can Change After Adjustment

Two variables can be correlated because they are associated with one another, because both are related to a third variable, or through a combination of these patterns. Partial correlation measures the remaining linear association between two variables after removing from each its linear prediction based on a third. It is therefore a standardized measure of association between residuals, not a measure of every possible kind of dependence.

For a jointly Gaussian triple, the linear predictions are also conditional expectations. This makes partial correlation especially interpretable: it describes the correlation between the two variables after their shared linear relation with the control variable has been removed. The value can be smaller than the marginal correlation, larger in magnitude, or even have the opposite sign.

Definition: Let $(X_1,X_2,X_3)$ be a jointly Gaussian random vector with positive-definite covariance matrix $\Sigma=(\sigma_{ij})$. The partial covariance of $X_1$ and $X_2$ controlling for $X_3$ is $$ \sigma_{12\mathbin{\cdot}3} = \sigma_{12}-\frac{\sigma_{13}\sigma_{23}}{\sigma_{33}}. $$ The partial correlation is the correlation of their linear-regression residuals: $$ \rho_{12\mathbin{\cdot}3} = \frac{\sigma_{12}-\sigma_{13}\sigma_{23}/\sigma_{33}} {\sqrt{\left(\sigma_{11}-\sigma_{13}^2/\sigma_{33}\right) \left(\sigma_{22}-\sigma_{23}^2/\sigma_{33}\right)}}. $$ Positive definiteness ensures the residual variances in the denominator are positive.

Residuals and the Partial-Correlation Formula

Write $\mu_i=E[X_i]$. The best linear predictor of $X_1$ from an intercept and $X_3$ is $\mu_1+(\sigma_{13}/\sigma_{33})(X_3-\mu_3)$, and the corresponding predictor of $X_2$ is $\mu_2+(\sigma_{23}/\sigma_{33})(X_3-\mu_3)$. Define the centered residuals $$ U=X_1-\mu_1-\frac{\sigma_{13}}{\sigma_{33}}(X_3-\mu_3), \qquad V=X_2-\mu_2-\frac{\sigma_{23}}{\sigma_{33}}(X_3-\mu_3). $$ These residuals have zero covariance with $X_3$. Their covariance and variances give the partial covariance and correlation.

Theorem: For a jointly Gaussian triple with positive-definite covariance matrix, $\operatorname{Cov}(U,V)=\sigma_{12\mathbin{\cdot}3}$, $\operatorname{Var}(U)=\sigma_{11}-\sigma_{13}^2/\sigma_{33}$, and $\operatorname{Var}(V)=\sigma_{22}-\sigma_{23}^2/\sigma_{33}$. Consequently, $\rho_{12\mathbin{\cdot}3}=\operatorname{Corr}(U,V)$.

To verify the covariance identity, expand using bilinearity and the definitions of $U$ and $V$:

\begin{aligned} \operatorname{Cov}(U,V) &=\sigma_{12} -\frac{\sigma_{23}}{\sigma_{33}}\sigma_{13} -\frac{\sigma_{13}}{\sigma_{33}}\sigma_{23} +\frac{\sigma_{13}\sigma_{23}}{\sigma_{33}^2}\sigma_{33}\\ &=\sigma_{12}-\frac{\sigma_{13}\sigma_{23}}{\sigma_{33}}. \end{aligned}

For the first residual variance, expand its variance:

\begin{aligned} \operatorname{Var}(U) &=\sigma_{11} -2\frac{\sigma_{13}}{\sigma_{33}}\sigma_{13} +\frac{\sigma_{13}^2}{\sigma_{33}^2}\sigma_{33}\\ &=\sigma_{11}-\frac{\sigma_{13}^2}{\sigma_{33}}. \end{aligned}

The same calculation with $X_2$ gives $\operatorname{Var}(V)=\sigma_{22}-\sigma_{23}^2/\sigma_{33}$. By definition, the correlation of $U$ and $V$ is their covariance divided by the square root of the product of their variances, which proves the formula. Positive definiteness of $\Sigma$ implies that no nonzero linear combination of the coordinates has zero variance. Each residual is a nonzero linear combination, so both residual variances are strictly positive. $\square$

If the variables are standardized, let $r_{ij}=\operatorname{Corr}(X_i,X_j)$. Then the formula takes the familiar form

\rho_{12\mathbin{\cdot}3} = \frac{r_{12}-r_{13}r_{23}} {\sqrt{(1-r_{13}^2)(1-r_{23}^2)}}.

The numerator is the partial covariance in standardized units. The denominator rescales by the residual standard deviations. The formula requires $|r_{13}|<1$ and $|r_{23}|<1$, conditions implied by positive definiteness.

Worked Example: Computing Partial Correlation From Correlations

Suppose a standardized Gaussian triple has $r_{12}=0.60$, $r_{13}=0.50$, and $r_{23}=0.40$. First check that these numbers can form a positive-definite correlation matrix. Its determinant is

$$ 1+2(0.60)(0.50)(0.40)-0.60^2-0.50^2-0.40^2 =1+0.24-0.36-0.25-0.16 =0.47>0. $$

The diagonal entries are positive, and the two-by-two principal minors are $1-0.60^2=0.64$, $1-0.50^2=0.75$, and $1-0.40^2=0.84$, all positive. Thus the correlation matrix is positive definite. Now substitute into the partial-correlation formula:

\rho_{12\mathbin{\cdot}3} = \frac{0.60-(0.50)(0.40)} {\sqrt{(1-0.50^2)(1-0.40^2)}} = \frac{0.40}{\sqrt{(0.75)(0.84)}} = \frac{0.40}{\sqrt{0.63}} \approx 0.504.

The marginal correlation is $0.60$, while the partial correlation is about $0.504$. Removing the linear association each variable has with $X_3$ reduces, but does not eliminate, their positive residual association.

Connection to the Precision Matrix

The precision matrix is the inverse covariance matrix, $\Omega=\Sigma^{-1}$. In a nondegenerate three-variable Gaussian model, its off-diagonal entry $\Omega_{12}$, together with its two corresponding diagonal entries, gives the partial correlation:

Theorem: If $\Sigma$ is a positive-definite $3\times3$ covariance matrix and $\Omega=\Sigma^{-1}$, then $$ \rho_{12\mathbin{\cdot}3} = -\frac{\Omega_{12}}{\sqrt{\Omega_{11}\Omega_{22}}}. $$

Here is a direct derivation for three variables. Let $D=\det(\Sigma)>0$. The cofactor formula for the inverse gives

\Omega_{12}=\frac{\sigma_{13}\sigma_{23}-\sigma_{12}\sigma_{33}}{D}, \qquad \Omega_{11}=\frac{\sigma_{22}\sigma_{33}-\sigma_{23}^2}{D}, \qquad \Omega_{22}=\frac{\sigma_{11}\sigma_{33}-\sigma_{13}^2}{D}.

Define $A=\sigma_{22}\sigma_{33}-\sigma_{23}^2$ and $B=\sigma_{11}\sigma_{33}-\sigma_{13}^2$. Since $\Sigma$ is positive definite, $A$ and $B$ are positive. Substitution yields

-\frac{\Omega_{12}}{\sqrt{\Omega_{11}\Omega_{22}}} = \frac{\sigma_{12}\sigma_{33}-\sigma_{13}\sigma_{23}}{\sqrt{AB}}.

On the other hand, the residual formula has numerator $(\sigma_{12}\sigma_{33}-\sigma_{13}\sigma_{23})/\sigma_{33}$ and denominator $\sqrt{(B/\sigma_{33})(A/\sigma_{33})}=\sqrt{AB}/\sigma_{33}$, since $\sigma_{33}>0$. Their ratio is the same expression. This proves the stated identity. $\square$

The minus sign matters: under this convention, a positive partial correlation corresponds to a negative $\Omega_{12}$. The identity is a compact way to calculate the partial correlation, but it does not change its definition: it remains the correlation of the two regression residuals.

Worked Example: Checking the Precision-Matrix Formula

For the correlation matrix in the previous example, the determinant is $D=0.47$. The cofactor formulas give

\Omega_{12}=\frac{(0.50)(0.40)-0.60}{0.47} =\frac{-0.40}{0.47}\approx -0.8511, $$ $$ \Omega_{11}=\frac{1-0.40^2}{0.47} =\frac{0.84}{0.47}\approx1.7872, \qquad \Omega_{22}=\frac{1-0.50^2}{0.47} =\frac{0.75}{0.47}\approx1.5957.

Therefore,

-\frac{\Omega_{12}}{\sqrt{\Omega_{11}\Omega_{22}}} \approx \frac{0.8511}{\sqrt{(1.7872)(1.5957)}} \approx\frac{0.8511}{1.689} \approx0.504.

This agrees, up to rounding, with the direct residual calculation. The agreement checks both the inverse-matrix identity and the sign convention.

When Controlling for a Variable Reverses the Sign

The partial covariance subtracts $\sigma_{13}\sigma_{23}/\sigma_{33}$ from the marginal covariance. If this removed component is larger than $\sigma_{12}$, the residual covariance changes sign. This is possible even when the covariance matrix is positive definite: the three correlations cannot be chosen arbitrarily, but positive definiteness does not prevent a sign reversal.

Worked Example: A Valid Sign Reversal

Consider a standardized Gaussian triple with $r_{12}=0.20$, $r_{13}=0.80$, and $r_{23}=0.70$. Check that the proposed correlation matrix is valid: its two-by-two principal minors are $0.96$, $0.36$, and $0.51$, and its determinant is

$$ 1+2(0.20)(0.80)(0.70)-0.20^2-0.80^2-0.70^2 =1+0.224-0.04-0.64-0.49 =0.054>0. $$

Thus the matrix is positive definite. The partial correlation is

\rho_{12\mathbin{\cdot}3} = \frac{0.20-(0.80)(0.70)} {\sqrt{(1-0.80^2)(1-0.70^2)}} = \frac{-0.36}{\sqrt{(0.36)(0.51)}} = \frac{-0.36}{\sqrt{0.1836}} \approx-0.840.

The marginal correlation between $X_1$ and $X_2$ is positive, but the residual correlation after controlling for $X_3$ is negative. The numerator shows why: the product $r_{13}r_{23}=0.56$, the part removed by adjustment, exceeds the marginal correlation $r_{12}=0.20$. This is a sign reversal, not a contradiction; the two correlations describe different associations.

Interpretation and Common Pitfall

For a jointly Gaussian triple, the residuals $U$ and $V$ are jointly Gaussian and each has zero covariance with $X_3$. The Gaussian regression result gives $E[X_i\mid X_3]=\mu_i+(\sigma_{i3}/\sigma_{33})(X_3-\mu_3)$, so these residuals are precisely the deviations from the conditional means. Their covariance is the conditional covariance, which does not depend on the observed value of $X_3$; dividing by their residual standard deviations gives the corresponding conditional correlation.

The Gaussian assumption is important for this conditional interpretation. Residualizing by a linear predictor and calculating a residual correlation are meaningful whenever the variables have finite second moments and the control variable has positive variance. But outside a Gaussian model, the conditional mean need not be linear, and residual correlation need not summarize the full conditional association. Even in a Gaussian model, partial correlation measures linear association after controlling for $X_3$; it does not by itself establish causation.

Common mistake: treating partial correlation as the marginal correlation. The marginal quantity is $r_{12}$. Partial correlation subtracts the component explained by the linear relationships with $X_3$ and rescales by the residual standard deviations.
Common mistake: omitting the negative sign in the precision formula. With $\Omega=\Sigma^{-1}$, the formula is $-\Omega_{12}/\sqrt{\Omega_{11}\Omega_{22}}$, not the same expression without the minus sign.
Common mistake: interpreting a sign reversal as an impossible covariance matrix. Check positive definiteness separately. A valid triple can have marginal and partial correlations with opposite signs, as the worked example demonstrates.

For calculation, first choose the desired representation: use the covariance formula when covariances are given, the correlation formula when variables are standardized, or the precision formula when the inverse covariance matrix is available. Then interpret the result as a standardized residual association, keeping the marginal and adjusted quantities distinct.

Check Your Understanding

Use the residual definition and formulas to answer these questions.

Write the partial-correlation formula for $X_1$ and $X_2$ controlling for $X_3$ in terms of covariances.
For a standardized triple with $r_{12}=0.30$, $r_{13}=0.40$, and $r_{23}=0.50$, calculate $\rho_{12\mathbin{\cdot}3}$.
State the precision-matrix formula for $\rho_{12\mathbin{\cdot}3}$, including its sign.
Why does positive definiteness ensure that the denominator in the partial-correlation formula is positive?
Can partial correlation be interpreted as the correlation of deviations from conditional means in a non-Gaussian model solely from the residual formula? Explain.