Why Nearly Redundant Controls Require Care
A partial correlation describes the linear association between two variables after removing their linear associations with a set of controls. When the controls are nearly redundant, their covariance matrix is close to singular. The projection used to remove their effects can then depend strongly on small changes in the covariance estimates. Consequently, two analyses based on nearly identical data—or on differently rounded covariance entries—can produce noticeably different partial correlations.
This is a numerical and data-sensitivity issue, not a claim that every partial correlation with correlated controls is invalid. The key diagnostic is whether the control covariance block has a very small eigenvalue relative to its largest one. A small eigenvalue means there is a direction in the control space with very little variation; inverting the block gives that direction a large weight.
The covariance-block formula for the residual covariance is \( \operatorname{Cov}(e_X,e_Y)=\sigma_{XY}-\Sigma_{XZ}\Sigma_{ZZ}^{-1}\Sigma_{ZY} \). The analogous formulas for the residual variances use \(X\) or \(Y\) in both positions. Thus instability in \(\Sigma_{ZZ}^{-1}\) can affect both the numerator and denominator of the partial correlation. The examples below show that effect directly.
What Near-Singularity Does to an Inverse
For a symmetric positive-definite control covariance matrix \(A\), let its smallest and largest eigenvalues be \(\lambda_{\min}\) and \(\lambda_{\max}\). Its spectral condition number is \(\kappa_2(A)=\lambda_{\max}/\lambda_{\min}\). When \(\lambda_{\min}\) is small, the inverse has a large eigenvalue \(1/\lambda_{\min}\) in the corresponding direction. This explains why small perturbations aligned with that direction can have a large effect.
Proof. Write \(A+E=A(I+A^{-1}E)\). Since \(\lVert A^{-1}E\rVert_2<1\), the Neumann series \(\sum_{k=0}^{\infty}(-A^{-1}E)^k\) converges in operator norm and is the inverse of \(I+A^{-1}E\). Hence \(A+E\) is invertible and \( \lVert(A+E)^{-1}\rVert_2\leq \lVert A^{-1}\rVert_2/(1-\lVert A^{-1}E\rVert_2) \). The identity \( (A+E)^{-1}-A^{-1}=-A^{-1}E(A+E)^{-1} \) then gives the stated bound by submultiplicativity of the spectral norm and \(\lVert A^{-1}E\rVert_2\leq\lVert A^{-1}\rVert_2\lVert E\rVert_2\). \(\square\)
The condition \(\lVert A^{-1}\rVert_2\lVert E\rVert_2<1\) is a sufficient bound, not a promise that every smaller perturbation causes a large change. It does show why a fixed-sized perturbation can be harmless for a well-conditioned block yet consequential for a nearly singular one. In partial correlation, the inverse is multiplied by estimated covariances with \(X\) and \(Y\), so perturbations to those covariances also matter.
Worked Examples of Sensitivity
Worked Example: A Small Covariance Change Alters the Partial Correlation
Let the two controls have covariance matrix \(A=\begin{pmatrix}1&0.9999\\0.9999&1\end{pmatrix}\). Compare two covariance structures that have the same control block, \(\operatorname{Var}(X)=\operatorname{Var}(Y)=1\), and \(\operatorname{Cov}(X,Y)=0.04480224\). In both, \(\operatorname{Cov}(X,Z)=(0.200,0.199)\); in the first, \(\operatorname{Cov}(Y,Z)=(0.200,0.199)\), and in the second it is \((0.200,0.198)\). The second structure changes just one control covariance by \(0.001\).
For any two vectors \(u=(u_1,u_2)\) and \(v=(v_1,v_2)\), write \(s_u=u_1+u_2\) and \(d_u=u_1-u_2\), and likewise for \(v\). The sum and difference directions are eigenvectors of \(A\), giving \( u^\mathsf{T}A^{-1}v=s_us_v/[2(1+0.9999)]+d_ud_v/[2(1-0.9999)] \). Here the two denominators are \(3.9998\) and \(0.0002\), respectively.
For the first structure, \(s_X=s_Y=0.399\) and \(d_X=d_Y=0.001\). The projected cross-covariance is \(0.399^2/3.9998+0.001^2/0.0002\approx0.04480224\). It matches the raw covariance \(0.04480224\), so the residual covariance is approximately zero. Each projected variance is approximately \(0.04480224\), leaving residual variance approximately \(0.95519776\).
For the second structure, \(s_Y=0.398\) and \(d_Y=0.002\). The projected cross-covariance is \(0.399(0.398)/3.9998+(0.001)(0.002)/0.0002\approx0.04970249\). The residual covariance is therefore approximately \(0.04480224-0.04970249=-0.00490025\). The projected variance for \(Y\) is \(0.398^2/3.9998+0.002^2/0.0002\approx0.05960298\), so its residual variance is approximately \(0.94039702\); the residual variance for \(X\) remains approximately \(0.95519776\). The second partial correlation is thus approximately \(-0.00490025/\sqrt{(0.95519776)(0.94039702)}\approx-0.00517\), rather than approximately zero.
Both covariance structures are valid: in each, the residual variances are positive and the squared residual covariance is smaller than their product. The example does not show a huge absolute correlation; it shows that a small change can change the inferred residual association relative to a result near zero.
Worked Example: Rounding Can Hide a Sign Change
Take controls with correlation \(0.999\), so \(A=\begin{pmatrix}1&0.999\\0.999&1\end{pmatrix}\). Set \(\operatorname{Cov}(X,Z)=(0.201,0.199)\), so its sum and difference coordinates are \(0.400\) and \(0.002\). Suppose the displayed, three-decimal values for \(\operatorname{Cov}(Y,Z)\) are \((0.200,0.200)\). Two possible unrounded pairs that both display this way are \((0.2004,0.1996)\) and \((0.1996,0.2004)\); their difference coordinates are \(0.0008\) and \(-0.0008\), while both have sum \(0.400\).
The sum-direction contribution to the projected cross-covariance in either case is \(0.400(0.400)/3.998\approx0.04002001\). The difference-direction contribution is respectively \((0.002)(0.0008)/0.002=0.0008\) and \((0.002)(-0.0008)/0.002=-0.0008\). Thus the full projections are approximately \(0.04082001\) and \(0.03922001\).
Choose the same raw covariance \(\operatorname{Cov}(X,Y)=0.04002001\) in both cases. The residual covariance is then approximately \(-0.0008\) in the first case and \(+0.0008\) in the second. The residual variances are positive: for \(X\), the projection variance is \(0.400^2/3.998+0.002^2/0.002\approx0.04202001\), leaving \(0.95797999\); for \(Y\), it is \(0.400^2/3.998+0.0008^2/0.002\approx0.04034001\), leaving \(0.95965999\). Therefore the partial correlations are approximately \(-0.0008/\sqrt{(0.95797999)(0.95965999)}\approx-0.00083\) and \(+0.00083\). Identical rounded control covariances are compatible with opposite signs here. The rounded entries alone do not determine which partial correlation applies.
Worked Example: Reading the Condition Number
For two standardized controls with correlation \(r=0.9999\), the eigenvalues of their covariance matrix are \(1+r=1.9999\) and \(1-r=0.0001\). Its condition number is therefore \(1.9999/0.0001=19{,}999\), and its inverse has an eigenvalue \(1/0.0001=10{,}000\) in the difference direction.
For instance, the contrast vector \(v=(0.001,-0.001)\) has squared length \(0.000002\). Its quadratic form with the inverse is \(v^\mathsf{T}A^{-1}v=0.000002/0.0001=0.02\). The same vector's ordinary squared length is only \(0.000002\); inversion has magnified its contribution by a factor of \(10{,}000\). This calculation identifies the vulnerable direction. It does not by itself give a partial correlation: the cross-covariances with \(X\) and \(Y\), as well as their residual variances, are also required.
Diagnosis, Interpretation, and Pitfalls
A practical diagnosis should examine the control covariance block and the partial correlation together. For a covariance matrix, the eigenvalue ratio depends on measurement units; standardizing controls to form their correlation matrix can make near-redundancy easier to interpret, but it does not remove it. Report the controls used and, where relevant, the conditioning or near-dependence that affects interpretation. Recomputing the analysis with reasonable data precision or small perturbations can reveal whether the partial correlation is robust.
Near-singularity is distinct from exact singularity. If one control is an exact linear combination of others, the control block has no inverse, and the inverse-based formula is undefined. If it is merely near-singular, the formula exists, but small perturbations may change its value. A generalized inverse or a regularized inverse may be used under a stated method, but it changes the projection procedure; it should not be presented as though it were the same unmodified partial correlation.
- Mistake: “The controls are highly correlated, so the partial correlation must be wrong.” High correlation among controls is a warning about sensitivity, not proof that the calculation is invalid. Check the covariance block and assess how the result responds to plausible perturbations.
- Mistake: “The inverse exists, so the result is stable.” Invertibility only means the calculation is defined. A very small eigenvalue can make the inverse highly sensitive even when it exists.
- Mistake: “Rounding differences are cosmetic.” In a near-singular block, rounding can alter small contrasts between control covariances. Those contrasts may be strongly weighted by the inverse, as the sign-change example demonstrates.
- Mistake: “Standardizing fixes collinearity.” Standardization changes units, not linear redundancy. It can aid diagnosis but cannot make a near-singular correlation block well-conditioned.
Check Your Understanding
Use the definitions and calculations above to answer these questions.
- Why does a small eigenvalue of the control covariance matrix make its inverse sensitive to perturbations?
- In the two-control examples, which linear combination of controls corresponds to the small eigenvalue when their correlation is close to one?
- Why can identical rounded control covariance entries be compatible with partial correlations of opposite signs?
- What is the difference between a near-singular control block and an exactly singular one for the inverse-based definition?
- Would standardizing the controls eliminate exact linear dependence? Explain.