Why Diagnose Covariance Sensitivity?
PCA directions are calculated from a covariance matrix, but that matrix may depend on rounding, measurement precision, or small changes to estimated covariances. The important question is not simply whether an entry changed. It is whether the change is large compared with the eigenvalue gaps that separate the directions of interest. A small perturbation can noticeably rotate an axis when its eigenvalue is close to another one; the same perturbation may leave a well-separated direction nearly unchanged.
This tutorial treats the covariance change as deterministic. It provides tools for tracking eigenvalue changes and for judging whether sensitivity belongs to an individual principal axis or to a whole leading subspace. It does not assume a sampling model for the perturbation.
Representing a Covariance Perturbation
Symmetry matters: changing a covariance between variables \(i\) and \(j\) changes both positions \((i,j)\) and \((j,i)\). A change \(\eta\) in that covariance contributes \(\eta\) to each of those entries. Its perturbation matrix has operator norm \(|\eta|\). By contrast, changing a single diagonal variance by \(\eta\) also has operator norm \(|\eta|\).
When the pattern is more complicated, useful bounds are \(\|E\|_2\leq\|E\|_F\) and \(\|E\|_2\leq\sqrt{\|E\|_1\|E\|_\infty}\), where the Frobenius norm is the square root of the sum of squared entries, \(\|E\|_1\) is the largest absolute column sum, and \(\|E\|_\infty\) is the largest absolute row sum. A bound is enough for a conservative diagnosis; it need not equal the exact operator norm.
Worked Example: Translating Entry Changes Into a Norm
Suppose a covariance matrix for three variables changes by \(0.2\) in its first–second covariance and by \(0.1\) in its first–third covariance, with all variances and other covariances unchanged. The perturbation is
The Frobenius norm is \(\|E\|_F=\sqrt{2(0.2^2)+2(0.1^2)}=\sqrt{0.1}\approx0.3162\), because each changed covariance appears twice. To obtain the operator norm, the nonzero eigenvalues of \(E\) are \(\pm\sqrt{0.2^2+0.1^2}=\pm\sqrt{0.05}\); the third eigenvalue is zero. Since \(E\) is symmetric, its operator norm is its largest absolute eigenvalue, so \(\|E\|_2=\sqrt{0.05}\approx0.2236\). The Frobenius norm gives a valid but looser upper bound.
Eigenvalues Cannot Move More Than the Perturbation Norm
This result gives a direct scale for eigenvalue sensitivity. It also helps check whether the perturbed matrix can still be a covariance matrix: if \(\Sigma\) is positive semidefinite and \(\lambda_p>\|E\|_2\), then \(\widetilde{\lambda}_p\geq\lambda_p-\|E\|_2>0\), so the perturbed matrix remains positive definite.
Proof. For every unit vector \(x\), the Cauchy–Schwarz inequality and the definition of the operator norm give \(|x^\mathsf{T}Ex|\leq\|x\|_2\|Ex\|_2\leq\|E\|_2\). Thus each unit-vector quadratic form changes by at most \(\|E\|_2\). By the Courant–Fischer characterization, \(\lambda_j(\Sigma)=\max_{\dim L=j}\min_{x\in L,\,\|x\|_2=1}x^\mathsf{T}\Sigma x\). Take a \(j\)-dimensional subspace \(L\) attaining this maximum for \(\Sigma\). For every unit \(x\in L\), \(x^\mathsf{T}\widetilde{\Sigma}x\geq x^\mathsf{T}\Sigma x-\|E\|_2\). Taking the minimum over \(x\in L\), and then using the maximum characterization for \(\widetilde{\Sigma}\), gives \(\widetilde{\lambda}_j\geq\lambda_j-\|E\|_2\). Interchanging \(\Sigma\) and \(\widetilde{\Sigma}\) applies the same argument to \(-E\) and gives \(\lambda_j\geq\widetilde{\lambda}_j-\|E\|_2\). Combining the inequalities proves the claim. \(\square\)
Eigenvalue Gaps Diagnose Direction Stability
An eigenvalue bound alone does not determine how much an eigenvector rotates. The key scale for directions is the eigengap. For the leading \(k\)-dimensional principal subspace, define \(\delta_k=\lambda_k-\lambda_{k+1}\). A positive gap separates the selected eigenvalues from the rest. When the gap is small, the boundary between the leading subspace and its complement is weakly defined.
The Davis–Kahan sin-theta theorem makes this comparison quantitative. In the form useful here, let \(\epsilon=\|E\|_2\), suppose \(\delta_k>0\), and assume \(\epsilon<\delta_k/2\). Let \(U\) and \(\widetilde U\) contain orthonormal bases for the leading \(k\)-dimensional subspaces of \(\Sigma\) and \(\widetilde{\Sigma}\). The largest sine of their principal angles, denoted \(\|\sin\Theta(\widetilde U,U)\|_2\), satisfies \[ \|\sin\Theta(\widetilde U,U)\|_2\leq\frac{\epsilon}{\delta_k-\epsilon}\leq\frac{2\epsilon}{\delta_k}. \] The eigenvalue bound implies \(\widetilde{\lambda}_{k+1}\leq\lambda_{k+1}+\epsilon\), so the selected eigenvalues of \(\Sigma\) remain separated from the unselected eigenvalues of \(\widetilde{\Sigma}\) by at least \(\delta_k-\epsilon>0\). Applying the Davis–Kahan bound with this separation gives the first inequality; \(\epsilon<\delta_k/2\) gives the second.
This is a subspace guarantee, not automatically a guarantee for every named axis inside that subspace. In particular, a gap between \(\lambda_1\) and \(\lambda_2\) governs the leading individual direction. A gap between \(\lambda_k\) and \(\lambda_{k+1}\) governs the leading \(k\)-dimensional subspace. Gaps among \(\lambda_1,\ldots,\lambda_k\) do not enter that subspace’s boundary separation.
Worked Example: A Large Rotation Inside a Stable Subspace
Consider \(\Sigma=\operatorname{diag}(5.1,5.0,1)\), and perturb only the covariance between variables 1 and 2 by \(0.08\): \(\widetilde{\Sigma}=\begin{pmatrix}5.1&0.08&0\\0.08&5.0&0\\0&0&1\end{pmatrix}\). The leading two-dimensional subspace is exactly \(\operatorname{span}(e_1,e_2)\) both before and after perturbation, because the perturbation acts only within that coordinate plane and does not connect it to \(e_3\). Thus the leading two-dimensional subspace has not changed.
The leading axis within that plane does change substantially. The original leading eigengap is \(5.1-5.0=0.1\), while the two eigenvalues of the perturbed \(2\times2\) block are \(5.05\pm\sqrt{0.05^2+0.08^2}=5.05\pm0.09434\). The rotation angle \(\theta\) of the leading axis from \(e_1\) satisfies \(\tan(2\theta)=2(0.08)/(5.1-5.0)=1.6\), so \(\theta=\tfrac12\arctan(1.6)\approx0.506\) radians, or about \(29.0^\circ\). The full matrix remains positive definite: its smallest eigenvalue is \(1\), and the perturbed block’s smaller eigenvalue is about \(4.956\). This example shows why a sensitive individual axis does not imply that the larger subspace containing it is unstable.
Exact Rotation in Two Dimensions
For a two-variable covariance matrix, the axis change can be calculated exactly. Let the original matrix be \(\operatorname{diag}(a,b)\), where \(a>b\), and let the perturbation be \(\begin{pmatrix}u&v\\v&w\end{pmatrix}\). Put \(d=(a-b+u-w)/2\). If \(d>0\), the leading eigenvector of the perturbed matrix has angle \(\theta\) from the first coordinate axis satisfying \[ \tan(2\theta)=\frac{v}{d}=\frac{2v}{a-b+u-w}. \] To verify this, subtract the scalar midpoint of the perturbed diagonal entries. The matrix becomes \(\begin{pmatrix}d&v\\v&-d\end{pmatrix}\). Rotating coordinates by \(\theta\) makes the off-diagonal entry \(v\cos(2\theta)-d\sin(2\theta)\). Setting it to zero gives the formula. The eigenvalues of the original perturbed matrix are the diagonal midpoint plus and minus \(\sqrt{d^2+v^2}\).
Worked Example: A Moderate Covariance Change and Its Axis Rotation
Take \(\Sigma=\operatorname{diag}(4,1)\) and increase the covariance between the variables by \(0.6\), so that \(\widetilde{\Sigma}=\begin{pmatrix}4&0.6\\0.6&1\end{pmatrix}\). Here \(u=w=0\), \(v=0.6\), and \(d=(4-1)/2=1.5\). Therefore \(\tan(2\theta)=0.6/1.5=0.4\), giving \(\theta=\tfrac12\arctan(0.4)\approx0.1903\) radians, or about \(10.9^\circ\).
The eigenvalue gap after perturbation is \(2\sqrt{1.5^2+0.6^2}=\sqrt{10.44}\approx3.231\). The two eigenvalues are \((5+3.231)/2\approx4.116\) and \((5-3.231)/2\approx0.884\), both positive, so the perturbed matrix is still a valid covariance matrix. The original gap was \(3\), much larger than the size \(0.6\) of the off-diagonal perturbation; the resulting axis rotation is noticeable but not extreme.
Putting the Diagnostics Together
Worked Example: Bounding the Change in a Leading Direction
Let \(\Sigma=\operatorname{diag}(9,4,1)\), and use the perturbation from the earlier norm calculation. The original leading eigengap is \(\delta_1=9-4=5\), and the exact perturbation norm is \(\epsilon=\sqrt{0.05}\approx0.2236\). Since \(0.2236<5/2\), the Davis–Kahan bound applies: \(\sin\theta\leq0.2236/(5-0.2236)\approx0.0468\). Consequently \(\theta\leq\arcsin(0.0468)\approx2.68^\circ\).
The eigenvalue bound also gives \(\widetilde{\lambda}_1\in[9-0.2236,9+0.2236]\) and \(\widetilde{\lambda}_3\in[1-0.2236,1+0.2236]\). In particular, \(\widetilde{\lambda}_3\geq0.7764>0\), so the perturbed matrix remains positive definite. The direction bound is conservative: it certifies a small maximum rotation without requiring an exact eigendecomposition of the perturbed matrix.
Common Pitfall and Practical Interpretation
Wrong reasoning: “Every covariance entry changed by at most \(0.1\), so every principal direction can rotate by at most \(0.1\) radians.” Entrywise changes do not directly bound angles. Their combined effect determines \(\|E\|_2\), and direction sensitivity depends on that norm relative to the relevant eigengap. A perturbation small in absolute terms can rotate an axis considerably when its leading eigengap is even smaller.
Correction: Form the symmetric perturbation matrix, bound or calculate its operator norm, and compare that scale with the eigengap for the object of interest. For an individual leading axis, inspect \(\lambda_1-\lambda_2\). For a leading \(k\)-dimensional subspace, inspect \(\lambda_k-\lambda_{k+1}\). If the eigenvalues inside a leading cluster are close, avoid interpreting its individual axes as uniquely stable; describe the subspace instead.
The same checks should be made after changing measurement units or preprocessing, because those choices alter the covariance matrix and can alter both perturbation sizes and gaps. A sensitivity diagnosis is therefore tied to the covariance representation being analyzed. It is most useful when the perturbation has a clear interpretation, such as rounding or a specified covariance change, and when the eigengap is reported alongside the direction or subspace result.
Check Your Understanding
Use the definitions and bounds above to answer these questions.
- A symmetric perturbation changes one off-diagonal covariance by \(\eta\), with its matching symmetric entry changed by the same amount. What is the operator norm of that perturbation?
- If \(\|E\|_2=0.3\), what can the Weyl eigenvalue bound say about the change in each ordered eigenvalue?
- Why can a leading axis rotate substantially while the leading two-dimensional subspace remains unchanged?
- For a leading \(k\)-dimensional subspace, which eigengap is relevant to its stability, and why?
- A covariance matrix has smallest eigenvalue \(0.2\), while a perturbation has operator norm \(0.25\). Does the Weyl bound certify that the perturbed matrix is positive semidefinite? Explain.