First-Order Perturbation of Eigenvalues

Why Eigenvalues Have a First-Order Perturbation Formula

A sample covariance matrix is a noisy version of a population covariance matrix. Its eigenvalues therefore fluctuate, but the entries of the matrix do not affect each eigenvalue equally. To first order, a simple eigenvalue responds to the perturbation through one particular scalar: the perturbation’s quadratic form in the corresponding population eigenvector. This converts a matrix fluctuation into a scalar one and is the basic linearization used in PCA inference.

The formula applies when the eigenvalue is simple, meaning that it has multiplicity one. The size of its separation from the other eigenvalues determines how accurately the first-order approximation describes a finite perturbation. Repeated eigenvalues require a different treatment and are not covered by this formula.

First-Order Change Along a Matrix Path

Write the eigenvalues of \(A\) in decreasing order, \(\lambda_1\geq\cdots\geq\lambda_p\), and suppose \(\lambda_i\) is simple. For sufficiently small \(t\), the eigenvalue near \(\lambda_i\) remains a simple eigenvalue of \(A(t)\); call it \(\lambda_i(t)\). The central formula is:

The quantity \(u_i^\top H u_i\) is unchanged if the sign of \(u_i\) is reversed. It is a directional measurement of \(H\): only the component of the perturbation along the eigenvector enters at first order.

Proof by Differentiating the Characteristic Equation

The characteristic polynomial provides a concise proof that does not require choosing a derivative for the eigenvector. Define \[ q(t,z)=\det(A+tH-zI). \] At \((0,\lambda_i)\), \(q(0,\lambda_i)=0\). Because \(\lambda_i\) is simple, it is a simple root of \(q(0,z)\), so \(\partial q(0,\lambda_i)/\partial z\neq 0\). The implicit function theorem therefore gives a differentiable root \(z=\lambda_i(t)\) near \(t=0\), with \(q(t,\lambda_i(t))=0\).

For a differentiable matrix \(M(s)\), the derivative of its determinant is \[ \frac{d}{ds}\det M(s)=\operatorname{tr}\!\left(\operatorname{adj}(M(s))M'(s)\right), \] where \(\operatorname{adj}\) denotes the adjugate. At \(M=A-\lambda_i I\), symmetry and simplicity imply \[ \operatorname{adj}(A-\lambda_i I)=c\,u_i u_i^\top \] for a nonzero scalar \(c\). Indeed, the adjugate’s columns lie in the one-dimensional null space spanned by \(u_i\); symmetry gives the corresponding rank-one form, and the simple root ensures it is nonzero. Differentiate \(q(t,\lambda_i(t))=0\) at zero. The determinant derivative gives \[ 0=\operatorname{tr}\!\left(c\,u_i u_i^\top(H-\lambda_i'(0)I)\right) =c\left(u_i^\top H u_i-\lambda_i'(0)\right). \] Since \(c\neq0\), the claimed derivative follows. Differentiability then gives the stated first-order expansion. \(\square\)

A Remainder Bound and the Role of the Gap

The derivative formula describes a perturbation that tends to zero. A finite perturbation also admits a useful error bound when the eigenvalue is isolated. Define its spectral gap by \[ \delta_i=\min_{j\ne i}|\lambda_i-\lambda_j|>0. \] For a symmetric matrix \(E\), let \(\varepsilon=\|E\|_2\), the operator norm. The eigenvalue perturbation inequality \(|\tilde\lambda_i-\lambda_i|\leq\varepsilon\) follows from the min–max characterization: every unit-vector Rayleigh quotient changes by at most \(\varepsilon\), since \(|x^\top E x|\leq\varepsilon\), so the corresponding min–max values change by at most that amount. In particular, if \(\varepsilon<\delta_i/4\), the eigenvalue \(\tilde\lambda_i\) of \(A+E\) remains separated from the other ordered eigenvalues.

To prove the bound, put \(d=\tilde\lambda_i-\lambda_i\), and choose a unit eigenvector \(v\) of \(A+E\) for \(\tilde\lambda_i\). Decompose \(v=au_i+w\), where \(w\perp u_i\); choose the sign of \(v\) so \(a\geq0\). Then \(a^2+\|w\|^2=1\). Let \(P\) be the orthogonal projection onto the complement of \(u_i\). Projecting \((A+E)v=(\lambda_i+d)v\) onto that complement yields \[ (A-\lambda_i I)w+PEv=dw. \] On the complement, every eigenvalue of \(A-\lambda_i I\) has absolute value at least \(\delta_i\). Since \(|d|\leq\varepsilon\), the inverse of \(A-\lambda_i I-dI\) there has norm at most \(1/(\delta_i-\varepsilon)\). Hence \[ \|w\|\leq\frac{\varepsilon}{\delta_i-\varepsilon}<\frac13, \qquad a=\sqrt{1-\|w\|^2}\geq\frac{\sqrt{8}}{3}. \]

Taking the inner product of the eigenvalue equation with \(v\), and using \(v=au_i+w\), gives \[ d=w^\top(A-\lambda_i I)w+v^\top Ev. \] The projected equation also gives \(w^\top(A-\lambda_i I)w=d\|w\|^2-w^\top Ev\). Substitution and cancellation of \(d\|w\|^2\) imply \(da^2=a\,u_i^\top Ev\). Since \(a>0\), this is \(d=u_i^\top Ev/a\). Therefore \[ |d-u_i^\top Eu_i| =\frac{|u_i^\top Ew|}{a} \leq\frac{\varepsilon\|w\|}{a} \leq\frac{\varepsilon^2}{a(\delta_i-\varepsilon)} \leq\frac{2\varepsilon^2}{\delta_i}. \] The last inequality follows from \(a\geq\sqrt8/3\) and \(\delta_i-\varepsilon>3\delta_i/4\). This proves the bound. \(\square\)

The bound makes the gap’s role explicit: a larger gap limits the remainder, while a small gap makes a first-order approximation less reliable. The formula does not assert that eigenvectors are stable in every setting; it quantifies only the eigenvalue’s first-order approximation.

Application to Sample Covariance Fluctuations

Let \(X_1,\ldots,X_n\) be independent and identically distributed \(p\)-dimensional random vectors with mean \(\mu\), covariance matrix \(\Sigma\), and finite fourth moments. Keep \(p\) fixed. Use the divisor-\(n\) centered sample covariance \[ S_n=\frac1n\sum_{r=1}^n(X_r-\bar X)(X_r-\bar X)^\top, \qquad \bar X=\frac1n\sum_{r=1}^nX_r. \] Suppose \(\Sigma\) has a simple eigenvalue \(\lambda_i\), with unit eigenvector \(u_i\) and gap \(\delta_i>0\). Finite fourth moments ensure entrywise sample second moments fluctuate at order \(n^{-1/2}\); with fixed \(p\), this gives \(\|S_n-\Sigma\|_2=O_p(n^{-1/2})\). The remainder bound therefore yields \[ \hat\lambda_i-\lambda_i =u_i^\top(S_n-\Sigma)u_i+O_p(n^{-1}), \] where \(\hat\lambda_i\) is the corresponding ordered sample eigenvalue. In particular, the first-order fluctuation is the single scalar \(u_i^\top(S_n-\Sigma)u_i\), not the full matrix fluctuation considered entry by entry.

To see its sampling interpretation, put \(Y_r=u_i^\top(X_r-\mu)\). Since \(u_i^\top\Sigma u_i=\lambda_i\), direct substitution into the centered sample covariance gives \[ u_i^\top(S_n-\Sigma)u_i =\frac1n\sum_{r=1}^n(Y_r^2-\lambda_i)-\bar Y^2, \qquad \bar Y=\frac1n\sum_{r=1}^nY_r. \] The finite fourth-moment condition gives \(\bar Y=O_p(n^{-1/2})\), so \(\bar Y^2=O_p(n^{-1})\). Thus the leading sample eigenvalue fluctuation is governed by the centered average of the squared projected observations: \[ \hat\lambda_i-\lambda_i =\frac1n\sum_{r=1}^n(Y_r^2-\lambda_i)+O_p(n^{-1}). \] This identifies the scalar quantity to which a suitable scalar limit theorem can be applied; it does not require deriving a joint limit for multiple sample eigenvalues.

Common Pitfall and Practical Interpretation

A common mistake is to say that a zero value of \(u_i^\top E u_i\) means the eigenvalue does not change. The second worked example disproves this: the off-diagonal perturbation has zero first-order effect but changes the leading eigenvalue at second order. The correct conclusion is that the linear term vanishes; the remainder may still be nonzero.

A second caution is that the gap condition matters. The expansion is most informative when the target eigenvalue is well separated and the perturbation is small relative to that separation. When the gap is tiny, the quadratic remainder bound can be large even if the matrix perturbation itself appears modest. In PCA, the first-order formula is useful because it isolates the sample fluctuation relevant to a simple population eigenvalue, while the gap indicates when that linearization is trustworthy.