The Rayleigh Quotient Characterization of the First Component

Why the First Component Is a Variance Maximizer

A direction in a multivariate data space produces a single numerical score for each observation. Principal component analysis begins by asking which unit direction makes that score vary as much as possible. The unit-length requirement matters: without it, multiplying any direction by a large constant would increase the score variance without changing the direction itself.

The answer is encoded by the Rayleigh quotient of the covariance matrix. Its maximization gives a precise variational characterization of the first principal component: the greatest projected variance equals the largest covariance eigenvalue, and the maximizing directions are exactly the unit vectors in the corresponding eigenspace.

Projected Variance and the Rayleigh Quotient

Definition: Let \(X\) be a \(p\)-dimensional random vector with finite second moments, mean vector \(\mu=E[X]\), and covariance matrix \(\Sigma=\operatorname{Cov}(X)\). For a direction \(a\in\mathbb{R}^p\), the centered linear score is \(a^T(X-\mu)\). Its variance is \(a^T\Sigma a\). For every nonzero \(a\), the Rayleigh quotient of \(\Sigma\) at \(a\) is \[ R_\Sigma(a)=\frac{a^T\Sigma a}{a^Ta}. \] For a unit direction, \(a^Ta=1\), so the Rayleigh quotient is exactly the projected variance.

The covariance matrix \(\Sigma\) is real, symmetric, and positive semidefinite. Symmetry gives an orthonormal eigenbasis, and positive semidefiniteness ensures that every eigenvalue is nonnegative. Write its eigenvalues in descending order, including multiplicities, as \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\geq 0\). The goal here is to characterize the largest projected variance over all unit vectors \(a\).

Worked Example: A Diagonal Covariance Matrix

Let \(X_1,X_2,X_3\) be independent random variables with \(X_1\sim N(0,9)\), \(X_2\sim N(0,4)\), and \(X_3\sim N(0,1)\). For \(X=(X_1,X_2,X_3)^T\), independence gives \[ \Sigma=\begin{pmatrix}9&0&0\\0&4&0\\0&0&1\end{pmatrix}. \]

For any unit direction \(a=(a_1,a_2,a_3)^T\), its projected variance is \[ a^T\Sigma a=9a_1^2+4a_2^2+a_3^2, \qquad a_1^2+a_2^2+a_3^2=1. \] The coefficients are at most 9, so \[ 9a_1^2+4a_2^2+a_3^2 \leq 9(a_1^2+a_2^2+a_3^2)=9. \] Equality requires \(a_2=a_3=0\); the unit directions attaining it are \(a=(1,0,0)^T\) and \(a=(-1,0,0)^T\). For a check away from the maximum, \(a=(1,2,2)^T/3\) is a unit vector and its projected variance is \[ \frac{9(1)^2+4(2)^2+1(2)^2}{9}=\frac{29}{9}<9. \]

The Rayleigh Quotient Characterization

Theorem: Let \(\Sigma\) be a real symmetric \(p\times p\) matrix with eigenvalues \(\lambda_1\geq\cdots\geq\lambda_p\). Then \[ \max_{a\neq 0} R_\Sigma(a) = \max_{a^Ta=1} a^T\Sigma a = \lambda_1. \] A unit vector \(a\) attains this maximum if and only if it belongs to the eigenspace associated with \(\lambda_1\). In particular, every maximizing direction is an eigenvector for the largest eigenvalue, and every unit eigenvector for that eigenvalue is a maximizing direction.

Proof. By the spectral theorem, there is an orthogonal matrix \(V\) whose columns are orthonormal eigenvectors of \(\Sigma\), and \[ \Sigma=V\operatorname{diag}(\lambda_1,\ldots,\lambda_p)V^T. \] For any nonzero \(a\), set \(z=V^Ta\). Orthogonality gives \(a^Ta=z^Tz=\sum_{i=1}^p z_i^2\), and the diagonal representation gives \(a^T\Sigma a=\sum_{i=1}^p\lambda_i z_i^2\). Thus \[ R_\Sigma(a)=\frac{\sum_{i=1}^p\lambda_i z_i^2}{\sum_{i=1}^p z_i^2} \leq \frac{\lambda_1\sum_{i=1}^p z_i^2}{\sum_{i=1}^p z_i^2} =\lambda_1. \] This upper bound is attained by taking \(a\) to be any unit eigenvector associated with \(\lambda_1\), so the maximum equals \(\lambda_1\).

It remains to identify every case of equality. Subtracting the numerator from the upper-bound numerator shows that equality holds precisely when \[ \sum_{i=1}^p(\lambda_1-\lambda_i)z_i^2=0. \] Every term is nonnegative. Therefore the sum is zero exactly when \(z_i=0\) for every index with \(\lambda_i<\lambda_1\). The remaining coordinates correspond to eigenvectors with eigenvalue \(\lambda_1\), so \(a\) lies in that eigenspace. Conversely, a vector in that eigenspace has Rayleigh quotient \(\lambda_1\). This proves both the maximum and its full equality condition. \(\square\)

The quotient is unchanged by rescaling: for any nonzero scalar \(c\), \(R_\Sigma(ca)=R_\Sigma(a)\). This explains why maximizing the quotient over nonzero vectors is equivalent to maximizing projected variance over unit vectors. The unit-vector version fixes the scale of the score; the quotient version makes the scale invariance explicit.

Worked Example: A Non-Diagonal Covariance Matrix

Let \(U,V,W\) be independent \(N(0,1)\) random variables, and define \[ X_1=\sqrt{\frac{7}{2}}\,U+\sqrt{\frac{3}{2}}\,V,\qquad X_2=\sqrt{\frac{7}{2}}\,U-\sqrt{\frac{3}{2}}\,V,\qquad X_3=W. \] The means are zero. Independence and unit variances give \(\operatorname{Var}(X_1)=\operatorname{Var}(X_2)=7/2+3/2=5\), \(\operatorname{Cov}(X_1,X_2)=7/2-3/2=2\), and zero covariances with \(X_3\). Hence \[ \Sigma=\begin{pmatrix}5&2&0\\2&5&0\\0&0&1\end{pmatrix}. \]

The unit vectors \(v_+=(1,1,0)^T/\sqrt{2}\), \(v_-=(1,-1,0)^T/\sqrt{2}\), and \(e_3=(0,0,1)^T\) are orthonormal. Direct multiplication gives \[ \Sigma v_+=7v_+,\qquad \Sigma v_-=3v_-,\qquad \Sigma e_3=e_3. \] Thus the eigenvalues are \(7,3,1\), and the largest projected variance is 7, attained along \(v_+\) and its negative. For instance, the variance along \(v_-\) is \(v_-^T\Sigma v_-=3\), not 7. This illustrates why being an eigenvector alone is not enough: the eigenvalue determines the projected variance, and only an eigenvector for the largest eigenvalue maximizes it.

Stationary Directions and the Maximum

The variational result can also be seen through constrained optimization. This gives a useful distinction between being a stationary direction and being a maximizing direction.

Result: If \(\Sigma\) is real and symmetric, every stationary point of \(a^T\Sigma a\) subject to \(a^Ta=1\) is an eigenvector of \(\Sigma\). At such a point, the objective value is the corresponding eigenvalue. This condition is necessary for a maximum, but by itself does not say that the eigenvalue is the largest.

Derivation. The constraint set is the unit sphere, where the gradient of \(a^Ta-1\) is \(2a\), which is nonzero. The Lagrange multiplier condition therefore applies at every constrained stationary point. The gradient of \(a^T\Sigma a\) is \(2\Sigma a\) because \(\Sigma\) is symmetric. For some scalar \(\gamma\), the condition is \[ 2\Sigma a=2\gamma a,\qquad a^Ta=1. \] Consequently, \(\Sigma a=\gamma a\), so \(a\) is an eigenvector. Multiplying this equation on the left by \(a^T\) gives \(a^T\Sigma a=\gamma a^Ta=\gamma\), so the objective value is its eigenvalue. Conversely, a unit eigenvector satisfies this Lagrange multiplier condition. The unit sphere is compact and the objective is continuous, so a maximum exists; the Rayleigh quotient theorem identifies it as the eigenvalue \(\lambda_1\). Other eigenvectors are stationary too, but their objective values can be smaller. \(\square\)

Worked Example: A Repeated Largest Eigenvalue

Let \(X_1,X_2,X_3\) be independent, with \(X_1,X_2\sim N(0,4)\) and \(X_3\sim N(0,1)\). Then \(\Sigma=\operatorname{diag}(4,4,1)\). For a unit vector \(a=(a_1,a_2,a_3)^T\), \[ a^T\Sigma a=4a_1^2+4a_2^2+a_3^2 =4-3a_3^2, \] where the final equality uses \(a_1^2+a_2^2+a_3^2=1\).

The variance is at most 4, with equality exactly when \(a_3=0\). Thus every unit vector in the first two coordinate directions is a maximizing direction; for example, both \((1,0,0)^T\) and \((1,1,0)^T/\sqrt{2}\) attain variance 4. The largest eigenvalue is repeated, so there is not just one maximizing axis. A direction with a nonzero third coordinate has strictly smaller projected variance; in particular, \(e_3\) has variance 1.

Interpretation and a Common Pitfall

The characterization says exactly what “first” means when selecting a principal direction: among all unit linear scores, its variance is greatest. If the largest eigenvalue is simple, the only unit maximizers are the associated eigenvector and its negative. If it is repeated, every unit vector in the associated eigenspace maximizes variance. In either case, the maximum value is \(\lambda_1\).

Pitfall: maximizing without fixing scale. If one tries to maximize \(a^T\Sigma a\) over all \(a\) with no constraint, replacing \(a\) by \(ca\) multiplies the value by \(c^2\). For a positive maximum, the objective can grow without bound. The correction is to impose \(a^Ta=1\), or equivalently maximize the scale-invariant Rayleigh quotient.
Pitfall: treating any eigenvector as optimal. An eigenvector’s projected variance equals its own eigenvalue, which need not be the largest one. The correction is to compare eigenvalues and select the eigenspace for \(\lambda_1\).
Pitfall: assuming the maximizer is always unique. A repeated largest eigenvalue gives multiple maximizing directions. The equality condition in the theorem, rather than an assumption of uniqueness, determines the full set.

The Rayleigh quotient is useful because it turns an optimization over directions into a statement about the spectrum of the covariance matrix. It also supplies a check on calculations: every unit direction must have projected variance no greater than \(\lambda_1\), and a direction in the top eigenspace must attain that bound.

Check Your Understanding

Use the definitions and results above to answer the following questions.

For a nonzero vector \(a\), why is \(R_\Sigma(a)\) unchanged when \(a\) is multiplied by a nonzero scalar?
What is the maximum projected variance for a covariance matrix with eigenvalues \(6,2,0\), and which unit directions attain it?
If a unit eigenvector has eigenvalue 2 while the largest eigenvalue is 6, what is its projected variance, and is it a maximizing direction?
For \(\Sigma=\operatorname{diag}(5,5,1)\), describe all unit directions that maximize projected variance.
Why does an unconstrained maximization of \(a^T\Sigma a\) fail to define a direction when \(\Sigma\) has a positive eigenvalue?