Why Successive Principal Directions Are Orthogonal
Principal components are chosen one at a time: the first direction captures the greatest possible variation, and each later direction is required to be perpendicular to those already chosen. This requirement is not an arbitrary geometric convention. For a covariance matrix, it leads directly to the remaining eigenvectors and ensures that the resulting component scores have zero covariance with earlier scores.
The argument applies to a centered random vector with finite second moments, or to a centered data matrix summarized by its sample covariance matrix. Write the covariance matrix as \(\Sigma\) (or \(S\) for the sample covariance matrix). It is symmetric and positive semidefinite. For a unit direction \(w\), the variance along that direction is the quadratic form \(w^\mathsf{T}\Sigma w\). A direction must be normalized: without the condition \(w^\mathsf{T}w=1\), its length could be increased to make the quadratic form arbitrarily large.
The earlier material on the eigenvalue decomposition gives the key representation: a real symmetric covariance matrix has an orthonormal basis of eigenvectors. Order those eigenvectors \(v_1,\ldots,v_p\) so their eigenvalues satisfy \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\geq0\). The question here is why the successive constrained maximizations select this ordered eigenbasis.
The Constrained Maximization Argument
Suppose the first \(k-1\) directions have been chosen as \(v_1,\ldots,v_{k-1}\). Any unit direction perpendicular to all of them can be written using the remaining eigenvectors: \(w=\sum_{j=k}^{p}a_jv_j\), where \(\sum_{j=k}^{p}a_j^2=1\). Because the eigenvectors are orthonormal, the variance objective becomes a weighted average of the remaining eigenvalues. This both identifies the maximum and explains why the next eigenvector is a maximizing direction.
The bound is attained by setting \(w=v_k\), for which \(a_k=1\) and all other coefficients are zero. Thus \(v_k\) is a maximizing direction after the first \(k-1\) directions have been excluded. If \(\lambda_k\) is strictly larger than all the remaining eigenvalues, that maximizing direction is unique up to equivalent choices of orientation. If eigenvalues are tied, there can be several maximizing directions in the tied eigenspace; an orthonormal choice is still possible, but the individual directions need not be uniquely determined.
For the \(k\)th direction, require \(w^\mathsf{T}w=1\) and \(v_j^\mathsf{T}w=0\) for each previously chosen direction \(j<k\).
Orthogonality to \(v_1,\ldots,v_{k-1}\) leaves only coefficients on \(v_k,\ldots,v_p\). Unit length makes the squared coefficients sum to one.
The objective is a weighted average of \(\lambda_k,\ldots,\lambda_p\), so it cannot exceed \(\lambda_k\). The direction \(v_k\) attains that bound.
The same conclusion can be seen from the first-order conditions. For a unit vector constrained to be perpendicular to the earlier eigenvectors, introduce multipliers \(\alpha\) and \(\mu_1,\ldots,\mu_{k-1}\). The Lagrange conditions are \( \Sigma w-\alpha w-\sum_{j<k}\mu_jv_j=0\). Taking the inner product with any \(v_j\), \(j<k\), gives \(\mu_j=0\): since \(\Sigma v_j=\lambda_jv_j\) and \(v_j^\mathsf{T}w=0\), both \(v_j^\mathsf{T}\Sigma w\) and \(\alpha v_j^\mathsf{T}w\) are zero. Consequently \(\Sigma w=\alpha w\); a constrained stationary direction is an eigenvector. The eigenbasis comparison above then determines which remaining eigenvector attains the maximum. The first-order condition alone does not establish that a stationary eigenvector is the maximizing one.
Worked Examples
Worked Example: Selecting the Second Direction from a Diagonal Covariance Matrix
A hypothetical portfolio study uses three centered risk measurements with covariance matrix \(\Sigma=\operatorname{diag}(9,4,1)\). Find the first two principal directions by successive variance maximization, and check whether their scores are uncorrelated.
Method. For a unit direction \(w=(w_1,w_2,w_3)^\mathsf{T}\), the variance objective is \(w^\mathsf{T}\Sigma w=9w_1^2+4w_2^2+w_3^2\), with \(w_1^2+w_2^2+w_3^2=1\). The coefficients in the objective are ordered \(9>4>1\), so the maximum is attained by putting all the squared weight on the first coordinate. Thus the first direction is \(v_1=(1,0,0)^\mathsf{T}\), and the maximum objective value is \(9\).
For the second direction, impose \(v_1^\mathsf{T}w=0\). This gives \(w_1=0\), so \(w=(0,w_2,w_3)^\mathsf{T}\), with \(w_2^2+w_3^2=1\). Its objective is \(4w_2^2+w_3^2\). Since \(4>1\), this is at most \(4(w_2^2+w_3^2)=4\), attained at \(w_2=1,w_3=0\). Therefore \(v_2=(0,1,0)^\mathsf{T}\).
Score check. If \(Z_1=v_1^\mathsf{T}X\) and \(Z_2=v_2^\mathsf{T}X\), then \(\operatorname{Cov}(Z_1,Z_2)=v_1^\mathsf{T}\Sigma v_2=0\), because \(\Sigma v_2=(0,4,0)^\mathsf{T}\) and \(v_1^\mathsf{T}(0,4,0)^\mathsf{T}=0\). The answer is consistent: the selected directions are perpendicular, and their score covariance is zero.
Common mistake avoided: Maximizing over all directions a second time would simply select \(v_1\) again. The second maximization must include the constraint \(v_1^\mathsf{T}w=0\).
Worked Example: A Non-Diagonal Matrix and a Constrained Second Direction
A hypothetical insurer summarizes two centered exposure measures with \(\Sigma=\begin{pmatrix}5&2\\2&5\end{pmatrix}\). Find the first direction, then maximize variance subject to perpendicularity to it. Verify the score covariance.
First direction. The vectors \(v_1=(1,1)^\mathsf{T}/\sqrt{2}\) and \(v_2=(1,-1)^\mathsf{T}/\sqrt{2}\) are unit vectors and are perpendicular, since their inner product is \((1-1)/2=0\). Direct multiplication gives \(\Sigma v_1=(7,7)^\mathsf{T}/\sqrt{2}=7v_1\) and \(\Sigma v_2=(3,-3)^\mathsf{T}/\sqrt{2}=3v_2\). These form an orthonormal eigenbasis, with the larger eigenvalue \(7\) attached to \(v_1\). Hence \(v_1\) is the first maximizing direction.
Constrained second direction. Every unit vector in two dimensions perpendicular to \(v_1\) is either \(v_2\) or \(-v_2\). Both give the same objective: \(v_2^\mathsf{T}\Sigma v_2=v_2^\mathsf{T}(3v_2)=3\), because \(v_2^\mathsf{T}v_2=1\). Thus the second direction is \(v_2\), up to its orientation.
Score check. For centered \(X\), set \(Z_1=v_1^\mathsf{T}X\) and \(Z_2=v_2^\mathsf{T}X\). Then \(\operatorname{Cov}(Z_1,Z_2)=v_1^\mathsf{T}\Sigma v_2=v_1^\mathsf{T}(3v_2)=3v_1^\mathsf{T}v_2=0\). The calculation confirms that the constrained direction is an eigenvector perpendicular to the first, and the scores are uncorrelated.
Common mistake avoided: The matrix is not diagonal, so its coordinate axes are not the principal directions. Checking the eigenvectors, rather than assuming the original variables are uncorrelated, identifies the correct directions.
Worked Example: Successive Directions from a Centered Sample
A hypothetical pricing team records three centered observations on two risk indicators: \((2,0),(-1,1),(-1,-1)\). Using the usual sample covariance divisor \(n-1\), find the first two directions and check the covariance of the corresponding sample scores.
There are \(n=3\) observations, and their coordinate sums are \(2-1-1=0\) and \(0+1-1=0\), so the data are centered. The sums of squares and cross-products are \(\sum x_{i1}^2=4+1+1=6\), \(\sum x_{i2}^2=0+1+1=2\), and \(\sum x_{i1}x_{i2}=0-1+1=0\). Dividing each by \(n-1=2\) gives \(S=\begin{pmatrix}3&0\\0&1\end{pmatrix}\).
The first coordinate direction \(v_1=(1,0)^\mathsf{T}\) has objective \(3\); a unit vector \(w=(a,b)^\mathsf{T}\) has objective \(3a^2+b^2\leq3(a^2+b^2)=3\). Thus \(v_1\) maximizes the sample variance. The constraint \(v_1^\mathsf{T}w=0\) requires \(a=0\), and unit length then gives \(b=1\) or \(-1\). Choose \(v_2=(0,1)^\mathsf{T}\); its objective is \(1\).
The first-component scores for the three observations are \(2,-1,-1\), and the second-component scores are \(0,1,-1\). Their sample covariance is \(\{2(0)+(-1)(1)+(-1)(-1)\}/2=0\). Equivalently, \(v_1^\mathsf{T}Sv_2=(1,0)\begin{pmatrix}3&0\\0&1\end{pmatrix}(0,1)^\mathsf{T}=0\). Both checks agree.
Common mistake avoided: Dividing the sums by \(n=3\) instead of the specified sample-covariance divisor \(n-1=2\) would give the wrong covariance matrix values. Here the divisor is stated and applied consistently.
Why Orthogonal Directions Give Uncorrelated Scores
Let \(X\) be a centered random vector with covariance matrix \(\Sigma\), and let \(Z_i=v_i^\mathsf{T}X\) and \(Z_j=v_j^\mathsf{T}X\). By the definition of covariance and linearity, \(\operatorname{Cov}(Z_i,Z_j)=v_i^\mathsf{T}\Sigma v_j\). If \(v_j\) is an eigenvector with eigenvalue \(\lambda_j\), substitute \(\Sigma v_j=\lambda_jv_j\). When the directions are orthogonal, \(v_i^\mathsf{T}v_j=0\), so the covariance vanishes. This establishes the relationship between the eigenvector construction and uncorrelated component scores.
For a centered sample, the same calculation holds with the sample covariance matrix \(S\): the sample covariance between the two score columns is \(v_i^\mathsf{T}Sv_j\). Thus the conclusion is not limited to a population covariance matrix. It also applies to the scores obtained by projecting centered observations in a PCA based on \(S\).
Zero covariance means the scores are uncorrelated. It does not, in general, mean they are statistically independent. Independence requires additional distributional conditions; it does not follow from PCA orthogonality alone. This distinction matters when interpreting the transformed variables in a risk model.
Ties and Common Exam Traps
If two or more remaining eigenvalues are equal, the variance objective has the same maximum in every unit direction within their shared eigenspace. The chosen directions can be made mutually orthogonal, but the data covariance matrix does not select one unique basis within that tied space. The successive-maximization result therefore identifies a valid orthonormal set of directions, not always a unique set.
A related trap is to assume that any two perpendicular directions must yield uncorrelated scores. That is false unless the covariance structure supports it. For example, let \(\Sigma=\begin{pmatrix}4&0\\0&1\end{pmatrix}\), \(u=(1,1)^\mathsf{T}/\sqrt{2}\), and \(w=(1,-1)^\mathsf{T}/\sqrt{2}\). The directions are unit length and \(u^\mathsf{T}w=0\), but \(u^\mathsf{T}\Sigma w=(4-1)/2=1.5\), not zero. These directions are orthogonal, yet their scores are correlated. For principal directions, both ingredients matter: the directions are orthogonal eigenvectors of the covariance matrix.
Check Your Understanding
For each problem, identify the constrained direction or covariance result requested. Check that unit length and the stated orthogonality conditions are included.
- A centered two-variable dataset has covariance matrix \(\begin{pmatrix}6&0\\0&2\end{pmatrix}\). State the first direction and the second direction constrained to be perpendicular to the first. (Answer: \(v_1=(1,0)^\mathsf{T}\), \(v_2=(0,1)^\mathsf{T}\), up to orientation.)
- A covariance matrix has orthonormal eigenvectors \(q_1,q_2,q_3\) with eigenvalues \(10,5,2\), respectively. For a unit direction constrained to be perpendicular to \(q_1\), what is the maximum objective value, and which direction attains it? (Answer: Maximum \(5\), attained by \(q_2\), up to orientation.)
- Suppose \(v_1\) and \(v_2\) are unit eigenvectors of a covariance matrix, with \(v_1^\mathsf{T}v_2=0\) and \(\Sigma v_2=4v_2\). Find the covariance between the corresponding centered component scores. (Answer: \(v_1^\mathsf{T}\Sigma v_2=4v_1^\mathsf{T}v_2=0\).)
- A symmetric covariance matrix has remaining eigenvalues \(7,7,1\) after the first direction is selected. What is the maximum objective for the next direction, and is its maximizing direction unique? (Answer: Maximum \(7\); not unique, since any unit vector in the two-dimensional eigenspace for eigenvalue \(7\) attains it.)
- For \(\Sigma=\begin{pmatrix}5&0\\0&2\end{pmatrix}\), take perpendicular unit directions \(u=(1,1)^\mathsf{T}/\sqrt{2}\) and \(w=(1,-1)^\mathsf{T}/\sqrt{2}\). Are their scores uncorrelated? Compute their covariance and explain what this demonstrates. (Answer: No; \(u^\mathsf{T}\Sigma w=(5-2)/2=1.5\). Perpendicular directions alone do not guarantee uncorrelated scores.)