Successive Components Under Orthogonality Constraints

Why Principal Directions Are Chosen Successively

The first principal direction captures the greatest variance among all unit-length linear scores. A second direction should describe substantial variation that the first score has not already captured, so it is required to be orthogonal to the first. Each later direction is chosen in the same way: maximize variance while remaining orthogonal to all directions already selected. This sequential rule is the constrained optimization characterization of principal components.

Let $X$ be a $p$-dimensional random vector with finite second moments, mean $\mu=E[X]$, and covariance matrix $\Sigma=\operatorname{Cov}(X)$. For a unit vector $a$, the centered linear score $a^T(X-\mu)$ has variance $a^T\Sigma a$. The first-component problem maximizes this quantity over unit vectors; the successive problems add orthogonality constraints.

Definition: Suppose $a_1,\ldots,a_{k-1}$ are the previously selected unit directions. The $k$th principal direction is any solution of $$ \max_{a\in\mathbb{R}^p} a^T\Sigma a \quad\text{subject to}\quad a^Ta=1,\qquad a^Ta_j=0\quad (j=1,\ldots,k-1). $$ The corresponding principal score is $a^T(X-\mu)$. The constraints require the new direction to have unit length and to be orthogonal to every earlier direction.

The covariance matrix is symmetric and positive semidefinite. Write its eigenvalues in nonincreasing order, including multiplicities, as $\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\geq 0$, and take an orthonormal eigenbasis $u_1,\ldots,u_p$ with $\Sigma u_i=\lambda_i u_i$. By the Rayleigh quotient characterization, the first maximum is $\lambda_1$, attained by a unit eigenvector associated with $\lambda_1$. The question now is what the same optimization gives after imposing orthogonality to the directions already chosen.

The Successive-Maximization Theorem

Theorem: For $1\leq k\leq p$, suppose the directions $a_1,\ldots,a_{k-1}$ are orthonormal eigenvectors of $\Sigma$, selected in nonincreasing eigenvalue order. Then the constrained maximum for $a_k$ is $\lambda_k$. Every maximizing $a_k$ is an eigenvector with eigenvalue $\lambda_k$, and a maximizing direction exists. When eigenvalues are tied, there may be more than one maximizing direction.

The condition on the earlier directions includes their eigenvector property, not just their mutual orthogonality. Sequential maximization supplies that property at each stage. To see why, the orthonormal eigenbasis can be chosen to begin with the already selected directions: within any repeated-eigenvalue eigenspace, extend the selected orthonormal directions to an orthonormal basis of that eigenspace. The remaining basis vectors are still eigenvectors, and their eigenvalues, in order, are $\lambda_k,\ldots,\lambda_p$.

Expand a feasible direction.
Write $a$ in the adapted orthonormal eigenbasis. Orthogonality to $a_1,\ldots,a_{k-1}$ makes the coefficients of those directions zero.

Bound its variance.
All remaining eigenvalues are at most $\lambda_k$, so the quadratic form cannot exceed $\lambda_k$ for a unit vector.

Check equality.
A remaining eigenvector with eigenvalue $\lambda_k$ attains the bound. Equality for any maximizer requires its nonzero coefficients to lie only in eigendirections with that eigenvalue.

Here is the bound and equality condition in full. Let $w_k,\ldots,w_p$ be the remaining orthonormal eigenvectors in the adapted basis, with respective eigenvalues $\lambda_k,\ldots,\lambda_p$. A feasible unit vector has the expansion $a=\sum_{i=k}^p c_iw_i$, where $\sum_{i=k}^p c_i^2=1$. Therefore

a^T\Sigma a =\sum_{i=k}^p\lambda_i c_i^2 \leq \lambda_k\sum_{i=k}^p c_i^2 =\lambda_k.

The bound is attained by $w_k$. Moreover, the difference between the bound and the objective is $\sum_{i=k}^p(\lambda_k-\lambda_i)c_i^2$. Each term is nonnegative, so equality holds exactly when $c_i=0$ for every $i$ with $\lambda_i<\lambda_k$. Thus every maximizer lies in the eigenspace for $\lambda_k$, and so satisfies $\Sigma a=\lambda_k a$. This proves both the maximum and the eigenvector claim. The argument applies at each stage, beginning with the first direction; after choosing a maximizer, it is an eigenvector that can be included in the adapted basis for the next stage. $\square$

Why the Constrained Maximizer Is an Eigenvector

There is also a useful stationarity argument that explains why the orthogonality constraints do not leave extra components in the eigenvector equation. Fix orthonormal eigenvectors $a_1,\ldots,a_{k-1}$ for the earlier directions, where $\Sigma a_j=\lambda_j a_j$. At a constrained maximum, the Lagrange multiplier equations for the unit-length and orthogonality constraints give

\Sigma a=\gamma a+\sum_{j=1}^{k-1}\eta_j a_j.

The constraint gradients are linearly independent: $a$ is a unit vector orthogonal to the $k-1$ orthonormal earlier directions. Thus the Lagrange multiplier condition applies on the feasible set. Taking the inner product with $a$ gives $\gamma=a^T\Sigma a$, since $a^Ta=1$ and $a^Ta_j=0$. For each $j$, symmetry of $\Sigma$ gives

a_j^T\Sigma a=(\Sigma a_j)^Ta =\lambda_j a_j^Ta=0.

Taking the inner product of the multiplier equation with $a_j$ now yields $\eta_j=0$. Hence $\Sigma a=\gamma a$: a constrained stationary direction is an eigenvector. This calculation alone does not establish which eigenvalue the maximizing direction has. The eigenbasis bound in the theorem supplies that missing ordering argument and shows that the maximum is specifically $\lambda_k$.

Worked Examples

Worked Example: Successive Directions for a Diagonal Covariance Matrix

Let $X$ have covariance matrix $\Sigma=\operatorname{diag}(5,3,1)$. Find the first three successive directions and their variances.

The coordinate vectors $e_1=(1,0,0)^T$, $e_2=(0,1,0)^T$, and $e_3=(0,0,1)^T$ are orthonormal eigenvectors with eigenvalues $5,3,1$, respectively. The first direction maximizes the variance at $e_1$, giving $a_1=e_1$ and variance $a_1^T\Sigma a_1=5$.

For the second direction, orthogonality to $e_1$ forces $a=(0,b,c)^T$, where $b^2+c^2=1$. Its variance is

a^T\Sigma a=3b^2+c^2 =3b^2+3c^2-2c^2 =3-2c^2\leq 3.

Equality holds when $c=0$, so $a_2=e_2$ or its negative, and the maximum variance is $3$. For the third direction, orthogonality to both $e_1$ and $e_2$ leaves only $a=e_3$ or its negative. Its variance is $1$. The successive maximum values are therefore $5,3,1$, in the same order as the eigenvalues.

Worked Example: Finding a Second Direction in a Nondiagonal Case

Consider

\Sigma= \begin{pmatrix} 4&1&0\\ 1&4&0\\ 0&0&1 \end{pmatrix}.

Define $q_1=(1,1,0)^T/\sqrt{2}$, $q_2=(1,-1,0)^T/\sqrt{2}$, and $q_3=(0,0,1)^T$. Direct multiplication gives $\Sigma q_1=5q_1$, $\Sigma q_2=3q_2$, and $\Sigma q_3=q_3$. These vectors are orthonormal, so $q_1$ is the first direction.

A unit vector orthogonal to $q_1$ can be written $a=sq_2+tq_3$, with $s^2+t^2=1$. Using the eigenvector relations, its variance is

a^T\Sigma a=3s^2+t^2 =3-2t^2\leq 3.

The maximum occurs at $t=0$, so $a_2=q_2$ or $-q_2$, with variance $3$. A third direction must be orthogonal to both $q_1$ and $q_2$, leaving $q_3$ or $-q_3$, with variance $1$. This example shows that the successive argument does not depend on the covariance matrix being diagonal in the original coordinates.

Worked Example: A Tied Eigenvalue and Multiple Valid Directions

Let $\Sigma=\operatorname{diag}(6,3,3,1)$. The first direction is $a_1=e_1$, with variance $6$. For a unit vector $a=(0,b,c,d)^T$ orthogonal to $e_1$, the variance is

a^T\Sigma a=3b^2+3c^2+d^2 =3-2d^2\leq 3.

Thus any unit vector in the plane spanned by $e_2$ and $e_3$ is a valid second direction, with variance $3$. For example, choose $a_2=e_2$. A third direction must be orthogonal to both $e_1$ and $e_2$; $a_3=e_3$ is valid and also has variance $3$. If instead the second direction is $a_2=\cos(\theta)e_2+\sin(\theta)e_3$, then $a_3=-\sin(\theta)e_2+\cos(\theta)e_3$ is an orthogonal unit direction with variance $3$. The maximum value is well-defined even though the maximizing directions need not be unique.

Common Mistake: Orthogonality Alone Is Not Enough

A common error is to assume that any unit vector orthogonal to the first direction is the second principal direction. Orthogonality makes a direction eligible for the second search; it does not ensure that its variance is as large as possible among eligible directions. In the first worked example, $e_3$ is orthogonal to $e_1$, but its variance is $1$, whereas the eligible direction $e_2$ has variance $3$. The correction is to impose the orthogonality constraints and then maximize the quadratic form over all directions that satisfy them.

Another point to keep straight is that the theorem concerns eigenvalue order, not a unique signed vector. Replacing $a_k$ by $-a_k$ leaves its variance and orthogonality constraints unchanged. Tied eigenvalues can also allow several different maximizing directions, as the last example illustrates. In every case, the maximum remaining variance is the next eigenvalue in nonincreasing order; the particular direction attaining it may depend on the choice made within a tied eigenspace.

Key takeaway: At stage $k$, maximize $a^T\Sigma a$ over unit vectors orthogonal to all earlier principal directions. The maximum is $\lambda_k$, and every maximizing direction is an eigenvector associated with that eigenvalue.

Check Your Understanding

Use the constrained-maximization definition and the theorem to answer these questions.

Why does the $k$th optimization include orthogonality constraints against every earlier direction, rather than only the first?
For $\Sigma=\operatorname{diag}(7,4,2)$, what is the maximum variance at the second stage, and which directions attain it?
In the Lagrange multiplier argument, why does symmetry of $\Sigma$ imply $a_j^T\Sigma a=0$ at a feasible direction?
What extra argument is needed beyond the Lagrange multiplier equations to identify the maximizing eigenvalue as $\lambda_k$?
If two consecutive eigenvalues are equal, must the corresponding maximizing directions be unique? Explain briefly.