Population Principal Components as Covariance Eigenvectors

Introduction

A multivariate measurement can vary in several directions at once. Principal component analysis (PCA) describes those directions using the population covariance matrix: its eigenvectors provide the directions, and the corresponding eigenvalues quantify the variation of the scores in those directions. This tutorial defines these population quantities and derives the connection between eigenvalues and score variances.

The definition requires only finite second moments. It does not require a multivariate normal distribution. In particular, knowing the covariance matrix determines the variances and covariances of the linear scores considered here, but it does not by itself determine their full distributions.

Population covariance and principal directions

Let \(X=(X_1,\ldots,X_p)^T\) be a random vector in \(\mathbb{R}^p\), with \(E[X_i^2]<\infty\) for every coordinate. Write its mean vector as \(\mu=E[X]\). Its population covariance matrix is

\Sigma=\operatorname{Cov}(X)=E\big[(X-\mu)(X-\mu)^T\big].

The entries of \(\Sigma\) are \(\Sigma_{ij}=\operatorname{Cov}(X_i,X_j)\). Thus \(\Sigma\) summarizes the variances of the coordinates and their pairwise covariances. It is symmetric and positive semidefinite, so the spectral theorem gives an orthonormal basis of eigenvectors with real, nonnegative eigenvalues.

Definition: A population principal direction is a unit eigenvector \(v_j\) of \(\Sigma\), with associated eigenvalue \(\lambda_j\). Order the eigenvalues, counting multiplicities, as \(\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_p\geq 0\), and choose corresponding orthonormal eigenvectors \(v_1,\ldots,v_p\). The population principal component score in direction \(v_j\) is the centered scalar random variable \(Z_j=v_j^T(X-\mu)\).

Centering makes the score have mean zero. The uncentered score \(v_j^TX\) has the same variance, since it differs from \(Z_j\) by the constant \(v_j^T\mu\). Each direction is taken to have unit length, \(v_j^Tv_j=1\), so its scale is fixed. Replacing \(v_j\) by \(-v_j\) gives the same direction in the geometric sense and reverses the sign of the score, without changing its variance.

An eigenvector is not changed by multiplying it by a nonzero constant, but a score's variance is multiplied by the square of that constant. The unit-length convention is therefore important when interpreting the eigenvalue as a score variance.

Why eigenvalues are score variances

The key calculation applies to any fixed vector \(a\in\mathbb{R}^p\). Its centered linear score is \(a^T(X-\mu)\). Finite second moments ensure that this score has a finite variance.

Theorem: For any fixed \(a\in\mathbb{R}^p\), \(\operatorname{Var}(a^T(X-\mu))=a^T\Sigma a\). If \(v_j\) is a unit eigenvector of \(\Sigma\) with eigenvalue \(\lambda_j\), then \(\operatorname{Var}(Z_j)=\lambda_j\). For two orthonormal eigenvectors \(v_i,v_j\) with distinct indices, \(\operatorname{Cov}(Z_i,Z_j)=0\).

Proof. Put \(Y=X-\mu\). Then \(E[Y]=0\), and the scalar \(a^TY\) has mean zero. Using the definition of variance and the identity \(a^TYY^Ta=(a^TY)^2\),

\operatorname{Var}(a^TY) =E[(a^TY)^2] =E[a^TYY^Ta] =a^TE[YY^T]a =a^T\Sigma a.

The expectation can be moved past the finite sums in the matrix product because all entries \(Y_kY_l\) are integrable: by Cauchy–Schwarz, \(E[|Y_kY_l|]\leq \sqrt{E[Y_k^2]E[Y_l^2]}<\infty\). For an eigenvector \(v_j\), the eigenvector equation gives \(\Sigma v_j=\lambda_jv_j\). Therefore

\operatorname{Var}(Z_j)=v_j^T\Sigma v_j =v_j^T(\lambda_jv_j) =\lambda_jv_j^Tv_j =\lambda_j.

For \(i\ne j\), the same finite-moment calculation gives

\operatorname{Cov}(Z_i,Z_j) =E[(v_i^TY)(v_j^TY)] =v_i^T\Sigma v_j =\lambda_jv_i^Tv_j =0,

since the eigenvectors are orthonormal. This proves the claims. \(\square\)

Consequently, the descending eigenvalue order is exactly the descending order of the variances of the corresponding principal component scores. If \(\lambda_i>\lambda_j\), then the score in direction \(v_i\) has greater variance than the score in direction \(v_j\). This statement concerns the defined eigenvector scores; characterizations in terms of optimizing over directions are separate results.

Worked population examples

Worked Example: Three Distinct Coordinate Variances

Suppose \(X=(X_1,X_2,X_3)^T\) has mean \(\mu=(2,-1,0)^T\) and covariance matrix

\Sigma= \begin{pmatrix} 9&0&0\\ 0&4&0\\ 0&0&1 \end{pmatrix}.

The coordinate vectors \(e_1=(1,0,0)^T\), \(e_2=(0,1,0)^T\), and \(e_3=(0,0,1)^T\) are unit eigenvectors. Direct multiplication gives \(\Sigma e_1=9e_1\), \(\Sigma e_2=4e_2\), and \(\Sigma e_3=e_3\). Thus the ordered eigenvalues are \(9,4,1\).

The centered scores are \(Z_1=X_1-2\), \(Z_2=X_2+1\), and \(Z_3=X_3\). Their variances are respectively \(e_1^T\Sigma e_1=9\), \(e_2^T\Sigma e_2=4\), and \(e_3^T\Sigma e_3=1\). Their pairwise covariances are zero, as the off-diagonal entries of \(\Sigma\) are zero. The first score therefore has the largest variance among these principal component scores.

Worked Example: Correlated Coordinates and Rotated Directions

Consider a random vector with mean zero and covariance

\Sigma= \begin{pmatrix} 4&3\\ 3&4 \end{pmatrix}.

Take \(v_1=(1,1)^T/\sqrt{2}\) and \(v_2=(1,-1)^T/\sqrt{2}\). Both are unit vectors and \(v_1^Tv_2=0\). Multiplying explicitly,

\Sigma v_1=\frac{1}{\sqrt{2}}\begin{pmatrix}7\\7\end{pmatrix}=7v_1, \qquad \Sigma v_2=\frac{1}{\sqrt{2}}\begin{pmatrix}1\\-1\end{pmatrix}=v_2.

The eigenvalues are \(7\) and \(1\), so these directions are in descending order. The scores are \(Z_1=(X_1+X_2)/\sqrt{2}\) and \(Z_2=(X_1-X_2)/\sqrt{2}\). Their variances, calculated directly, are

\operatorname{Var}(Z_1) =\frac{1}{2}(4+2\cdot3+4)=7, \qquad \operatorname{Var}(Z_2) =\frac{1}{2}(4-2\cdot3+4)=1.

The original coordinates each have variance \(4\), but their covariance \(3\) means that the sum and difference directions have different variances. The eigenvectors identify those directions, and their eigenvalues give the corresponding score variances.

Worked Example: A Repeated Eigenvalue

Suppose \(X=(X_1,X_2,X_3)^T\) has mean zero and covariance \(\Sigma=\operatorname{diag}(5,5,1)\). The first two coordinate vectors are eigenvectors with the same eigenvalue, \(5\). For any angle \(\phi\), define

v_\phi=(\cos\phi,\sin\phi,0)^T.

This vector has unit length, and multiplication shows \(\Sigma v_\phi=(5\cos\phi,5\sin\phi,0)^T=5v_\phi\). Thus every such direction is an eigenvector with eigenvalue \(5\). For example, the directions \(e_1\) and \(e_2\) give scores \(X_1\) and \(X_2\), each with variance \(5\); the rotated direction \(v_{\pi/4}\) gives score \((X_1+X_2)/\sqrt{2}\), also with variance

v_{\pi/4}^T\Sigma v_{\pi/4} =5\left(\frac12\right)+5\left(\frac12\right)=5.

The first principal direction is therefore not unique in this example. The two-dimensional eigenspace associated with eigenvalue \(5\) is determined, but no single direction within it is distinguished by covariance alone.

Ordering, ties, and interpretation

An eigenvalue of zero is possible. If \(\lambda_j=0\), then the theorem gives \(\operatorname{Var}(Z_j)=0\), so \(Z_j=0\) with probability one. Indeed, a random variable with finite second moment and zero variance equals its mean almost surely, and this score has mean zero. Such a direction has no variation in the population.

When eigenvalues are distinct, their ordered values identify the corresponding one-dimensional eigendirections, although each eigenvector still has an arbitrary sign. When eigenvalues are tied, the eigenvectors within the tied eigenspace are not uniquely determined. Any orthonormal basis of that eigenspace consists of eigenvectors with the same eigenvalue, hence scores with the same variance. The repeated-eigenvalue example illustrates why reporting one particular basis in a tied eigenspace should not be mistaken for a uniquely determined population direction.

Covariance determines the variance of each linear score and the covariance between principal component scores. In particular, the theorem shows that distinct scores are uncorrelated. Uncorrelatedness does not in general imply independence, and the covariance matrix does not determine whether the scores are normally distributed. Additional distributional assumptions would be needed for those conclusions.

The eigenvalue ordering is useful when the goal is to describe variation: it ranks the principal component scores by their population variances. It is also essential to interpret the result in the units and coordinates of the specified random vector. Changing the variables or their scales changes the covariance matrix and can change its eigenvectors and eigenvalues.

Common pitfall: using the uncentered second moment

A common mistake is to take eigenvectors of \(E[XX^T]\) and call them covariance principal directions without checking whether \(X\) is centered. In general,

E[XX^T]=\Sigma+\mu\mu^T.

To verify this identity, write \(X=(X-\mu)+\mu\), expand \(XX^T\), and take expectations. The two cross terms have expectation zero because \(E[X-\mu]=0\); the remaining terms are \(\Sigma\) and \(\mu\mu^T\). Unless \(\mu=0\), the uncentered second moment can have different eigenvectors and eigenvalues from the covariance matrix. Population PCA as defined here uses \(\Sigma\), so scores are formed from \(X-\mu\).

Key takeaway: For a finite-second-moment random vector, population principal directions are unit eigenvectors of its covariance matrix. The score along an eigenvector has variance equal to its eigenvalue; descending eigenvalues therefore order the principal component scores by variance, while ties allow nonunique directions.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

What moment condition on \(X\) ensures that its population covariance matrix is finite?
If \(\Sigma v=\lambda v\) and \(v\) is a unit vector, derive the variance of the centered score \(v^T(X-\mu)\).
Why are scores along two orthogonal covariance eigenvectors uncorrelated?
If two largest eigenvalues are equal, what is determined by the covariance matrix, and what may fail to be unique?
Under what condition does \(E[XX^T]\) equal the covariance matrix \(\Sigma\)?