Projection Geometry for Principal Component Scores

Scores as Coordinates in a Principal Subspace

A principal component score is a coordinate, not a vector in the original variable space. For a centered observation, the retained scores tell us how far the observation extends along the retained principal directions. Multiplying those coordinates by the directions puts a vector back in the original variable space: its orthogonal projection onto the subspace spanned by those directions. Keeping these two roles distinct makes the geometry of principal component analysis (PCA) easier to read.

Let \(X\) be a \(p\)-dimensional random vector with finite second moments and mean \(\mu=E[X]\). Write \(Z=X-\mu\), and let \(\Sigma=\operatorname{Cov}(X)\). Since \(\Sigma\) is symmetric and positive semidefinite, choose orthonormal eigenvectors \(q_1,\ldots,q_p\), ordered with their eigenvalues from largest to smallest. For a retained dimension \(K\), where \(1\leq K\leq p\), place the first \(K\) directions in the \(p\times K\) matrix \(U_K=[q_1\ \cdots\ q_K]\). Then \(U_K^\mathsf{T}U_K=I_K\).

The score \(t_K\) records coordinates in the retained directions; \(\hat z_K\) is a \(p\)-dimensional vector. If \(z\) is an uncentered observation \(x\), apply these definitions to \(z=x-\mu\). The corresponding fitted observation on the original scale is \(\mu+\hat z_K\). Thus the projection is performed on the centered vector, with the mean added back only when expressing the fitted observation in the original coordinates.

Why This Is an Orthogonal Projection

An orthogonal projection of \(z\) onto a subspace \(\mathcal S\) is a vector in \(\mathcal S\) whose difference from \(z\) is orthogonal to every vector in \(\mathcal S\). For the retained principal subspace, the columns of \(U_K\) form an orthonormal basis. The score calculation takes inner products with those basis directions, and the fitted-vector calculation combines the basis directions using those scores.

Proof. Symmetry follows from

Using \(U_K^\mathsf{T}U_K=I_K\), idempotence follows:

The fitted vector \(P_Kz=U_Kt_K\) lies in \(\mathcal S_K\), since it is a linear combination of the columns of \(U_K\). Its residual satisfies

Every vector in \(\mathcal S_K\) has the form \(U_Ka\) for some \(a\in\mathbb{R}^K\). Its inner product with the residual is \((U_Ka)^\mathsf{T}r_K=a^\mathsf{T}U_K^\mathsf{T}r_K=0\). Hence the residual is orthogonal to the whole subspace, which verifies the defining property of the projection. This property also ensures uniqueness: if two vectors in \(\mathcal S_K\) both had residuals orthogonal to \(\mathcal S_K\), their difference would lie in \(\mathcal S_K\) and be orthogonal to itself, so its squared norm would be zero and the two vectors would be equal.

For the score and fitted-vector lengths, use orthonormality again:

Finally, \(z=\hat z_K+r_K\), and the two terms are orthogonal because \(\hat z_K\in\mathcal S_K\) and \(r_K\) is orthogonal to \(\mathcal S_K\). Expanding the squared norm gives

This proves all the stated identities. \(\square\)

Worked Projection Calculations

Reading the Identities Correctly

These calculations show what the score-to-vector conversion does: it maps the \(K\) retained coordinates back to the original \(p\)-dimensional space. The retained scores and the fitted vector have the same squared length because the columns of \(U_K\) are orthonormal. The residual describes the part of the centered observation perpendicular to every retained direction. These are geometric statements about the chosen subspace and the observation; they do not depend on the numerical values of the covariance eigenvalues.

When \(K=p\), the retained directions span all of \(\mathbb{R}^p\), so \(P_K=I_p\), the fitted vector equals \(z\), and the residual is zero. When \(K<p\), the residual may be nonzero, even though it has zero inner product with every retained direction. For an uncentered observation, the same statements apply after subtracting \(\mu\); adding \(\mu\) back shifts the fitted vector but does not change the centered projection calculations.

A practical verification can be made in three steps: calculate the scores, map them back with \(U_K\), and check orthogonality using \(U_K^\mathsf{T}r_K\). Comparing the score and fitted-vector lengths, and then checking the Pythagorean identity, provides independent arithmetic checks. The projection matrix \(P_K\) offers another route to the same fitted vector, since \(P_Kz=U_KU_K^\mathsf{T}z\).