A Worked Two-Dimensional PCA Calculation

From a Covariance Matrix to a PCA Calculation

In population PCA, the principal directions are unit eigenvectors of the covariance matrix. Their eigenvalues give the variances of the corresponding component scores. A two-variable calculation lets us see how these ideas fit together: first find the directions, then calculate their variances, and finally use one direction to approximate a centered observation.

Let \(X\) be a random vector with mean \(\mu\), and let \(Z=X-\mu\) be its centered version. For this tutorial, take its covariance matrix to be

A unit direction \(v\) produces the scalar score \(v^\mathsf{T}Z\). Its variance is \(v^\mathsf{T}\Sigma v\). The first principal direction maximizes this variance among unit directions; the second is orthogonal to the first and captures the remaining direction. Here we calculate these quantities directly rather than relying on a software output.

Worked Example: Eigenvalues and Unit Eigenvectors

Why the Eigenvalues Are the Projected Variances

The calculation above identifies candidate directions. The following derivation confirms exactly how their eigenvalues relate to score variability. For any fixed unit vector \(a\), the score \(a^\mathsf{T}Z\) has mean zero because \(E[Z]=0\). By the definition of covariance,

The second equality is the scalar identity \((a^\mathsf{T}Z)^2=a^\mathsf{T}ZZ^\mathsf{T}a\). The third moves the fixed vector \(a\) outside the expectation; this requires only that the second moments in \(\Sigma\) exist. In particular, for a unit eigenvector \(v_j\),

So the first score has variance approximately \(5.562\), and the second has variance approximately \(1.438\). Their sum is \(7\), which equals the total variance \(\operatorname{tr}(\Sigma)=4+3=7\). In this example the first component accounts for a proportion \(\lambda_1/7\approx0.795\) of the total variance.

Reconstructing an Observation with One Component

A component score describes one observation’s coordinate along a direction. For a centered observation \(z\), keeping only its first principal component gives the projection \(t_1v_1\), where \(t_1=v_1^\mathsf{T}z\). This reconstruction is a vector, whereas the eigenvalue describes variance across the population. The two quantities are related, but they are not interchangeable.

Reconstruction Error and Its Population Meaning

The previous example illustrates a general two-dimensional identity. For any centered vector \(z\), its coordinates in the orthonormal basis \(v_1,v_2\) are \(t_1=v_1^\mathsf{T}z\) and \(t_2=v_2^\mathsf{T}z\). Therefore \(z=t_1v_1+t_2v_2\). A one-component reconstruction retains the first term, so its residual is \(t_2v_2\). Taking squared lengths gives

For a random centered observation \(Z\), the expected squared one-component error is consequently

The last equality holds because the score has mean zero and variance \(\lambda_2\). For this covariance matrix, the population mean squared error from retaining one component is \((7-\sqrt{17})/2\approx1.438\). It need not equal the error for a particular observation: the worked observation’s squared error is approximately \(1.119\).

If the original observation is not centered, say \(x\) with population mean \(\mu\), apply the calculation to \(z=x-\mu\). The corresponding one-component reconstruction on the original scale is \(\mu+t_1v_1\). Adding the mean back is essential; the projection calculated here is specifically for a centered vector.

Common Mistakes and Checks

• Confusing an eigenvalue with an individual score. The eigenvalue is the variance of a component score across the population. A particular observation has a score such as \(t_1\approx2.980\), not a score equal to \(\lambda_1\).
• Using an unnormalized eigenvector as if it were a PCA direction. A scalar multiple is still an eigenvector, but projected variance equals the eigenvalue when the direction has unit length. Normalize before interpreting score variance.
• Reporting the retained part as the residual. The reconstruction is \(t_1v_1\); the residual is \(z-t_1v_1=t_2v_2\). In the example these are approximately \((2.35,1.83)^\mathsf{T}\) and \((0.65,-0.83)^\mathsf{T}\), respectively.
• Assuming each observation’s error equals the omitted eigenvalue. The eigenvalue \(\lambda_2\) is the expected squared error over the population. An individual error is \(t_2^2\), which can differ from that expectation.

A useful arithmetic check is that the component variances sum to the trace, \(5.562+1.438=7\), and that the centered observation’s squared length splits into its two squared scores, \(3^2+1^2=t_1^2+t_2^2\), up to rounding. These checks catch many calculation and transcription errors.