A Worked Sample PCA with Three Variables

From Centered Data to Sample PCA

A sample PCA calculation connects several objects that are easy to confuse: the sample covariance matrix, its eigenvectors, the scores of individual observations, and the reconstructed data. Keeping their roles distinct makes the calculation auditable. In this tutorial, each observation has three variables, the observations have already been centered, and PCA uses the ordinary sample covariance matrix.

Let \(X\) be an \(n \times 3\) data matrix whose \(i\)th row is the centered observation \(x_i^T\). Its sample covariance matrix is \(S=X^T X/(n-1)\). If the columns of \(V\) are orthonormal eigenvectors of \(S\), ordered from largest eigenvalue to smallest, the principal component scores form the matrix \(T=XV\). The score in component \(j\) for observation \(i\) is \(t_{ij}=x_i^T v_j\).

The divisor \(n-1\) is used because this is the sample covariance matrix. Replacing it by another positive common divisor would multiply every eigenvalue by the same positive number and leave the eigenvectors unchanged; here we retain the sample divisor so that the eigenvalues have the usual sample-variance interpretation.

Worked Example: Forming the Sample Covariance Matrix

Consider four centered observations on three variables. The data matrix is

Each column sums to zero: for the first column the numerators are \(5+1-1-5=0\), for the second they are \(1+5-5-1=0\), and for the third they are \(1-1-1+1=0\). Thus the displayed data are centered. Since \(n=4\), the sample covariance divisor is \(n-1=3\).

For the first diagonal entry, the sum of squares in column one is \(25/8+1/8+1/8+25/8=52/8=13/2\). Hence \(S_{11}=(13/2)/3=13/6\). The second column has the same sum of squares, so \(S_{22}=13/6\). The third column has sum of squares \(1/4+1/4+1/4+1/4=1\), giving \(S_{33}=1/3\).

For the covariance between the first two variables, the four rowwise products are \(5/8,5/8,5/8,5/8\). Their sum is \(5/2\), so \(S_{12}=S_{21}=(5/2)/3=5/6\). For the first and third variables, the rowwise products are \(5/(4\sqrt{2}),-1/(4\sqrt{2}),1/(4\sqrt{2}),-5/(4\sqrt{2})\); they sum to zero. The products for the second and third variables also sum to zero: they are \(1/(4\sqrt{2}),-5/(4\sqrt{2}),5/(4\sqrt{2}),-1/(4\sqrt{2})\). Therefore

This is a symmetric matrix, as a sample covariance matrix must be. Its trace is \(13/6+13/6+1/3=14/3\), a useful check for the eigenvalue calculation that follows.

Eigenpairs and Observation Scores

The first two coordinates of \(S\) form a symmetric two-by-two block. Its diagonal entries are \(13/6\) and its off-diagonal entries are \(5/6\). The vector \((1,1)^T\) is an eigenvector of that block with eigenvalue \(13/6+5/6=3\); the vector \((1,-1)^T\) has eigenvalue \(13/6-5/6=4/3\). The third coordinate is already an eigenvector with eigenvalue \(1/3\). Normalizing the first two vectors gives the ordered eigenpairs

The vectors have unit length and are mutually orthogonal. Multiplication verifies the eigenvalue equations. For example, the first coordinate of \(Sv_1\) is \((13/6+5/6)/\sqrt{2}=3/\sqrt{2}\), and its second coordinate is the same, so \(Sv_1=3v_1\). For \(v_2\), those coordinates are \((13/6-5/6)/\sqrt{2}=4/(3\sqrt{2})\) and its negative, so \(Sv_2=(4/3)v_2\). Finally, \(Sv_3=(1/3)v_3\). The eigenvalues are nonnegative and sum to \(14/3\), agreeing with the trace check.

Why the Score Variances Equal the Eigenvalues

Proof. The score matrix is centered because the columns of \(X\) sum to zero and \(V\) is fixed: each column of \(T=XV\) is a linear combination of centered columns. Its sample covariance matrix is therefore \(T^TT/(n-1)\). Substitute \(T=XV\) and use the definition of \(S\):

Thus the score columns have sample variances \(\lambda_1,\lambda_2,\lambda_3\), and their sample covariances with one another are zero. In this example, the sums of squared score values are \(9\), \(4\), and \(1\). Dividing by \(n-1=3\) gives \(3\), \(4/3\), and \(1/3\), precisely the eigenvalues. The trace check also has a score-based interpretation: total sample variance is the sum of the three score variances, \(3+4/3+1/3=14/3\). \(\square\)

Retaining Two Components

Keeping the first two components means projecting each centered observation onto the span of \(v_1\) and \(v_2\). Write \(V_2=(v_1\ v_2)\). The reconstructed data matrix is \(\widehat{X}_2=XV_2V_2^T\); in score form, its rows are the first two scores multiplied back by their directions. This calculation retains the part of each observation along the first two directions and omits its part along \(v_3\).

To verify the error identity, note that \(X=XV V^T\), because \(VV^T=I\). The residual after retaining \(k\) directions is consequently

The matrices \((Xv_j)v_j^T\) for distinct \(j\) are orthogonal under the Frobenius inner product: for \(j\ne \ell\), their inner product is \(\operatorname{tr}(v_j (Xv_j)^T Xv_\ell v_\ell^T) =(v_j^Tv_\ell)((Xv_j)^TXv_\ell)=0\), since \(v_j^Tv_\ell=0\). The squared Frobenius norm of each term is \(\lVert Xv_j\rVert^2\lVert v_j\rVert^2=\lVert Xv_j\rVert^2\), since \(v_j\) has unit length. Also, \(\lVert Xv_j\rVert^2=v_j^T X^TXv_j=(n-1)v_j^TSv_j=(n-1)\lambda_j\). Adding these squared norms for the omitted directions proves the identity. \(\square\)

Common Mistakes and Practical Checks

A common mistake is to treat an eigenvalue as the sum of squared scores rather than their sample variance. In this example, the first score column has sum of squares \(9\), but its sample variance is \(9/(4-1)=3\), which is the eigenvalue. The divisor matters when interpreting eigenvalues as sample variances, even though it does not change the sample directions.

Another mistake is to reconstruct using the score values alone. Scores are coordinates in the principal-direction system; to return to the original three-variable coordinates, multiply each retained score by its eigenvector and add the resulting vectors. For two components, the first observation is reconstructed as \((3/2)v_1+1v_2\), not as the three-number vector \((3/2,1,0)\).

Several checks catch arithmetic errors: the covariance matrix should be symmetric; the eigenvectors should have unit length and be mutually orthogonal; the eigenvalues should sum to the trace of \(S\); and the total squared reconstruction error should match the omitted eigenvalues multiplied by \(n-1\). These checks do not replace the calculation, but they make it much easier to locate an inconsistent entry or divisor.