Introduction
A PCA calculation is a chain of connected steps: center the observations, summarize their variation in a covariance matrix, find the matrix's eigenpairs, and project the observations onto the resulting directions. Checking each link matters. A sign error in one covariance entry can change the principal direction, while using the wrong denominator changes the eigenvalues and the reported variance.
This tutorial carries out that chain for small, hypothetical datasets with two numeric variables. It uses sample covariance, with divisor \(n-1\), throughout. PCA is descriptive here: it does not require a normal distribution, but it does require that the numerical coding and scales of the variables be meaningful for the analysis.
For a covariance-based PCA, the eigenvectors of \(S\) give component directions, and the corresponding eigenvalues give the variances of the component scores. Since a covariance matrix is symmetric and positive semidefinite, its eigenvalues are real and nonnegative, and eigenvectors for distinct eigenvalues are orthogonal. Normalize each eigenvector to have length one before using it to calculate scores.
Why the Eigenvalues Describe Component Variance
Let \(\mathbf{q}\) be a unit direction and let \(\mathbf{z}\) be a centered observation. The score in this direction is \(t=\mathbf{z}\mathbf{q}\). Across the sample, its sample variance is \(\mathbf{q}^\mathsf{T}S\mathbf{q}\). The first principal direction maximizes this variance over unit directions. The eigenvector equation \(S\mathbf{q}=\lambda\mathbf{q}\) makes the variance in that direction equal to \(\lambda\):
Thus the largest eigenvalue is the variance captured by PC1. For two variables, the total variance is the trace of \(S\), which is the sum of its eigenvalues. The explained-variance proportion for component \(k\) is its eigenvalue divided by that sum.
Calculate each sample mean and subtract it from the corresponding observation.
Sum centered cross-products and divide each sum by \(n-1\).
Solve \(\det(S-\lambda I)=0\), then solve for and normalize each eigenvector.
Project each centered observation onto the unit eigenvectors; compare eigenvalues with total variance.
Worked Calculation: Covariance Matrix, Eigenpairs, and Scores
Worked Example: A Complete PCA for Three Observations
A hypothetical analyst records two numeric measurements for three items: \((1,2)\), \((-1,0)\), and \((0,-2)\). Using sample covariance, calculate the PCA, the scores, and the explained-variance proportions. The method is to carry the same centered data through each step; this avoids confusing raw observations with the centered values used by PCA.
Step 1: Center the observations. The first-variable mean is \((1-1+0)/3=0\), and the second-variable mean is \((2+0-2)/3=0\). The data are already centered, so the centered rows remain \((1,2)\), \((-1,0)\), and \((0,-2)\).
Step 2: Calculate the sample covariance matrix. The first-variable sum of squares is \(1^2+(-1)^2+0^2=2\). The second-variable sum of squares is \(2^2+0^2+(-2)^2=8\). The cross-product sum is \(1(2)+(-1)(0)+0(-2)=2\). Here \(n-1=2\), so
Step 3: Find the eigenvalues. The characteristic equation is
The quadratic formula gives \(\lambda_1=(5+\sqrt{13})/2\approx4.3028\) and \(\lambda_2=(5-\sqrt{13})/2\approx0.6972\). The first eigenvalue is larger, so its eigenvector gives the PC1 direction.
Step 4: Find unit eigenvectors. For \(\lambda_1\), the first row of \((S-\lambda_1 I)\mathbf{q}_1=0\) gives \(q_{12}=(\lambda_1-1)q_{11}\). Set \(q_{11}=1\) temporarily, so the direction is proportional to \((1,r)\), where \(r=(3+\sqrt{13})/2\approx3.3028\). Its length is \(\sqrt{1+r^2}\approx3.4508\). A unit eigenvector is therefore \(\mathbf{q}_1\approx(0.2898,0.9571)\). An orthogonal unit eigenvector for PC2 is \(\mathbf{q}_2\approx(-0.9571,0.2898)\). Eigenvector signs may be reversed without changing the PCA direction.
Step 5: Project each centered observation. A score is the dot product of a centered row and the component's unit eigenvector. Rounding scores to four decimal places gives:
Step 6: Interpret and check. The total variance is the trace, \(1+4=5\), and also equals \(\lambda_1+\lambda_2=5\). The PC1 proportion is \(4.3028/5\approx0.8606\), or \(86.06\%\); PC2 accounts for the remaining \(13.94\%\). The PC1 scores sum to zero, as expected for projections of centered data. Their sample variance is approximately \(4.3028\), matching \(\lambda_1\). This check confirms the projection and eigenvalue calculations are consistent.
A Covariance Sign That Changes the Answer
Worked Example: Checking the Cross-Product Sum
For the hypothetical observations \((0,0)\), \((2,0)\), and \((0,1)\), calculate the sample covariance matrix and eigenvalues. The method is to center first and then form every cross-product from the centered values; this avoids treating the raw cross-products as covariances.
The means are \(\bar{x}=2/3\) and \(\bar{y}=1/3\). The centered rows are \((-2/3,-1/3)\), \((4/3,-1/3)\), and \((-2/3,2/3)\). The first-variable sum of squares is \(4/9+16/9+4/9=24/9=8/3\). The second-variable sum of squares is \(1/9+1/9+4/9=6/9=2/3\). The centered cross-product sum is
There are \(n-1=2\) degrees of freedom. Dividing each sum by two gives
The trace is \(5/3\), and the determinant is \((4/3)(1/3)-(-1/3)^2=1/3\). Therefore the characteristic equation is \(\lambda^2-(5/3)\lambda+1/3=0\), giving eigenvalues \((5+\sqrt{13})/6\approx1.4343\) and \((5-\sqrt{13})/6\approx0.2324\). Their sum is \(5/3\), and their product is \(1/3\), matching the trace and determinant. PC1 explains \(\{(5+\sqrt{13})/6\}/(5/3)=(5+\sqrt{13})/10\approx86.06\%\) of the sample variance.
When One Component Captures All the Variation
Worked Example: A Rank-One Dataset
Consider the hypothetical centered observations \((1,1)\), \((0,0)\), and \((-1,-1)\). Find the covariance matrix, eigenpairs, component scores, and explained variance. This example checks the limiting case in which both variables move together exactly.
The means are zero. Each variable has sum of squares \(1+0+1=2\), and the cross-product sum is also \(1+0+1=2\). Dividing by \(n-1=2\) gives \(S=\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Its characteristic equation is \((1-\lambda)^2-1=0\), or \(\lambda(\lambda-2)=0\). Thus \(\lambda_1=2\) and \(\lambda_2=0\), with unit eigenvectors \(\mathbf{q}_1=(1,1)/\sqrt{2}\) and \(\mathbf{q}_2=(1,-1)/\sqrt{2}\).
The PC1 scores are \(\sqrt{2},0,-\sqrt{2}\); the PC2 scores are \(0,0,0\). The sample variance of the PC1 scores is \((2+0+2)/2=2\), while the PC2 score variance is zero. Total variance is the trace \(1+1=2\), so PC1 explains \(2/2=100\%\) and PC2 explains \(0\%\). The zero eigenvalue expresses exact linear dependence in this dataset; it is not a negative variance or a computational failure.
Common Errors and Checks
A PCA result is easier to trust when its basic identities agree. For a two-variable sample covariance matrix, check that the trace equals the sum of the eigenvalues and that the determinant equals their product. For computed scores, check that each component's sample variance equals its eigenvalue, subject to rounding.
- Wrong cross-product sign. In the second example, the cross-product sum is \(2/9-4/9-4/9=-2/3\), so the sample covariance is \(-1/3\). A mistaken positive covariance changes the matrix and its eigenvectors. The correction is to multiply each pair of centered coordinates, preserve each product's sign, sum them, and only then divide by \(n-1\).
- Using \(n\) instead of \(n-1\). This produces a population-style covariance matrix for the observed sample rather than the sample covariance used here. Decide which convention is required before forming the matrix; do not switch conventions partway through.
- Projecting uncentered values. PCA scores in this calculation are projections of \(\mathbf{x}_j-\bar{\mathbf{x}}\), not of \(\mathbf{x}_j\) itself. Center before calculating scores.
- Using an unnormalized eigenvector. Scaling an eigenvector changes the numerical scores and their variances. Divide by its length so the direction vector has unit length.
- Interpreting eigenvector signs as substantive. If both entries of an eigenvector and all its scores are multiplied by \(-1\), the direction, variance, and explained proportion are unchanged.
Check Your Understanding
For each problem, use the sample covariance convention unless the question states otherwise. Answers are included for checking.
- For observations \((1,0)\) and \((0,1)\), calculate the sample covariance matrix and eigenvalues. Which unit direction can be used for PC1, and what proportion of variance does it explain? (Answer: \(S=\begin{pmatrix}0.5&-0.5\\-0.5&0.5\end{pmatrix}\); eigenvalues \(1,0\); PC1 direction \((1,-1)/\sqrt{2}\) or its negative; \(100\%\).)
- A centered dataset has covariance matrix \(S=\begin{pmatrix}4&2\\2&4\end{pmatrix}\). Find both eigenvalues, give a unit PC1 direction, and calculate the PC1 explained-variance proportion. (Answer: eigenvalues \(6,2\); PC1 direction \((1,1)/\sqrt{2}\) or its negative; \(6/(6+2)=75\%\).)
- For \((0,0)\), \((2,0)\), and \((0,1)\), state the sample covariance of the two variables and the sum of centered cross-products that produces it. (Answer: centered cross-product sum \(-2/3\), divided by \(n-1=2\), gives covariance \(-1/3\).)
- A centered observation is \((2,-1)\), and a unit PC1 eigenvector is \((1,0)\). Its covariance matrix has eigenvalues \(3\) and \(1\), with PC1 corresponding to \(3\). Calculate this observation's PC1 score and the PC1 explained-variance proportion. (Answer: score \(2\); PC1 proportion \(3/(3+1)=75\%\).)
- A two-variable sample covariance matrix has eigenvalues \(4\) and \(1\). For a centered observation, the unit eigenvectors are \((1,1)/\sqrt{2}\) for PC1 and \((1,-1)/\sqrt{2}\) for PC2. Calculate both scores for observation \((3,1)\), then state the total variance and PC1 proportion. (Answer: PC1 score \(4/\sqrt{2}=2\sqrt{2}\); PC2 score \(2/\sqrt{2}=\sqrt{2}\); total variance \(5\); PC1 proportion \(4/5=80\%\).)