A Two-Variable PCA By Hand

Introduction

A principal components analysis (PCA) can be reduced to a sequence of calculations when there are two measured variables: center the data, form a $2\times2$ covariance matrix, find its eigenvalues and eigenvectors, and project each centered observation onto the resulting directions. Working through these steps by hand makes it easier to catch errors that software output can conceal.

This tutorial uses covariance PCA: the analysis is based on the sample covariance matrix of the original measurement units. The observations are treated as a fixed data set; no probability distribution assumption is needed for these calculations. Inference about a population is a separate question.

Definition: Let $(x_i,y_i)$, $i=1,\ldots,n$, be observations on two quantitative variables. Their sample means are $\bar{x}$ and $\bar{y}$, and the centered observation $i$ is $\mathbf{z}_i=(x_i-\bar{x},\,y_i-\bar{y})^\mathsf{T}$. The sample covariance matrix is $$ \mathbf{S}=\frac{1}{n-1}\sum_{i=1}^{n}\mathbf{z}_i\mathbf{z}_i^\mathsf{T} =\begin{pmatrix}s_x^2&s_{xy}\\s_{xy}&s_y^2\end{pmatrix}. $$ A principal component direction is a unit eigenvector of $\mathbf{S}$. Its eigenvalue is the sample variance of the scores in that direction. The score for observation $i$ in direction $\mathbf{e}$ is $t_i=\mathbf{e}^\mathsf{T}\mathbf{z}_i$.

From Centering to the Covariance Matrix

Centering subtracts each variable’s own sample mean from every observation. It makes the coordinate origin the center of the observed data, so the PCA directions describe variation around that center rather than around zero. For two variables, the entries of the sample covariance matrix are

s_x^2=\frac{\sum_{i=1}^n(x_i-\bar{x})^2}{n-1},\qquad s_y^2=\frac{\sum_{i=1}^n(y_i-\bar{y})^2}{n-1},\qquad s_{xy}=\frac{\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{n-1}.

The divisor is $n-1$, because this is the sample covariance matrix. It is not $n$. The matrix is symmetric, so its eigenvalues are real and its eigenvectors for distinct eigenvalues are perpendicular. For a covariance matrix, the eigenvalues are nonnegative.

Worked Example: Centering and Finding Both Components

A hypothetical study records two measurements for four devices: $(1,2),(2,1),(3,4),(4,3)$. Use covariance PCA to find the covariance matrix, the component directions, and all scores.

Step 1: Find the means and center each row. The means are $\bar{x}=(1+2+3+4)/4=2.5$ and $\bar{y}=(2+1+4+3)/4=2.5$. The centered rows, in the original order, are $(-1.5,-0.5),(-0.5,-1.5),(0.5,1.5),(1.5,0.5)$.

Step 2: Calculate the covariance entries. The sum of squared first coordinates is $2.25+0.25+0.25+2.25=5$. The sum of squared second coordinates is also $5$. The sum of cross-products is $0.75+0.75+0.75+0.75=3$. Divide each sum by $n-1=3$:

\mathbf{S}=\begin{pmatrix}5/3&1\\1&5/3\end{pmatrix}.

Step 3: Find the eigenpairs. Multiplying $\mathbf{S}$ by $(1,1)^\mathsf{T}$ gives $(8/3,8/3)^\mathsf{T}$, so the unit eigenvector $\mathbf{e}_1=(1,1)^\mathsf{T}/\sqrt{2}$ has eigenvalue $\lambda_1=8/3$. Multiplying by $(1,-1)^\mathsf{T}$ gives $(2/3,-2/3)^\mathsf{T}$, so $\mathbf{e}_2=(1,-1)^\mathsf{T}/\sqrt{2}$ has eigenvalue $\lambda_2=2/3$. The directions are unit length and perpendicular.

Step 4: Project each centered row. For the first row, the scores are $(-1.5-0.5)/\sqrt{2}=-\sqrt{2}$ and $(-1.5+0.5)/\sqrt{2}=-1/\sqrt{2}$. Repeating the same dot products gives:

(t_{i1},t_{i2})= \left(-\sqrt{2},-\frac{1}{\sqrt{2}}\right), \left(-\sqrt{2},\frac{1}{\sqrt{2}}\right), \left(\sqrt{2},-\frac{1}{\sqrt{2}}\right), \left(\sqrt{2},\frac{1}{\sqrt{2}}\right).

Step 5: Check the variance interpretation. The first scores have mean zero and sum of squares $2+2+2+2=8$, so their sample variance is $8/3=\lambda_1$. The second scores have sum of squares $1/2+1/2+1/2+1/2=2$, so their sample variance is $2/3=\lambda_2$. The total variance is the trace of $\mathbf{S}$, $5/3+5/3=10/3$, which also equals $\lambda_1+\lambda_2$.

Method and mistake avoided. This is covariance PCA using centered rows and the sample covariance denominator. Using the raw first row $(1,2)$ for a score would give $3/\sqrt{2}$, not $-\sqrt{2}$; scores must use centered observations.

Why the Eigenvalues and Scores Match

The eigenvalue calculation can be done directly for any symmetric two-variable covariance matrix. Write $\mathbf{S}=\begin{pmatrix}a&b\\b&d\end{pmatrix}$. An eigenvalue $\lambda$ must make $\mathbf{S}-\lambda\mathbf{I}$ singular, so its determinant must be zero. Expanding that determinant gives the quadratic equation and its roots:

\det(\mathbf{S}-\lambda\mathbf{I})=(a-\lambda)(d-\lambda)-b^2=0, \qquad \lambda^2-(a+d)\lambda+(ad-b^2)=0, $$ $$ \lambda_{\pm}=\frac{a+d\pm\sqrt{(a-d)^2+4b^2}}{2}.

For each eigenvalue, solve $(\mathbf{S}-\lambda\mathbf{I})\mathbf{e}=\mathbf{0}$, then normalize the resulting eigenvector to length one. If the two eigenvalues are distinct, their directions are perpendicular. In the special case $a=d$ and $b=0$, the eigenvalues are tied; there is no unique preferred direction, since every unit direction has the same variance.

The eigenvalue equals the sample variance of the scores for a direct algebraic reason. Let $\mathbf{e}$ be a unit eigenvector with eigenvalue $\lambda$. The scores are $t_i=\mathbf{e}^\mathsf{T}\mathbf{z}_i$. Since the centered scores sum to zero, their sample variance is:

\frac{1}{n-1}\sum_{i=1}^{n}t_i^2 =\frac{1}{n-1}\sum_{i=1}^{n} \left(\mathbf{e}^\mathsf{T}\mathbf{z}_i\right) \left(\mathbf{z}_i^\mathsf{T}\mathbf{e}\right) =\mathbf{e}^\mathsf{T}\mathbf{S}\mathbf{e} =\mathbf{e}^\mathsf{T}(\lambda\mathbf{e}) =\lambda,

where the second equality uses the definition of $\mathbf{S}$, and the last equality uses $\mathbf{e}^\mathsf{T}\mathbf{e}=1$. This also gives a practical check: after calculating scores, their sample variance should agree with the corresponding eigenvalue.

Worked Example: A Second Data Set and Its Variance Interpretation

A hypothetical calibration exercise records $(12,21),(8,19),(11,22),(9,18)$. Find its covariance PCA and check the variance represented by each component.

Step 1: Center. The means are $\bar{x}=(12+8+11+9)/4=10$ and $\bar{y}=(21+19+22+18)/4=20$. The centered rows are $(2,1),(-2,-1),(1,2),(-1,-2)$.

Step 2: Form the covariance matrix. The first-coordinate squared sum is $4+4+1+1=10$; the second-coordinate squared sum is also $10$. The cross-product sum is $2+2+2+2=8$. Dividing by $3$ gives

\mathbf{S}=\begin{pmatrix}10/3&8/3\\8/3&10/3\end{pmatrix}.

Step 3: Decompose the matrix. The vector $(1,1)^\mathsf{T}$ is multiplied by $18/3=6$, and $(1,-1)^\mathsf{T}$ is multiplied by $2/3$. Thus the unit directions are $\mathbf{e}_1=(1,1)^\mathsf{T}/\sqrt{2}$, $\mathbf{e}_2=(1,-1)^\mathsf{T}/\sqrt{2}$, with eigenvalues $6$ and $2/3$.

Step 4: Calculate scores. Dot each centered row with each direction. The ordered scores are

\left(\frac{3}{\sqrt{2}},\frac{1}{\sqrt{2}}\right), \left(-\frac{3}{\sqrt{2}},-\frac{1}{\sqrt{2}}\right), \left(\frac{3}{\sqrt{2}},-\frac{1}{\sqrt{2}}\right), \left(-\frac{3}{\sqrt{2}},\frac{1}{\sqrt{2}}\right).

Step 5: Check and interpret. The first scores have squared sum $4(9/2)=18$, giving sample variance $18/3=6$. The second scores have squared sum $4(1/2)=2$, giving sample variance $2/3$. Total variance is $10/3+10/3=20/3$, also $6+2/3=20/3$. The first direction therefore has the larger score variance; in this data set its share of the total variance is $6/(20/3)=0.9$, or $90\%$. This percentage describes this covariance PCA, not a guarantee about new observations.

Method and mistake avoided. The eigenvectors are obtained from the symmetric covariance matrix, and the scores are projections of centered—not raw—rows. Checking both score variances against $6$ and $2/3$ verifies the decomposition and the projection arithmetic.

A Component Direction That Is Not at 45 Degrees

The principal directions need not be the sum-and-difference directions $(1,1)^\mathsf{T}/\sqrt{2}$ and $(1,-1)^\mathsf{T}/\sqrt{2}$. The following example uses unequal eigenvector coordinates and illustrates why solving the eigenvector equation matters.

Worked Example: Finding Unequal-Coordinate Loadings and Scores

Consider the hypothetical centered measurements $(4,2),(-4,-2),(1,-2),(-1,2)$. Their coordinate sums are both zero, so the displayed observations are already centered. Find the two components and verify their sample variances.

Step 1: Calculate the covariance matrix. The sum of squared first coordinates is $16+16+1+1=34$; the sum of squared second coordinates is $4+4+4+4=16$. The cross-product sum is $8+8-2-2=12$. Since $n-1=3$,

\mathbf{S}=\begin{pmatrix}34/3&4\\4&16/3\end{pmatrix}.

Step 2: Verify the eigenpairs. For $\mathbf{e}_1=(2,1)^\mathsf{T}/\sqrt{5}$, multiplication by $\mathbf{S}$ gives $(80/3,40/3)^\mathsf{T}/\sqrt{5}=(40/3)\mathbf{e}_1$. For $\mathbf{e}_2=(1,-2)^\mathsf{T}/\sqrt{5}$, it gives $(10/3,-20/3)^\mathsf{T}/\sqrt{5}=(10/3)\mathbf{e}_2$. Both directions have length one, and their dot product is $(2-2)/5=0$. The eigenvalues are $40/3$ and $10/3$.

Step 3: Calculate all scores. Project the observations in order onto the two unit directions:

(t_{i1},t_{i2})= (2\sqrt{5},0),\ (-2\sqrt{5},0),\ (0,\sqrt{5}),\ (0,-\sqrt{5}).

For example, the first observation’s first score is $(4\cdot2+2\cdot1)/\sqrt{5}=10/\sqrt{5}=2\sqrt{5}$, and its second score is $(4-2\cdot2)/\sqrt{5}=0$.

Step 4: Check the variances. The first scores have squared sum $20+20=40$, so their sample variance is $40/3$. The second scores have squared sum $5+5=10$, so their sample variance is $10/3$. These match the eigenvalues. Their sum $50/3$ also equals the trace $34/3+16/3=50/3$.

Method and mistake avoided. Solving for eigenvectors rather than assuming 45-degree directions is essential here. The verified directions $(2,1)^\mathsf{T}/\sqrt{5}$ and $(1,-2)^\mathsf{T}/\sqrt{5}$ are perpendicular but are not equal-coordinate directions.

Common Calculation Traps

A compact hand calculation is especially vulnerable to using a correct formula at the wrong stage. Keep the mean, denominator, and direction associated with each step.

Dividing by $n$. In Example 1, dividing the centered sums by $4$ would produce $\begin{pmatrix}5/4&3/4\\3/4&5/4\end{pmatrix}$, rather than the sample covariance matrix. Its eigenvalues would be $2$ and $1/2$, not the correct sample values $8/3$ and $2/3$.
Projecting the original row. A score uses $x_i-\bar{x}$ and $y_i-\bar{y}$. Applying a loading to the uncentered measurements shifts the scores and does not give the PCA scores defined here.
Forgetting to normalize an eigenvector. Eigenvectors may be multiplied by nonzero constants, but scores and their variances change with that scaling. Use unit eigenvectors when interpreting the eigenvalues as score variances.
Mixing the component directions. Calculate every score with the coordinates of its own eigenvector. A score is a dot product; changing a sign or coordinate in only one part of that product changes the answer.

Key takeaway: For a two-variable covariance PCA, center first, divide covariance sums by $n-1$, use unit eigenvectors, and project each centered observation. The sample variance of each resulting score column must equal its eigenvalue, and the eigenvalues must sum to the trace of the covariance matrix.

Check Your Understanding

Work each question with the stated covariance-PCA convention. Answers are included for checking.

A centered data set has covariance matrix $\begin{pmatrix}2/3&0\\0&2/3\end{pmatrix}$. What are its eigenvalues, and is there a unique first-component direction? (Answer: Both eigenvalues are $2/3$; there is no unique first direction because every unit direction has the same variance.)
Four measurements are $(3,5),(5,3),(7,5),(5,7)$. Find the sample means and covariance matrix. (Answer: $\bar{x}=\bar{y}=5$, and $\mathbf{S}=\begin{pmatrix}8/3&0\\0&8/3\end{pmatrix}$.)
For the centered observations $(-2,1),(-1,-1),(1,1),(2,-1)$, project each observation, in the given order, onto the direction $(1,-1)^\mathsf{T}/\sqrt{2}$. (Answer: $-3/\sqrt{2},\,0,\,0,\,3/\sqrt{2}$.)
The covariance matrix is $\begin{pmatrix}3&1\\1&3\end{pmatrix}$. Give unit eigenvectors, their eigenvalues, and the total variance. (Answer: Directions $(1,1)^\mathsf{T}/\sqrt{2}$ and $(1,-1)^\mathsf{T}/\sqrt{2}$, with eigenvalues $4$ and $2$; total variance $6$.)
A covariance matrix is $\begin{pmatrix}5&1\\1&2\end{pmatrix}$. Find both eigenvalues and check their sum against the total variance. (Answer: $(7+\sqrt{13})/2$ and $(7-\sqrt{13})/2$; their sum is $7$, equal to the trace $5+2$.)

What You'll Learn