Centering and Sample Covariance PCA

Why Centering Comes Before Sample PCA

Principal component analysis describes variation around the sample mean. The observations must therefore be centered before their covariance is used to find sample principal directions. This step is more than a convenient rearrangement: it makes the data matrix encode deviations from the mean, and its cross-product then records the sample covariances.

A useful computational connection follows. The eigenvectors of the sample covariance matrix are the right singular vectors of the centered data matrix. This lets us interpret sample PCA through either covariance eigenanalysis or a singular value decomposition (SVD), while keeping clear what each calculation represents.

Definition: Suppose \(n\geq 2\) observations are recorded on \(p\) variables. Write observation \(i\) as a column vector \(x_i\in\mathbb{R}^p\), and place the transposed observations in the \(n\times p\) data matrix \(X\), so row \(i\) of \(X\) is \(x_i^T\). The sample mean is \(\bar{x}=\frac{1}{n}\sum_{i=1}^n x_i\). The centered data matrix \(X_c\) has row \(i\) equal to \((x_i-\bar{x})^T\). The sample covariance matrix, using the conventional divisor \(n-1\), is \(S=\frac{1}{n-1}X_c^T X_c\).

The \(j\)th column of \(X\) contains all \(n\) observations of variable \(j\). Centering subtracts that column’s sample mean from each of its entries. The resulting columns have sample mean zero. The \((j,k)\) entry of \(X_c^T X_c\) is the sum, over observations, of the product of the centered values of variables \(j\) and \(k\); dividing by \(n-1\) gives their sample covariance.

Centering as a Matrix Operation

Let \(\mathbf{1}\in\mathbb{R}^n\) denote the vector whose entries are all one, and define the \(n\times n\) centering matrix

H=I_n-\frac{1}{n}\mathbf{1}\mathbf{1}^T.

Left-multiplication by \(H\) subtracts a column mean from each column of a matrix. In particular,

$$ X_c=HX. $$

To see this, note that \(\mathbf{1}\mathbf{1}^T X\) is a matrix whose every row equals the sum of the rows of \(X\). Thus every row of \(\frac{1}{n}\mathbf{1}\mathbf{1}^T X\) is \(\bar{x}^T\), and subtracting it from \(X\) centers each observation.

Result: The centering matrix is symmetric and idempotent: \(H^T=H\) and \(H^2=H\). Also, \(H\mathbf{1}=0\), so \(\mathbf{1}^T X_c=0\): each column of the centered data sums to zero.

These properties can be checked directly. Symmetry follows because both \(I_n\) and \(\mathbf{1}\mathbf{1}^T\) are symmetric. For idempotence, use \(\mathbf{1}^T\mathbf{1}=n\):

H^2 =I_n-\frac{2}{n}\mathbf{1}\mathbf{1}^T +\frac{1}{n^2}\mathbf{1}(\mathbf{1}^T\mathbf{1})\mathbf{1}^T =I_n-\frac{1}{n}\mathbf{1}\mathbf{1}^T =H.

Finally, \(H\mathbf{1}=\mathbf{1}-\mathbf{1}(\mathbf{1}^T\mathbf{1})/n=0\). Since \(H\) is symmetric, \(\mathbf{1}^T X_c=\mathbf{1}^T HX=(H\mathbf{1})^T X=0\), as claimed.

Worked Example: Centering a Two-Variable Data Set

Take three observations \(x_1=(1,1)^T\), \(x_2=(2,0)^T\), and \(x_3=(3,2)^T\). Construct the centered data matrix and sample covariance.

The sample mean is

\bar{x}=\frac{1}{3}\left[ \begin{pmatrix}1\\1\end{pmatrix} +\begin{pmatrix}2\\0\end{pmatrix} +\begin{pmatrix}3\\2\end{pmatrix} \right] =\begin{pmatrix}2\\1\end{pmatrix}.

Subtract this mean from each observation. In the convention that observations are rows, the result is

X_c= \begin{pmatrix} -1&0\\ 0&-1\\ 1&1 \end{pmatrix}.

Each column sums to zero. The cross-product and sample covariance are

X_c^T X_c= \begin{pmatrix}2&1\\1&2\end{pmatrix}, \qquad S=\frac{1}{3-1}X_c^T X_c =\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}.

For example, the off-diagonal entry is \((-1)(0)+(0)(-1)+(1)(1)\), divided by two, giving the sample covariance \(1/2\).

Covariance Eigenvectors and Right Singular Vectors

For any vector \(v\in\mathbb{R}^p\), the \(i\)th entry of \(X_c v\) is \((x_i-\bar{x})^T v\), the centered score of observation \(i\) in direction \(v\). Its squared length is

\|X_c v\|^2 =v^T X_c^T X_c v =(n-1)v^T S v.

Thus the covariance quadratic form \(v^T S v\), which is the sample variance of the centered scores when \(v\) is a unit vector, is tied directly to the length of \(X_c v\). The SVD makes this relationship precise.

Theorem: Let \(X_c=U D V^T\) be a singular value decomposition of the centered data matrix, where the positive singular values are \(d_1,\ldots,d_r\), and the corresponding columns of \(V\) are \(v_1,\ldots,v_r\). Then \(S=\frac{1}{n-1}X_c^T X_c\) has eigenvectors \(v_j\) with eigenvalues \(d_j^2/(n-1)\), for \(j=1,\ldots,r\). The remaining right singular vectors, if any, span the null space of \(S\) and have eigenvalue zero.

Proof. By the SVD and the orthonormality of the columns of \(U\),

$$ X_c^T X_c =V D^T U^T U D V^T =V D^T D V^T. $$

The diagonal entries of \(D^T D\) are the squared singular values, with zeros for directions beyond the rank \(r\). Therefore, for each right singular vector \(v_j\),

S v_j =\frac{1}{n-1}V D^T D V^T v_j =\frac{d_j^2}{n-1}v_j

when \(d_j>0\), and \(S v_j=0\) when \(d_j=0\). This proves the stated relationship. \(\square\)

The eigenvalues give another check on the score calculation. If \(v_j\) is a unit eigenvector of \(S\) with eigenvalue \(\lambda_j\), then

\|X_c v_j\|^2 =(n-1)v_j^T S v_j =(n-1)\lambda_j.

So the singular value associated with a nonzero principal direction is \(\sqrt{(n-1)\lambda_j}\). This is a relationship between the two decompositions of the same centered sample; it does not change what the observations or covariance matrix represent.

Worked Example: Checking the Singular Values

Continue with the centered matrix and covariance from the first example. The vectors \(v_1=(1,1)^T/\sqrt{2}\) and \(v_2=(1,-1)^T/\sqrt{2}\) are unit eigenvectors of \(S\). Direct multiplication gives

S v_1=\frac{3}{2}v_1, \qquad S v_2=\frac{1}{2}v_2.

Here \(n-1=2\), so the corresponding squared singular values must be \(2(3/2)=3\) and \(2(1/2)=1\). Check by applying the centered matrix:

X_c v_1= \begin{pmatrix}-1/\sqrt{2}\\-1/\sqrt{2}\\\sqrt{2}\end{pmatrix}, \qquad \|X_c v_1\|^2=\frac12+\frac12+2=3, $$ $$ X_c v_2= \begin{pmatrix}-1/\sqrt{2}\\1/\sqrt{2}\\0\end{pmatrix}, \qquad \|X_c v_2\|^2=\frac12+\frac12=1.

The singular values are therefore \(\sqrt{3}\) and \(1\), matching the covariance eigenvalues through \(d_j^2=(n-1)\lambda_j\).

Rank and Zero Sample Variance Directions

Centering forces the rows of \(X_c\) to sum to zero, so they cannot all be linearly independent. In particular, \(\operatorname{rank}(X_c)\leq n-1\). Also, \(\operatorname{rank}(X_c)\leq p\). Since \(S\) is a scaled cross-product, it has the same rank as \(X_c\). Consequently, if \(p>n-1\), at least one eigenvalue of \(S\) is zero. Zero eigenvalues can also occur when some variables are exact linear combinations of others in the centered sample.

A zero eigenvalue means the centered observations have no variation in that direction. It does not mean that an observation has a zero score in every direction; it means the score in that particular direction is zero for every centered row.

Worked Example: A Rank-Deficient Centered Sample

Consider the three observations \((0,0)^T\), \((1,2)^T\), and \((2,4)^T\). Their mean is \((1,2)^T\), so

X_c= \begin{pmatrix} -1&-2\\ 0&0\\ 1&2 \end{pmatrix}, \qquad X_c^T X_c= \begin{pmatrix}2&4\\4&8\end{pmatrix}.

There are \(n=3\) observations. Thus

S=\frac12X_c^T X_c =\begin{pmatrix}1&2\\2&4\end{pmatrix}.

The vector \(v_1=(1,2)^T/\sqrt{5}\) is a unit vector and satisfies \(S v_1=5v_1\). The orthogonal unit vector \(v_2=(-2,1)^T/\sqrt{5}\) satisfies \(S v_2=0\). Accordingly, the singular values of \(X_c\) are \(\sqrt{2\cdot5}=\sqrt{10}\) and \(0\). Indeed, the second variable is exactly twice the first in every observation, so the centered data have no variation in the orthogonal direction \(v_2\).

Common Mistake: Using the Uncentered Cross-Product

A frequent error is to use \(X^T X\) as though it were the sample covariance. That cross-product includes contributions from the location of the data relative to zero. The sample covariance instead uses deviations from the sample mean: \(X_c^T X_c/(n-1)\).

For example, take observations \((10,0)^T\), \((11,1)^T\), and \((12,2)^T\). The mean is \((11,1)^T\), and centering gives rows \((-1,-1)\), \((0,0)\), and \((1,1)\). Hence

S=\frac12 \begin{pmatrix}2&2\\2&2\end{pmatrix} =\begin{pmatrix}1&1\\1&1\end{pmatrix}.

By contrast, the uncentered cross-product is

X^T X= \begin{pmatrix}365&35\\35&5\end{pmatrix}.

It is not the centered sum of cross-products and does not give the sample covariance. The correction is to subtract the sample mean from each variable first, or equivalently to calculate \(X_c=HX\), and then form \(X_c^T X_c/(n-1)\).

Key takeaway: Center the columns of the observation-row data matrix to obtain \(X_c\), then form \(S=X_c^T X_c/(n-1)\). The covariance eigenvectors are the right singular vectors of \(X_c\), and each covariance eigenvalue is the associated squared singular value divided by \(n-1\).

Check Your Understanding

Use the definitions and relationships above to answer the following.

For an \(n\times p\) data matrix with observations in rows, what does left-multiplication by the centering matrix do to each column?
Why must the columns of \(X_c\) sum to zero?
If a centered matrix has a singular value \(d\), what is the corresponding sample covariance eigenvalue?
What does a zero eigenvalue of the sample covariance indicate about the centered observations?
For a unit covariance eigenvector with eigenvalue \(\lambda\), what is the squared length of its centered score vector?