Why Center the Data?
Principal component analysis (PCA) describes variation in a collection of observations. The direction of greatest variation is not generally the direction pointing from the origin toward the data: it is the direction along which the observations vary most around their mean. Centering the data makes that distinction precise. It also makes the singular value decomposition (SVD) directly reveal both the principal directions and the amount of variation in each one.
Suppose there are \(d\) features measured on \(m\) observations. We arrange the observations as columns, so the data matrix is \(\mathbf{X}\in\mathbb{R}^{d\times m}\). Its \(i\)th row records feature \(i\) across all observations, and its \(j\)th column is observation \(j\). Assume \(m\ge2\) when using sample variance. Let \(\mathbf{1}_m\in\mathbb{R}^m\) be the column vector whose entries are all \(1\).
The product \(\bar{\mathbf{x}}\mathbf{1}_m^{\top}\) repeats the same feature-mean vector in all \(m\) columns. This matters: the mean is computed separately for each feature, across observations. Subtracting a different mean from each column would not center the features.
A principal direction is a unit feature-space direction that maximizes this sample variance. Further principal directions maximize the same variance subject to being orthogonal to the directions already chosen. A direction is a vector in the feature space, not an observation: in the SVD of \(\mathbf{A}\), principal directions come from the left singular vectors.
What Centering Guarantees
Centering removes the mean from each feature, so the columns of \(\mathbf{A}\) add to zero. If every observation is shifted by the same vector, centering removes that shift again. Both facts follow directly from the definition.
Proof. By the definition of \(\bar{\mathbf{x}}\), \(\mathbf{X}\mathbf{1}_m=m\bar{\mathbf{x}}\). Also \(\mathbf{1}_m^{\top}\mathbf{1}_m=m\). Therefore \(\mathbf{A}\mathbf{1}_m=\mathbf{X}\mathbf{1}_m-\bar{\mathbf{x}}(\mathbf{1}_m^{\top}\mathbf{1}_m)=m\bar{\mathbf{x}}-m\bar{\mathbf{x}}=\mathbf{0}\). Now replace \(\mathbf{X}\) by \(\mathbf{X}+\mathbf{c}\mathbf{1}_m^{\top}\). Its mean is \(\frac{1}{m}(\mathbf{X}+\mathbf{c}\mathbf{1}_m^{\top})\mathbf{1}_m =\bar{\mathbf{x}}+\mathbf{c}\). Its centered matrix is consequently \(\mathbf{X}+\mathbf{c}\mathbf{1}_m^{\top}-(\bar{\mathbf{x}}+\mathbf{c})\mathbf{1}_m^{\top}=\mathbf{X}-\bar{\mathbf{x}}\mathbf{1}_m^{\top}=\mathbf{A}\). Thus both claims hold. \(\square\)
The first identity says the centered feature values sum to zero across observations. The second says that the principal directions and variances calculated from centered data do not depend on a common choice of origin in feature space. It does not say that arbitrary changes to the features, such as rescaling one feature, leave PCA unchanged.
Worked Example: Centering a One-Feature Data Set
Three observations of one feature are \(5\), \(6\), and \(7\). Write them as columns, center them, and find their sample variance from the SVD.
The data matrix and feature mean are \(\mathbf{X}=\begin{bmatrix}5&6&7\end{bmatrix}\) and \(\bar{\mathbf{x}}=(5+6+7)/3=6\). Subtract \(6\) from each entry:
An SVD is \(\mathbf{A}=\mathbf{u}_1\sigma_1\mathbf{v}_1^{\top}\), where \(\mathbf{u}_1=\begin{bmatrix}1\end{bmatrix}\), \(\sigma_1=\sqrt{2}\), and \(\mathbf{v}_1=\begin{bmatrix}-1/\sqrt{2}&0&1/\sqrt{2}\end{bmatrix}^{\top}\). Multiplying gives \(\sqrt{2}\mathbf{v}_1^{\top}=\begin{bmatrix}-1&0&1\end{bmatrix}\), so the SVD reproduces \(\mathbf{A}\). The only feature-space direction, up to sign, is \(\mathbf{u}_1\). Its sample variance is \(\sigma_1^2/(m-1)=2/2=1\).
Check: The centered values sum to \(-1+0+1=0\), and their sample variance is \(((-1)^2+0^2+1^2)/2=1\), agreeing with the SVD calculation.
Singular Values and Principal Directions
Take a full SVD of the centered matrix, with its singular values ordered from largest to smallest: \(\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\). Write \(\mathbf{u}_1,\dots,\mathbf{u}_d\) for the columns of the orthogonal matrix \(\mathbf{U}\), and \(\sigma_1\ge\cdots\ge\sigma_d\ge0\) for the singular values, including zeros as needed. The sample covariance matrix is \(\mathbf{C}=\frac{1}{m-1}\mathbf{A}\mathbf{A}^{\top}\). The key connection is that its principal directions are the left singular vectors of \(\mathbf{A}\), and their variances are the squared singular values divided by \(m-1\).
Proof. For a unit direction \(\mathbf{z}\), the vector \(\mathbf{A}^{\top}\mathbf{z}\) contains its projected centered scores, so \(s^2(\mathbf{z})=\|\mathbf{A}^{\top}\mathbf{z}\|^2/(m-1)\). Since \(\mathbf{A}\mathbf{A}^{\top}=\mathbf{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{\top}\mathbf{U}^{\top}\), write \(\mathbf{z}=\sum_{i=1}^{d}c_i\mathbf{u}_i\). Orthogonality of the columns of \(\mathbf{U}\) gives \(\sum_i c_i^2=\|\mathbf{z}\|^2=1\), and $$ s^2(\mathbf{z})=\frac{\mathbf{z}^{\top}\mathbf{A}\mathbf{A}^{\top}\mathbf{z}}{m-1} =\frac{\sum_{i=1}^{d}\sigma_i^2c_i^2}{m-1} \le \frac{\sigma_1^2}{m-1}\sum_{i=1}^{d}c_i^2 =\frac{\sigma_1^2}{m-1}. $$ Equality is attained by \(\mathbf{z}=\mathbf{u}_1\). For the successive claim, if \(\mathbf{z}\) is orthogonal to \(\mathbf{u}_1,\dots,\mathbf{u}_{j-1}\), then \(c_1=\cdots=c_{j-1}=0\). The same calculation, now using \(\sigma_i\le\sigma_j\) for \(i\ge j\), gives \(s^2(\mathbf{z})\le\sigma_j^2/(m-1)\). The unit vector \(\mathbf{u}_j\) meets the orthogonality constraints and attains this value. If \(\sigma_j=0\), all remaining singular values are zero and every allowed direction has variance zero. If singular values tie, the inequality still holds, and each corresponding singular vector attains the tied value. \(\square\)
Equivalently, \(\mathbf{C}\mathbf{u}_i=(\sigma_i^2/(m-1))\mathbf{u}_i\). Thus the singular values give the sizes of the variation, while the left singular vectors give its directions in feature space. If instead a population-variance convention uses divisor \(m\), the directions are unchanged and the variances become \(\sigma_i^2/m\). The divisor must be stated when reporting numerical variances.
Worked Example: Two Features With Different Variances
Consider four observations of two features, stored as columns: \(\mathbf{X}=\begin{bmatrix}9&9&11&11\\18&22&18&22\end{bmatrix}\). Compute the centered matrix, an SVD, and the sample variance along each principal direction.
The feature means are \((9+9+11+11)/4=10\) and \((18+22+18+22)/4=20\). Subtracting them row by row gives
Set \(\mathbf{u}_1=\begin{bmatrix}0\\1\end{bmatrix}\), \(\mathbf{u}_2=\begin{bmatrix}1\\0\end{bmatrix}\), \(\mathbf{v}_1=\frac12\begin{bmatrix}-1\\1\\-1\\1\end{bmatrix}\), and \(\mathbf{v}_2=\frac12\begin{bmatrix}-1\\-1\\1\\1\end{bmatrix}\). Each vector has norm \(1\), and the two \(\mathbf{v}_i\) are orthogonal because their dot product is \((1-1-1+1)/4=0\). The two \(\mathbf{u}_i\) are also orthonormal. With singular values \(4\) and \(2\), the product is \(\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}=4\mathbf{u}_1\mathbf{v}_1^{\top}+2\mathbf{u}_2\mathbf{v}_2^{\top}\), which equals the displayed \(\mathbf{A}\): its first row is \((-1,-1,1,1)\), and its second row is \((-2,2,-2,2)\).
Therefore \(\mathbf{u}_1\) is the first principal direction and has sample variance \(4^2/(4-1)=16/3\). The second direction \(\mathbf{u}_2\) has sample variance \(2^2/3=4/3\). The covariance matrix confirms this: \(\mathbf{A}\mathbf{A}^{\top}=\begin{bmatrix}4&0\\0&16\end{bmatrix}\), so \(\mathbf{C}=\begin{bmatrix}4/3&0\\0&16/3\end{bmatrix}\). The total sample variance across the two directions is \(16/3+4/3=20/3\).
Check: Multiplying \(4\mathbf{u}_1\mathbf{v}_1^{\top}+2\mathbf{u}_2\mathbf{v}_2^{\top}\) gives the centered matrix entry by entry, and the covariance diagonal entries agree with the two calculated sample variances.
Worked Example: A Constant Feature Has Zero Variance
Consider \(\mathbf{X}=\begin{bmatrix}4&5&6\\7&7&7\end{bmatrix}\). Center the features and identify the nonzero principal direction and its variance.
The feature means are \(5\) and \(7\), so \(\mathbf{A}=\begin{bmatrix}-1&0&1\\0&0&0\end{bmatrix}\). An SVD has left singular vectors \(\mathbf{u}_1=\begin{bmatrix}1\\0\end{bmatrix}\) and \(\mathbf{u}_2=\begin{bmatrix}0\\1\end{bmatrix}\), singular values \(\sqrt{2}\) and \(0\), and first right singular vector \(\mathbf{v}_1=\begin{bmatrix}-1/\sqrt{2}\\0\\1/\sqrt{2}\end{bmatrix}\). Indeed, \(\sqrt{2}\mathbf{u}_1\mathbf{v}_1^{\top}=\begin{bmatrix}-1&0&1\\0&0&0\end{bmatrix}=\mathbf{A}\). The first direction has sample variance \(2/(3-1)=1\); the second has variance \(0\).
Check: The centered second feature is identically zero, so every projected score along \(\mathbf{u}_2\) is zero. The zero singular value and zero variance are consistent with the original data.
Common Mistakes in Centering and Interpretation
Subtracting the wrong means changes the question. For the two-feature data above, the feature means are \(10\) and \(20\). A tempting but incorrect operation is to subtract the mean of each observation column instead. For the first column, that column mean is \((9+18)/2=27/2\), so subtracting it from both entries gives \((-9/2,9/2)^{\top}\), not the correctly centered first column \((-1,-2)^{\top}\). This incorrectly centers each observation across features rather than centering each feature across observations. The correction is to compute one mean per row and subtract that mean from every entry in the row.
The singular vectors have different roles. With observations as columns, the left singular vectors are in \(\mathbb{R}^d\), the feature space, and give principal directions. The right singular vectors are in \(\mathbb{R}^m\), the observation-index space; they are not feature directions. Also, singular values themselves are not sample variances: the variance along \(\mathbf{u}_i\) is \(\sigma_i^2/(m-1)\) under the sample convention.
Centering is a data-preparation step that determines what variation PCA measures. The mean shift is removed, and the SVD then orders the remaining feature-space directions by sample variance. Constant features produce zero singular values in their directions; repeated singular values allow multiple equally valid choices of orthonormal principal directions within the tied directions.
Check Your Understanding
Use the convention that observations are columns and sample variance uses divisor \(m-1\).
- For \(\mathbf{X}=\begin{bmatrix}2&4&6\\1&1&4\end{bmatrix}\), find the feature means and centered data matrix.
- Explain why \(\mathbf{A}\mathbf{1}_m=\mathbf{0}\) for a centered data matrix \(\mathbf{A}\).
- If a centered data matrix has singular values \(5\) and \(2\), what are the sample variances along the corresponding principal directions when \(m=6\)?
- Which singular vectors of the centered data matrix give directions in feature space: left or right singular vectors?
- What happens to the centered data matrix if the same vector is added to every observation?
