Centering Pitfalls: Mean Direction Versus Variation

Why Centering Changes the Question

Principal component analysis is often described as finding directions of greatest variation. That description refers to variation around the mean. If the data are analyzed without centering, a large mean can instead make a direction look important because observations lie far from the origin along it. The resulting leading direction may describe the location of the data cloud rather than its internal spread.

The distinction is between the covariance matrix, which measures variation around the population mean, and the uncentered second-moment matrix, which measures squared distance from the origin in each direction. These matrices coincide when the mean is zero, but need not otherwise.

Two Matrices, Two Objectives

Definition: Let \(\mathbf X\) be a random vector in \(\mathbb R^p\), with finite second moments, mean \(\boldsymbol\mu=E[\mathbf X]\), and covariance matrix \(\Sigma=E[(\mathbf X-\boldsymbol\mu)(\mathbf X-\boldsymbol\mu)^T]\). The uncentered second-moment matrix is \(M=E[\mathbf X\mathbf X^T]\). For a unit vector \(\mathbf a\), centered PCA evaluates the variance \(\operatorname{Var}(\mathbf a^T\mathbf X)\); an uncentered second-moment analysis evaluates \(E[(\mathbf a^T\mathbf X)^2]\).

A principal direction is a unit vector. The first centered principal direction maximizes \(\mathbf a^T\Sigma\mathbf a\) over unit vectors \(\mathbf a\); equivalently, it is a leading eigenvector of \(\Sigma\). The corresponding uncentered direction maximizes \(\mathbf a^TM\mathbf a\), and is a leading eigenvector of \(M\). These are different optimization problems when the mean contributes appreciably to the second moment.

The key identity follows by writing \(\mathbf X=(\mathbf X-\boldsymbol\mu)+\boldsymbol\mu\):

M=\Sigma+\boldsymbol\mu\boldsymbol\mu^T.

The added matrix has rank one when \(\boldsymbol\mu\ne\mathbf 0\). For any unit vector \(\mathbf a\), the corresponding scalar identity is

E[(\mathbf a^T\mathbf X)^2] =\operatorname{Var}(\mathbf a^T\mathbf X) +\{E[\mathbf a^T\mathbf X]\}^2 =\mathbf a^T\Sigma\mathbf a+(\mathbf a^T\boldsymbol\mu)^2.

Thus an uncentered analysis rewards both spread and alignment with the mean. In contrast, centered PCA removes the common location before measuring variation.

Worked Example: A Mean Tilts the Leading Direction

Let \(Z_1,Z_2\) be independent random variables, each taking values \(-1\) and \(1\) with probability \(1/2\). Define \(\mathbf X=(2+2Z_1,\;1+Z_2)^T\). Find the leading centered and uncentered directions.

Since \(E[Z_1]=E[Z_2]=0\), the mean is \(\boldsymbol\mu=(2,1)^T\). Independence and \(\operatorname{Var}(Z_1)=\operatorname{Var}(Z_2)=1\) give \(\Sigma=\begin{pmatrix}4&0\\0&1\end{pmatrix}\). Its leading eigenvector is \((1,0)^T\).

The second-moment matrix is \(M=\Sigma+\boldsymbol\mu\boldsymbol\mu^T =\begin{pmatrix}4&0\\0&1\end{pmatrix}+\begin{pmatrix}4&2\\2&1\end{pmatrix} =\begin{pmatrix}8&2\\2&2\end{pmatrix}\). Its characteristic equation is \((8-\lambda)(2-\lambda)-4=0\), or \(\lambda^2-10\lambda+12=0\). The larger eigenvalue is \(5+\sqrt{13}\). For a corresponding eigenvector \((x,y)^T\), the first row gives \((8-\lambda)x+2y=0\), so \(y/x=(\lambda-8)/2=(\sqrt{13}-3)/2\approx0.303\).

The uncentered direction is therefore tilted about \(16.8\) degrees from the centered direction \((1,0)^T\). The tilt comes from the mean's nonzero component in the second coordinate, even though the centered variation is four times as large in the first coordinate.

Why a Large Mean Can Dominate

The rank-one term does not guarantee that the uncentered leading direction exactly equals the mean direction. The covariance term can still matter, especially when the mean is modest. But when the mean is large compared with the scale of the covariance, the leading uncentered direction must be close to the mean. The following bound makes that statement precise.

Theorem: Suppose \(\boldsymbol\mu\ne\mathbf 0\), and let \(\mathbf v\) be any unit leading eigenvector of \(M=\Sigma+\boldsymbol\mu\boldsymbol\mu^T\), with eigenvalue \(\lambda_{\max}(M)\). Define \(\mathbf u=\boldsymbol\mu/\|\boldsymbol\mu\|\). Then \(\sin\angle(\mathbf v,\mathbf u)\leq \|\Sigma\|_{\mathrm{op}}/\|\boldsymbol\mu\|^2\), where the angle between directions is taken in \([0,\pi/2]\), and \(\|\Sigma\|_{\mathrm{op}}\) is the largest eigenvalue of the positive semidefinite matrix \(\Sigma\).

Proof. By the Rayleigh quotient characterization of the largest eigenvalue, \(\lambda_{\max}(M)\geq\mathbf u^TM\mathbf u =\|\boldsymbol\mu\|^2+\mathbf u^T\Sigma\mathbf u \geq\|\boldsymbol\mu\|^2\). Let \(P=I-\mathbf u\mathbf u^T\), the orthogonal projection onto the space perpendicular to \(\mathbf u\). Since \(P\boldsymbol\mu=\mathbf 0\), applying \(P\) to \(M\mathbf v=\lambda_{\max}(M)\mathbf v\) gives \(\lambda_{\max}(M)P\mathbf v=P\Sigma\mathbf v\). Therefore \(\|P\mathbf v\|\leq\|\Sigma\|_{\mathrm{op}}/\lambda_{\max}(M)\leq\|\Sigma\|_{\mathrm{op}}/\|\boldsymbol\mu\|^2\). For unit vectors, \(\|P\mathbf v\|=\sin\angle(\mathbf v,\mathbf u)\), proving the result. \(\square\)

The bound is informative when \(\|\boldsymbol\mu\|^2\) is much larger than the covariance's largest eigenvalue. It is a sufficient bound, not an exact formula for the angle; if it exceeds one, it gives no useful restriction. Its practical message is that a large location relative to the spread can strongly control the uncentered direction.

Worked Example: The Mean Direction Wins Exactly

Let \(\mathbf X=(2Z_1,\;3+Z_2)^T\), with the same independent symmetric variables \(Z_1,Z_2\) as above. Here \(\boldsymbol\mu=(0,3)^T\) and \(\Sigma=\begin{pmatrix}4&0\\0&1\end{pmatrix}\), so centered PCA selects \((1,0)^T\).

Adding the mean contribution gives \(M=\begin{pmatrix}4&0\\0&1\end{pmatrix}+\begin{pmatrix}0&0\\0&9\end{pmatrix} =\begin{pmatrix}4&0\\0&10\end{pmatrix}\). Its largest eigenvalue is \(10\), with eigenvector \((0,1)^T\). The directions are perpendicular: centered PCA chooses the direction with variance \(4\), while the uncentered analysis chooses the direction with variance \(1\) because its squared mean contribution is \(9\).

The objectives confirm the reversal directly. For \(\mathbf a=(1,0)^T\), the variance is \(4\) and the second moment is \(4\). For \(\mathbf a=(0,1)^T\), the variance is \(1\), the squared projected mean is \(9\), and the second moment is \(10\). Both the matrix calculation and the scalar objective give the same answer.

The Sample Version

For observations \(\mathbf x_1,\ldots,\mathbf x_n\), define the sample mean \(\bar{\mathbf x}=n^{-1}\sum_{i=1}^n\mathbf x_i\), the centered covariance with divisor \(n\), and the uncentered empirical second moment by

S_n=\frac1n\sum_{i=1}^n(\mathbf x_i-\bar{\mathbf x})(\mathbf x_i-\bar{\mathbf x})^T, \qquad Q_n=\frac1n\sum_{i=1}^n\mathbf x_i\mathbf x_i^T.

Expanding each outer product around \(\bar{\mathbf x}\) yields the exact sample decomposition:

Q_n=S_n+\bar{\mathbf x}\bar{\mathbf x}^T.

Indeed, \(\mathbf x_i=(\mathbf x_i-\bar{\mathbf x})+\bar{\mathbf x}\); after summing the expansion, the cross terms vanish because \(\sum_i(\mathbf x_i-\bar{\mathbf x})=\mathbf 0\). If the usual sample covariance uses divisor \(n-1\), it equals \(nS_n/(n-1)\). This positive scalar rescaling does not change its eigenvectors, but it does not remove the mean term from \(Q_n\).

Worked Example: A Sample Direction Reverses After Omitting Centering

Consider four observations in two variables: \((-2,9.5)^T,\;(2,9.5)^T,\;(-2,10.5)^T,\;(2,10.5)^T\). Their sample mean is \(\bar{\mathbf x}=(0,10)^T\). Centering gives the four deviations \((-2,-0.5)^T,\;(2,-0.5)^T,\;(-2,0.5)^T,\;(2,0.5)^T\).

The sum of the squared first coordinates is \(4+4+4+4=16\), and the sum for the second coordinates is \(0.25+0.25+0.25+0.25=1\). The cross-products cancel: \(1-1-1+1=0\). Thus \(S_n=\begin{pmatrix}4&0\\0&0.25\end{pmatrix}\), and centered PCA selects the first coordinate direction.

For the uncentered matrix, the first-coordinate second moment is \(16/4=4\). The second-coordinate second moment is \((9.5^2+9.5^2+10.5^2+10.5^2)/4=(90.25+90.25+110.25+110.25)/4=100.25\). The cross-product average is zero because the positive and negative first coordinates cancel at each second-coordinate value. Hence \(Q_n=\begin{pmatrix}4&0\\0&100.25\end{pmatrix}\), whose leading direction is the second coordinate. The large offset of the second coordinates, not their small centered spread, drives this result.

Translation and a Common Pitfall

Centering makes covariance PCA invariant to a common translation. If a constant vector \(\mathbf c\) is added to every observation, the new mean is \(\boldsymbol\mu+\mathbf c\), but each centered observation remains \(\mathbf x_i-\bar{\mathbf x}\). The centered covariance and its principal directions are unchanged. The uncentered matrix, however, becomes \(E[(\mathbf X+\mathbf c)(\mathbf X+\mathbf c)^T]=\Sigma+(\boldsymbol\mu+\mathbf c)(\boldsymbol\mu+\mathbf c)^T\). It depends on the chosen origin.

Wrong reasoning: “The second coordinate has the largest raw values, so it must have the largest variation.” Large values can reflect a nonzero mean rather than spread.
Correction: Decide which quantity answers the scientific question. If the target is variation around the data cloud's center, center the variables and use covariance PCA. If squared distance from a meaningful fixed origin is itself the target, the uncentered second moment may be appropriate.
Another pitfall: Calling \(Q_n\) a covariance matrix. It is an uncentered second moment; unless the mean is zero, it includes the outer product of the mean.

Key takeaway: Centered PCA describes variation around the mean. Uncentered second-moment analysis describes squared size relative to the origin, combining variation with the projected mean. A nonzero, sufficiently large mean can therefore dominate or even reverse the leading direction.

Check Your Understanding

Use the covariance-versus-second-moment distinction to answer the following.

State the population identity relating \(M\), \(\Sigma\), and \(\boldsymbol\mu\).
For a unit direction \(\mathbf a\), what extra term does the uncentered objective include beyond variance?
When do the centered and uncentered population matrices coincide?
Why can translating every observation leave centered PCA unchanged while altering uncentered PCA?
If the mean norm is large relative to \(\|\Sigma\|_{\mathrm{op}}\), what does the angle bound imply about a leading uncentered direction?