Scaling Variables and Choosing Units

Why Measurement Units Matter in PCA

A covariance-based principal component analysis depends on the numerical scale of each variable. If a length is recorded in meters rather than centimeters, its numerical values—and its covariances with other variables—change. When every variable is rescaled in the same way, the principal directions are preserved. When variables are rescaled separately, the relative contributions to variance can change, and so can the directions PCA selects.

The key is to distinguish a change in the units used to record a variable from a change in the underlying observations. A change of units multiplies each variable by a fixed conversion factor. It does not change the ordering of observations on that variable, but it can change the numerical optimization that defines covariance PCA.

For covariance PCA, a unit-length vector \(v\) specifies the score \(v^T X\); its variance is \(v^T\Sigma v\). The first principal direction maximizes this variance over vectors with \(v^T v=1\), and subsequent directions maximize variance under orthogonality constraints. The scale changes below affect that optimization through the covariance matrix.

The Covariance Matrix After a Change of Units

For \(Y=DX\), the covariance between components \(Y_j=d_jX_j\) and \(Y_k=d_kX_k\) is \(d_jd_k\operatorname{Cov}(X_j,X_k)\). In matrix form, this gives the first central result.

Proof. Write \(\mu=E[X]\). By linearity of expectation, \(E[Y]=DE[X]=D\mu\), and therefore \(Y-E[Y]=D(X-\mu)\). Using the definition of covariance,

The matrix \(D\) is fixed, so taking it outside the expectation requires no interchange of a limit and an integral. If a constant vector \(b\) is also added, \(Y=DX+b\), then \(Y-E[Y]=D(X-\mu)\) as well; the covariance remains \(D\Sigma D^T\). \(\square\)

For diagonal \(D\), each covariance entry is multiplied by the product of the two variables’ conversion factors:

In particular, a variable’s variance is multiplied by \(d_j^2\). Thus a change of units can change not only the displayed variance values, but also how much each variable contributes to the total variance used by PCA.

Uniform Rescaling Preserves Principal Directions

When every variable is multiplied by the same nonzero constant \(c\), the new covariance matrix is \(c^2\Sigma\). The principal directions remain the same, although the eigenvalues, which represent component variances, are multiplied by \(c^2\).

Proof. By the covariance transformation result, \(\operatorname{Cov}(cX)=c^2\Sigma\). If \(\Sigma v=\lambda v\), then

So every eigenvector of \(\Sigma\) is an eigenvector of \(c^2\Sigma\), with eigenvalue \(c^2\lambda\). Since \(c^2>0\), multiplication by \(c^2\) preserves the ordering of the eigenvalues. Thus the principal directions are unchanged. \(\square\)

This result covers common conversions applied to all variables, such as converting every length from centimeters to meters. It does not say that the numerical component variances are unchanged: they are rescaled along with the data.

Separate Rescaling Can Change the Directions

If the conversion factors differ by variable, the covariance becomes \(D\Sigma D\), which is generally not a constant multiple of \(\Sigma\). Its eigenvectors therefore need not match those of the original covariance matrix. There is also a useful way to describe what the new PCA means in the original coordinates.

Derivation. Since \(D\) is diagonal and invertible, it is symmetric and \(v=D^{-1}a\). Hence

The score variance is consequently \(\operatorname{Var}(a^TX)=a^T\Sigma a\), while unit length of \(v\) imposes \(a^TD^{-2}a=1\). Equivalently, if \(v\) is an eigenvector of \(D\Sigma D\) with eigenvalue \(\lambda\), then multiplying \(D\Sigma Dv=\lambda v\) by \(D^{-1}\) gives

This equation makes clear why the new direction, expressed in the original variables, is generally not an ordinary eigenvector of \(\Sigma\): it solves a different constrained optimization problem.

Axes Can Stay Fixed While Their Ordering Changes

Separate scaling does not guarantee that every direction changes. For example, if the covariance matrix is diagonal, its coordinate axes are eigenvectors before and after any diagonal rescaling. But the eigenvalues can change by different factors, so which axis is the first principal direction can switch.

Common Pitfall: Treating Every Unit Change Like a Common Factor

A frequent error is to reason that because changing units multiplies measurements by constants, it must only multiply all eigenvalues by one constant and leave PCA directions intact. That conclusion is correct only when the same factor applies to every variable. With distinct factors, each covariance entry is multiplied by \(d_jd_k\), and the matrix is generally altered in a way that changes its eigenvectors or their ordering.

A practical check is to write the conversion matrix \(D\) explicitly and calculate \(D\Sigma D\). If \(D=cI\), the directions are preserved. If the diagonal entries differ, do not assume preservation: examine the new covariance matrix and its eigenstructure. A correlation-based analysis is designed to remove positive changes of scale from individual variables, but that is a different PCA input matrix; it does not make covariance PCA itself invariant to separate changes of units.