Why Measurement Units Matter in PCA
A variable measured in dollars has a different numerical scale from the same variable measured in thousands of dollars. That change of units does not alter the underlying observations, but it can alter a principal component analysis based on the covariance matrix. This matters whenever the variables have different units or when someone changes how a variable is recorded.
The central distinction is between covariance-based PCA, which uses the variables’ original scales, and correlation-based PCA, which uses their correlations. Multiplying a variable by a positive conversion factor changes its variance and its covariances, so it can change covariance-based principal directions. Correlations, by contrast, are unchanged by positive rescaling, so the correlation matrix and its PCA directions remain unchanged.
A unit conversion may also add a constant, as when converting temperatures between scales. Because PCA is applied to centered variables, adding a constant to every observation of a variable does not affect its covariance or correlation. The multiplier is what changes the covariance matrix; for ordinary unit conversions the multiplier is positive.
How Rescaling Changes Covariances
For any pair of variables \(X_j\) and \(X_k\), multiplying them by \(d_j\) and \(d_k\) gives \(\operatorname{Cov}(d_jX_j,d_kX_k)=d_jd_k\operatorname{Cov}(X_j,X_k)\). In particular, a variance is multiplied by the square of its variable’s conversion factor. It is not multiplied by the factor only once.
This matrix identity follows from the definition of covariance. With means \(\mu_X=E[X]\) and \(\mu_Z=E[Z]=D\mu_X\), the centered vector is \(Z-\mu_Z=D(X-\mu_X)\). Therefore \[ E[(Z-\mu_Z)(Z-\mu_Z)^\mathsf{T}] =E[D(X-\mu_X)(X-\mu_X)^\mathsf{T}D] =DSD, \] where the deterministic matrix \(D\) can be taken outside the expectation. Since \(D\) is diagonal and symmetric, this gives the stated covariance matrix.
PCA based on covariance finds eigenvectors of \(S\). After changing units, it finds eigenvectors of \(DSD\). In general, multiplying rows and columns of a matrix by different scale factors does not preserve its eigenvectors. The directions may therefore change, and the amount of variance assigned to each direction may also change. In special cases the directions can remain the same, but that should not be assumed.
Why Correlations Are Unchanged
The correlation between two variables is their covariance divided by the product of their standard deviations. Positive rescaling multiplies the covariance by \(d_jd_k\) and the two standard deviations by \(d_j\) and \(d_k\). The factors cancel.
Thus, the correlation matrix before and after a positive unit conversion is exactly the same. Correlation-based PCA consequently has the same eigenvalues and directions in standardized-variable coordinates, apart from the arbitrary choice of sign for an eigenvector. For an observation, its standardized value is also unchanged: subtracting its mean and dividing by its standard deviation cancels the positive conversion factor.
This does not mean that every numerical expression involving raw observations stays the same. A component score calculated from unstandardized values and covariance-based loadings is tied to the units used to fit that PCA. The invariance claim concerns correlations and PCA carried out on the correlation matrix, equivalently on standardized variables.
Worked Example: Rescaling One Variable Changes a Covariance PCA Direction
A hypothetical portfolio study uses two variables whose sample covariance matrix is \(S=\begin{pmatrix}16&4\\4&4\end{pmatrix}\). The first variable is recorded in a unit that is replaced by a unit twice as large numerically, so each recorded value is multiplied by 2. Find the transformed covariance matrix, compare correlations, and determine whether the leading covariance-PCA direction changes.
Method. Use \(D=\begin{pmatrix}2&0\\0&1\end{pmatrix}\) and calculate \(DSD\). The common mistake avoided here is multiplying the off-diagonal covariance by \(2^2\); it is multiplied by 2 because only one of the two variables is rescaled.
First, the original standard deviations are \(\sqrt{16}=4\) and \(\sqrt{4}=2\). The original correlation is \[ \frac{4}{4\cdot2}=0.5000. \] For the transformed matrix, the first variance is \(2^2(16)=64\), the covariance is \(2(4)=8\), and the second variance remains 4. Hence \[ DSD=\begin{pmatrix}2&0\\0&1\end{pmatrix} \begin{pmatrix}16&4\\4&4\end{pmatrix} \begin{pmatrix}2&0\\0&1\end{pmatrix} =\begin{pmatrix}64&8\\8&4\end{pmatrix}. \]
The transformed standard deviations are 8 and 2, so the transformed correlation is \(8/(8\cdot2)=0.5000\), as required.
To compare leading directions, the eigenvalues of the original matrix solve \(\lambda^2-20\lambda+48=0\), giving \(\lambda=10\pm2\sqrt{13}\). The largest is approximately \(17.2111\). For its eigenvector \((1,r)^\mathsf{T}\), the first row of the eigenvalue equation gives \(16+4r=\lambda\), so \(r=(\lambda-16)/4\approx0.3028\). Normalizing gives a leading direction approximately \((0.9571,0.2898)^\mathsf{T}\).
For the transformed matrix, the eigenvalues solve \(\lambda^2-68\lambda+192=0\), giving \(\lambda=(68\pm\sqrt{3856})/2\). The largest is approximately \(65.0484\). Its eigenvector ratio is \(r=(\lambda-64)/8\approx0.1310\), giving the normalized direction approximately \((0.9915,0.1299)^\mathsf{T}\). The direction has changed. The method and the final values are consistent: the covariance is 8, not 16, and the correlation is 0.5000 both before and after conversion.
Correlation-Based PCA Preserves Its Directions
The following example shows the same conversion viewed through correlation-based PCA. The key is to recompute the standard deviations along with the covariance; changing only one part of the correlation calculation leads to an incorrect result.
Worked Example: Comparing Correlation PCA Before and After Conversion
Suppose two hypothetical risk measurements have covariance matrix \(S=\begin{pmatrix}4&3\\3&9\end{pmatrix}\). The first measurement is converted by multiplying all its values by 10. Find the correlation matrix before and after conversion, and give the leading correlation-PCA direction.
Method. Compute the original correlations, transform the covariance matrix using \(D=\operatorname{diag}(10,1)\), and then calculate the new correlations. The trap avoided is to infer that a larger covariance necessarily means a larger correlation.
The original standard deviations are 2 and 3, so the off-diagonal correlation is \(3/(2\cdot3)=0.5000\). The original correlation matrix is therefore \[ R=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}. \]
Under the conversion, the first variance becomes \(10^2(4)=400\), the covariance becomes \(10(3)=30\), and the second variance remains 9. Thus the transformed covariance matrix is \[ DSD=\begin{pmatrix}400&30\\30&9\end{pmatrix}. \] The transformed standard deviations are 20 and 3. The transformed correlation is \(30/(20\cdot3)=0.5000\), so the correlation matrix is still \(R\).
The eigenvectors of \(R\) are proportional to \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\). Their eigenvalues are 1.5 and 0.5, respectively, since multiplying \(R\) by those vectors gives 1.5 and 0.5 times the corresponding vector. The leading correlation-PCA direction is therefore \((1,1)^\mathsf{T}/\sqrt{2}\), both before and after the unit conversion. The standardized value of the first observation is unchanged because both its deviation from the mean and its standard deviation are multiplied by 10.
Several Variables and Several Conversion Factors
With more than two variables, each diagonal entry of \(D\) has its own conversion factor. Each variance receives the square of its factor, while each covariance receives the product of the two factors involved. Correlations still remain unchanged pair by pair.
Worked Example: Transforming a Three-Variable Covariance Matrix
A hypothetical study records three risk indicators with sample covariance matrix \[ S=\begin{pmatrix}100&20&10\\20&25&5\\10&5&4\end{pmatrix}. \] The recorded values are converted using factors \(0.01\), \(2\), and \(0.5\), respectively. Find the new covariance matrix and verify all three correlations.
Method. Apply \(DSD\) with \(D=\operatorname{diag}(0.01,2,0.5)\). The trap avoided is applying one factor to a covariance without accounting for the conversion of both variables in that pair.
The new variances are \(100(0.01)^2=0.01\), \(25(2)^2=100\), and \(4(0.5)^2=1\). The new covariances are \(20(0.01)(2)=0.4\), \(10(0.01)(0.5)=0.05\), and \(5(2)(0.5)=5\). Therefore, \[ DSD=\begin{pmatrix}0.01&0.4&0.05\\0.4&100&5\\0.05&5&1\end{pmatrix}. \]
Before conversion, the standard deviations are 10, 5, and 2. The correlations are \(20/(10\cdot5)=0.4000\), \(10/(10\cdot2)=0.5000\), and \(5/(5\cdot2)=0.5000\).
After conversion, the standard deviations are 0.1, 10, and 1. The correlations are \(0.4/(0.1\cdot10)=0.4000\), \(0.05/(0.1\cdot1)=0.5000\), and \(5/(10\cdot1)=0.5000\). Each correlation is unchanged, so the entire correlation matrix is unchanged. In contrast, the covariance matrix entries have changed substantially, so its PCA need not have the same directions.
When the Unit Effect Matters
Covariance-based PCA is appropriate when the original units and relative scales are meaningful to the analysis. Its sensitivity to scale is not an arithmetic flaw; it is a property of optimizing variance in those units. But it creates a comparability issue: a variable can dominate the covariance matrix partly because it is measured on a numerically large scale. Changing dollars to cents, for example, can greatly increase its variance in numerical terms without adding information.
Correlation-based PCA removes this direct dependence on positive measurement units by expressing each variable in standard-deviation units. It is useful when the variables have different units or when equal treatment of standardized variation is intended. The choice between covariance and correlation PCA should therefore be based on the analysis goal, not on which matrix produces a more convenient answer.
Common Mistakes
- Scaling a covariance by a squared factor when only one variable changes. If \(X_j\) is multiplied by \(c\) and \(X_k\) is not, their covariance is multiplied by \(c\), not \(c^2\). The variance of \(X_j\) is multiplied by \(c^2\).
- Assuming unit changes leave covariance-PCA directions unchanged. The transformed matrix is \(DSD\), whose eigenvectors can differ from those of \(S\).
- Assuming a covariance change implies a correlation change. Standard deviations change at the same time; under positive rescaling their factors cancel in the correlation.
- Claiming correlation PCA is invariant for every transformation. The result here concerns positive rescaling. A negative multiplier reverses the sign of correlations involving that variable, and a nonlinear transformation can change correlations as well.
- Comparing component coefficients without stating the coordinate system. Correlation-PCA directions describe standardized variables; covariance-PCA directions are based on the original scale.
Check Your Understanding
For each problem, state the transformed covariance entries and use the appropriate standard deviations when checking correlations.
- A covariance matrix is \(\begin{pmatrix}9&3\\3&4\end{pmatrix}\). The first variable is multiplied by 2. Give the new covariance matrix and the correlation before and after conversion. (Answer: \(\begin{pmatrix}36&6\\6&4\end{pmatrix}\); correlation \(0.5000\) before and after.)
- Two variables have variances 25 and 16 and covariance \(-10\). The second variable is multiplied by \(0.5\). Give the transformed variances and covariance, and the correlation before and after. (Answer: variances 25 and 4, covariance \(-5\); correlation \(-0.5000\) before and after.)
- A covariance matrix is \(\begin{pmatrix}1&0.6\\0.6&4\end{pmatrix}\). The first variable is multiplied by 3 and the second by \(0.5\). Give the transformed covariance matrix and correlation. (Answer: \(\begin{pmatrix}9&0.9\\0.9&1\end{pmatrix}\); correlation \(0.3000\), unchanged.)
- A correlation matrix is \(\begin{pmatrix}1&-0.4\\-0.4&1\end{pmatrix}\). The first variable is multiplied by 100. Give the leading eigenvalue and a unit-length leading eigenvector before and after conversion. (Answer: eigenvalue \(1.4\) and direction \((1,-1)^\mathsf{T}/\sqrt{2}\) before and after, up to an overall sign.)
- Suppose a variable \(X\) is converted to \(Z=5X+12\), where 12 is a constant offset. How does its variance change, and what happens to its correlation with another variable \(Y\)? (Answer: \(\operatorname{Var}(Z)=25\operatorname{Var}(X)\); if \(\operatorname{SD}(X)>0\) and \(\operatorname{SD}(Y)>0\), \(\operatorname{Corr}(Z,Y)=\operatorname{Corr}(X,Y)\).)