Why Measurement Units Matter
PCA finds directions of variation from the numerical values in a data matrix. Those numbers depend on how each variable is measured. A distance recorded in meters has different numerical values from the same distance recorded in centimeters; changing units can therefore change a covariance matrix and, in turn, the principal directions calculated from it. This is not a computational error: covariance-based PCA treats the numerical scales as meaningful.
Correlation-based PCA takes a different approach. It first expresses each variable in standard-deviation units, so a change from dollars to thousands of dollars, for example, does not by itself change the standardized data. The choice between these approaches should reflect the analysis: use covariance-based PCA when the original scales are meaningful and comparable, and consider correlation-based PCA when differences in units or numerical scales would otherwise dominate.
This tutorial focuses on the effect of units on those two matrices and on their principal directions. It is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
How a Unit Conversion Changes the Covariance Matrix
Suppose a vector of measurements \(x\) is transformed variable by variable according to \(x^*=Dx+b\), where \(D\) is a diagonal matrix of scale factors and \(b\) is a vector of additive constants. For example, converting meters to centimeters uses a scale factor of 100. If \(S\) is the sample covariance matrix before the conversion, then the covariance matrix afterward is
The additive constants disappear from the covariance calculation because covariance is calculated from deviations from the mean. The scale factors do not disappear. In particular, a variable's variance is multiplied by the square of its scale factor, while the covariance between two variables is multiplied by the product of their scale factors.
If all variables are multiplied by the same positive factor \(c\), then \(S^*=c^2S\). All eigenvalues are multiplied by \(c^2\), but the eigenvectors are unchanged. If variables are rescaled by different factors, there is generally no such preservation: the relative variances and covariances change, and so may the principal directions.
Why Correlation-Based PCA Is Invariant to Positive Unit Changes
Let \(s_j\) be the sample standard deviation of variable \(X_j\). Its standardized value is \(Z_j=(X_j-\bar X_j)/s_j\). Now convert \(X_j\) to \(X_j^*=a_jX_j+b_j\), with \(a_j>0\). Its converted mean is \(a_j\bar X_j+b_j\), and its converted standard deviation is \(a_js_j\). Therefore,
So standardizing after the conversion gives exactly the same standardized observations as standardizing before it. Their correlation matrix, and thus the PCA computed from that matrix, is unchanged. This justification assumes positive scale factors, as in ordinary unit conversions. A negative scale factor reverses the sign of a variable and can reverse the corresponding coordinate signs of component directions.
Centering alone is not enough to remove scale effects. Subtracting the mean eliminates an additive shift, but leaves a multiplication by the unit-conversion factor in every centered observation. Standardization removes both the shift and a positive change of scale.
Worked Examples
Worked Example: Annual Dollars and a Dimensionless Rate
In a hypothetical portfolio study, \(X\) is an annual amount measured in dollars and \(Y\) is a dimensionless rate. Their sample covariance matrix is
Compare the first principal direction from covariance-based PCA with the direction from correlation-based PCA. State which calculation is most affected by the difference in variable scales.
Method and covariance-based calculation. The standard deviations are \(\sqrt{40000}=200\) dollars and \(\sqrt{1}=1\). The covariance is 120, so the correlation is \(120/(200\cdot1)=0.6\). The covariance matrix is valid: its determinant is \(40000-120^2=25600>0\).
The eigenvalues of \(S\) are
The larger eigenvalue is approximately \(40000.3600\). If the first coordinate of its eigenvector is set to 1, the first row of the eigenvector equation gives \(40000+120r=\lambda_1\), where \(r\) is the second coordinate. Hence \(r=(40000.3600-40000)/120\), approximately \(0.0030\). A corresponding direction is approximately \((1,0.0030)\): it is very close to the dollar-variable axis.
Correlation-based calculation. The correlation matrix is
Its eigenvalues are \(1.6\) and \(0.4\), with a first principal direction proportional to \((1,1)\). The standardized variables therefore contribute equally to this direction, unlike the covariance-based result. The covariance calculation is strongly affected by the dollar variable's much larger numerical variance.
Check and common mistake. The correlation is \(0.6\), not 120: 120 is the covariance in the stated units. A common mistake is to treat the covariance-based direction as evidence that the dollar variable is inherently more important, without first asking whether its scale is appropriate for the analysis.
Worked Example: Converting Dollars to Thousands of Dollars
Use the same hypothetical variables and covariance matrix as in the preceding calculation. Convert \(X\) from dollars to thousands of dollars, so \(X^*=X/1000\). Find the new covariance matrix and approximate first principal direction. Compare it with the original direction and with correlation-based PCA.
Method. The scale matrix is \(D=\operatorname{diag}(0.001,1)\), so \(S^*=DSD\). Thus the variance of \(X\) is multiplied by \(0.001^2\), and its covariance with \(Y\) by \(0.001\):
The determinant is \(0.04-0.12^2=0.0256>0\). The eigenvalues are
The larger eigenvalue is approximately \(1.0148\). Set the first coordinate of its eigenvector to 1. The first row gives \(0.04+0.12r=\lambda_1\), so \(r=(1.0148-0.04)/0.12\), approximately \(8.1231\). The direction in the converted coordinates is therefore proportional to \((1,8.1231)\), much closer to the \(Y\)-axis than the original covariance-based direction \((1,0.0030)\).
The correlation remains \(0.12/(\sqrt{0.04}\sqrt{1})=0.12/(0.2\cdot1)=0.6\). Consequently, correlation-based PCA still gives the direction \((1,1)\). The numerical covariance-based direction changes because the units change; the correlation-based direction does not.
Check and common mistake. The converted covariance is \(0.12\), not \(1.2\): changing dollars to thousands divides \(X\) by 1000, so its covariance with \(Y\) is divided by 1000. The variance of \(X\) is divided by \(1000^2\). Applying the same divisor to both entries is a unit-conversion error.
Worked Example: Celsius and Fahrenheit
A hypothetical data set has temperature \(T\), measured in degrees Celsius, and another measurement \(W\). Its sample covariance matrix is
Convert temperature to degrees Fahrenheit, \(T^*=1.8T+32\). Compare the covariance-based first direction before and after conversion, and verify what happens to correlation-based PCA.
Method. The additive 32 is removed by centering. The scale factor is 1.8, so the temperature variance is multiplied by \(1.8^2=3.24\), and its covariance with \(W\) is multiplied by 1.8. The converted covariance matrix is
Before conversion, the larger eigenvalue is \((41+\sqrt{481})/2\), approximately \(31.4659\). Setting the temperature coordinate to 1, the corresponding second coordinate is \((31.4659-25)/10\), approximately \(0.6466\). After conversion, the larger eigenvalue is \((97+\sqrt{5521})/2\), approximately \(85.6517\). The corresponding second coordinate, with the Fahrenheit coordinate set to 1, is \((85.6517-81)/18\), approximately \(0.2584\). Thus the covariance-based direction changes when temperature is expressed in Fahrenheit.
The original correlation is \(10/(5\cdot4)=0.5\). After conversion, the standard deviations are 9 and 4, so the correlation is \(18/(9\cdot4)=0.5\) again. The correlation matrix is unchanged, and its first direction is proportional to \((1,1)\) both before and after conversion.
Check and common mistake. Adding 32 does not change centered covariance, but multiplying by 1.8 does. Treating the conversion as only an additive shift would incorrectly leave the covariance matrix unchanged.
Choosing a PCA Scale
The choice between covariance-based and correlation-based PCA is a modeling decision, not a rule that correlation-based PCA is always preferable. Covariance-based PCA preserves the original variances and covariances. It is appropriate when the variables are on comparable scales and their absolute variation is relevant to the question. Correlation-based PCA gives each variable unit variance before finding directions, preventing a variable from dominating solely because its measurement units produce larger numbers.
Standardization also has a consequence: a low-variability variable receives the same initial variance as a high-variability variable. If the low variability mostly reflects measurement noise, correlation-based PCA may give that noise more influence than intended. The scale choice should therefore be justified in the context of the variables and the purpose of the analysis.
Identify whether the variables are measured on substantially different scales, and whether those scales are meaningful for the analysis.
Use the covariance matrix if original variances are intended to affect the analysis; use the correlation matrix if variables should first be placed on a common standard-deviation scale.
When applying a fitted PCA to new observations, use the same centering and scaling decisions used to fit the components.
In R, prcomp with centering and scale.=FALSE performs PCA on centered variables without standardizing them. With scale.=TRUE, it standardizes variables before computing the components, corresponding to correlation-based PCA when the variables have nonzero variance. A unit conversion made before fitting can therefore change the result in the first case but not in the second, apart from possible sign conventions for component vectors.
Common Traps
- Assuming centering removes unit effects. Centering removes additive constants. It does not remove multiplicative scale factors.
- Rescaling a variance and a covariance the same way. If \(X\) is multiplied by \(a\), its variance is multiplied by \(a^2\), while its covariance with an unchanged variable is multiplied by \(a\).
- Comparing eigenvectors without checking their coordinate units. A numerical direction after a unit change is expressed in the new coordinate system. A changed vector does not mean the data themselves changed.
- Assuming correlation-based PCA answers every scale question. Standardization removes unit dependence, but it also changes the influence of variables with different levels of variability.
- Treating a unit conversion as a change in information. Converting the same observations between compatible units does not add or remove information; it changes the numerical representation. Whether the PCA result changes depends on the scale convention used.
Check Your Understanding
For each problem, state the relevant transformation before deciding whether the principal directions change.
- A variable \(A\) is converted from meters to centimeters by multiplying it by 100, while \(B\) is unchanged. What happens to \(\operatorname{Var}(A)\) and \(\operatorname{Cov}(A,B)\)? (Answer: \(\operatorname{Var}(A)\) is multiplied by 10,000; \(\operatorname{Cov}(A,B)\) is multiplied by 100.)
- A two-variable covariance matrix is \(S=\begin{pmatrix}4&3\\3&9\end{pmatrix}\). Find the correlation and the first direction from correlation-based PCA. (Answer: The correlation is \(3/(2\cdot3)=0.5\); the correlation matrix has first direction proportional to \((1,1)\).)
- A temperature variable \(T\) is changed to \(T^*=2T+10\), while \(W\) is unchanged. If \(\operatorname{Var}(T)=5\) and \(\operatorname{Cov}(T,W)=-3\), find the transformed variance and covariance. (Answer: \(\operatorname{Var}(T^*)=4(5)=20\); \(\operatorname{Cov}(T^*,W)=2(-3)=-6\).)
- Every variable in a data set is multiplied by 4. What happens to the covariance matrix, its eigenvalues, and its eigenvectors? (Answer: The covariance matrix and all eigenvalues are multiplied by 16; the eigenvectors, and thus the principal directions, are unchanged.)
- Two variables have covariance matrix \(\begin{pmatrix}16&6\\6&9\end{pmatrix}\). Convert the first variable to units that divide its values by 4. Give the new covariance matrix and say whether correlation-based PCA changes. (Answer: The new matrix is \(\begin{pmatrix}1&1.5\\1.5&9\end{pmatrix}\); correlation-based PCA is unchanged because the conversion is a positive rescaling.)