What Standardization Changes
PCA depends on the scale of the variables: the first component is chosen to have the greatest variance among unit-length linear combinations. Standardizing each variable changes those variances before the component directions are selected. This can be useful when the original units are arbitrary or incomparable, but it can also give a small-variance variable as much initial influence as a large-variance one, regardless of how noisy either measurement is.
The central diagnostic question is not simply whether variables have different variances. It is whether setting all marginal variances to one is appropriate for the scientific goal and the quality of the measurements. In particular, standardization is not a correction for noise, and it does not distinguish informative variation from technical variation.
Covariance PCA uses \(\Sigma\); PCA on the standardized variables uses \(R\). The first standardized component is a unit-length vector \(a\) maximizing \(\operatorname{Var}(a^\mathsf{T}Z)=a^\mathsf{T}Ra\). Thus, standardization changes the optimization problem, not merely the labels on a plot. The resulting score \(a^\mathsf{T}Z\) is a linear combination of standardized measurements; its coefficients apply after each variable has been divided by its own standard deviation.
Why Unit Variance Can Give Noise More Influence
For any unit vector \(a\), the variance of the standardized score has the decomposition $$ a^\mathsf{T}Ra = \sum_{j=1}^p a_j^2+ 2\sum_{j<k}a_ja_kR_{jk}. $$ The diagonal contribution is \(\sum_j a_j^2=1\). A variable with a tiny original variance therefore does not enter standardized PCA with a tiny diagonal contribution: its standardized variance is one, just like every other variable's. Correlations between variables can then make a group of low-variance measurements especially prominent.
This is not a mathematical defect in PCA. It is a consequence of the chosen representation and objective. It becomes a practical pitfall when small variance reflects limited signal, instrument noise, or a measurement process that should not receive equal weight. The eigenvalues of \(R\) describe variance on the standardized scale, not the amount of variation in the original measurement units.
Worked Example: Correlated Technical Noise Becomes the First Component
Suppose \(X_1\) is a high-variance measurement, while \(X_2\) and \(X_3\) are low-variance technical measurements with positively correlated noise. Let the population covariance matrix be $$ \Sigma= \begin{pmatrix} 100&0&0\\ 0&1&0.8\\ 0&0.8&1 \end{pmatrix}. $$ Assume the technical variation in \(X_2\) and \(X_3\) is not scientifically useful, even though their noise is correlated.
Covariance PCA. The first coordinate is an eigenvector with eigenvalue \(100\). The lower \(2\)-by-\(2\) block has eigenvectors \((1,1)^\mathsf{T}/\sqrt{2}\) and \((1,-1)^\mathsf{T}/\sqrt{2}\), with eigenvalues \(1.8\) and \(0.2\), respectively: multiplying the block by \((1,1)^\mathsf{T}\) gives \((1.8,1.8)^\mathsf{T}\), and multiplying by \((1,-1)^\mathsf{T}\) gives \((0.2,-0.2)^\mathsf{T}\). The covariance eigenvalues are therefore \(100,1.8,0.2\), so the first covariance component points along \(X_1\).
Standardized PCA. The standard deviations are \(10,1,1\), so standardization gives $$ R= \begin{pmatrix} 1&0&0\\ 0&1&0.8\\ 0&0.8&1 \end{pmatrix}. $$ Its eigenvalues are \(1,1.8,0.2\). The largest is \(1.8\), with unit eigenvector \((0,1,1)^\mathsf{T}/\sqrt{2}\). Thus, standardized PCA makes the correlated technical measurements the first component, even though their total original-scale variance in that direction is only \(1.8\), compared with variance \(100\) along \(X_1\).
The computation does not show that covariance PCA is always preferable. It shows why the choice of scale must match the goal: if the correlated variation is unwanted noise, correlation PCA promotes it; if that shared pattern is the target, the same result may be useful.
A Useful Consequence: Standardization Is Not Noise Correction
The identity \(R=D^{-1}\Sigma D^{-1}\) gives a precise account of the transformation. It divides each covariance \(\operatorname{Cov}(X_j,X_k)\) by the product \(\sigma_j\sigma_k\). It does not use information about measurement precision, replicate variability, or the proportion of variance that is signal. Those features are absent from this transformation unless they are encoded in the covariance matrix and assessed separately.
Proof. Since \(Z=D^{-1}(X-\mu)\), covariance transformation gives $$ \operatorname{Cov}(Z) =D^{-1}\operatorname{Cov}(X)D^{-1} =D^{-1}\Sigma D^{-1}. $$ For a fixed vector \(a\), covariance of a linear transformation gives $$ \operatorname{Var}(a^\mathsf{T}Z) =a^\mathsf{T}\operatorname{Cov}(Z)a =a^\mathsf{T}Ra. $$ The \(j\)-th diagonal entry of \(R\) is $$ R_{jj}=\frac{\operatorname{Var}(X_j)}{\sigma_j^2}=1. $$ Summing the diagonal entries yields \(\operatorname{tr}(R)=p\). \(\square\)
This result explains a common interpretive error: a component accounting for a large fraction of standardized total variance is not necessarily capturing a large amount of original-scale variation. The total standardized variance is always \(p\); the original total variance is \(\operatorname{tr}(\Sigma)=\sum_j\sigma_j^2\), which can be very different.
Worked Example: A Low-Variance Variable Carries a Shared Pattern
Now consider two variables with covariance matrix $$ \Sigma= \begin{pmatrix} 100&9\\ 9&1 \end{pmatrix}. $$ This matrix is positive definite: its leading diagonal entry is positive and its determinant is \(100-81=19>0\). The standard deviations are \(10\) and \(1\), so the correlation is \(9/(10\cdot1)=0.9\).
Covariance PCA. The eigenvalues solve $$ \det(\Sigma-\lambda I) =(100-\lambda)(1-\lambda)-81 =\lambda^2-101\lambda+19=0. $$ The largest eigenvalue is \((101+\sqrt{10125})/2\), approximately \(100.812\). If its eigenvector is \((x,y)^\mathsf{T}\), the second row of the eigenvector equation gives \(9x+y=\lambda y\), so $$ \frac{y}{x}=\frac{9}{\lambda-1}\approx 0.090. $$ The first covariance direction is therefore mostly aligned with the high-variance variable.
Standardized PCA. The correlation matrix is $$ R= \begin{pmatrix} 1&0.9\\ 0.9&1 \end{pmatrix}. $$ Its eigenvectors are \((1,1)^\mathsf{T}/\sqrt{2}\) and \((1,-1)^\mathsf{T}/\sqrt{2}\), with eigenvalues \(1.9\) and \(0.1\). Standardized PCA emphasizes the shared positive pattern across the two variables.
If the association is scientifically meaningful and the original units are incomparable, that emphasis may be desirable. If the small-scale variable is an unreliable measurement, giving it equal standardized variance may instead exaggerate its contribution. The calculation alone cannot decide between these interpretations; the measurement context does.
When Standardization Removes a Unique Direction
Standardization can also make different component directions equally good. This occurs, for example, when variables are mutually uncorrelated: their correlation matrix is the identity, regardless of how different their original variances were. In that case every unit vector has the same standardized score variance, so the first standardized component is not uniquely determined.
Worked Example: Independent Variables Become Tied
Let \(X_1\) and \(X_2\) be uncorrelated, with variances \(100\) and \(0.01\). Then $$ \Sigma= \begin{pmatrix} 100&0\\ 0&0.01 \end{pmatrix}, \qquad D= \begin{pmatrix} 10&0\\ 0&0.1 \end{pmatrix}. $$ Covariance PCA has eigenvalues \(100\) and \(0.01\); the first direction is uniquely the \(X_1\) axis.
After standardization, $$ R=D^{-1}\Sigma D^{-1} = \begin{pmatrix} 1&0\\ 0&1 \end{pmatrix}. $$ For any unit vector \(a=(a_1,a_2)^\mathsf{T}\), the standardized score variance is $$ a^\mathsf{T}Ra=a_1^2+a_2^2=1. $$ Thus every direction is a maximizing direction. Standardization has removed the variance distinction that selected \(X_1\) in covariance PCA; it has not revealed a preferred shared structure.
Choosing and Checking a Standardization
Standardization is most defensible when a one-standard-deviation change has comparable meaning across variables, or when units are arbitrary and the goal is to study relative association patterns. It deserves scrutiny when small variances arise from low signal, measurement limits, or highly correlated technical errors. Before interpreting a standardized component, ask what variation was amplified by dividing by each standard deviation.
A practical comparison is to examine covariance and correlation PCA side by side, together with the marginal variances, measurement context, and correlations. This is not a rule that the two analyses must agree. Their disagreement is informative: it identifies conclusions that depend on treating each variable's marginal scale as equally important.
Common Pitfall
Wrong reasoning: “A low-variance variable has little influence, so standardizing it simply puts it on a fair scale and corrects for its noise.” The first clause may be true for covariance PCA, but the conclusion does not follow. Standardization divides by the variable's standard deviation, making its variance one; it does not estimate how much of that variance is noise. A weak or noisy variable can consequently receive substantial influence, especially if its noise is correlated with other standardized variables.
Correction: Treat standardization as a deliberate change in the PCA objective, not as an automatic quality adjustment. Check whether equal marginal variance is scientifically appropriate, and investigate measurement reliability separately when it matters.
Check Your Understanding
Use the definitions and calculations above to assess each situation.
- Why can a low-variance variable have unit variance after standardization without becoming more reliable?
- In the three-variable example, which component is first under covariance PCA, and which is first under standardized PCA?
- What is the trace of the correlation matrix for \(p\) variables with positive marginal variances, and why?
- Why does an identity correlation matrix fail to select a unique first direction?
- Name one setting where standardization could be useful and one where it could be misleading.