Why preprocessing changes PCA
A profile matrix can contain measurements on very different scales. In genomic data, columns might represent genes with different expression variability, or features recorded in different units. PCA finds directions of high variation, so the scale assigned to each column affects which directions count as high variation. Centering and scaling are therefore choices about the data PCA analyzes, not cosmetic adjustments made after the analysis.
Let \(X\) be an \(n \times p\) matrix, with samples in rows and measured features in columns. Write \(x_{ij}\) for the value of feature \(j\) in sample \(i\). Centering subtracts each column's sample mean; variance scaling then divides each centered column by its sample standard deviation. PCA on the centered matrix uses its sample covariance matrix. PCA on the centered and standardized matrix uses the sample correlation matrix.
The covariance matrix of the centered data is \(S=C^\mathsf{T}C/(n-1)\), where \(C\) is the matrix of centered values. Its diagonal entries are the feature sample variances, and its off-diagonal entries are sample covariances. Let \(D\) be the diagonal matrix with entries \(s_1,\ldots,s_p\). When every \(s_j\) is positive, the standardized matrix is \(Z=CD^{-1}\), and its covariance matrix is
where \(R\) is the sample correlation matrix. Its diagonal entries are 1, and each off-diagonal entry is the sample correlation between two features. Thus, PCA of centered data is often called covariance-based PCA, while PCA of standardized data is correlation-based PCA.
What centering and scaling do
Centering removes each feature's observed mean before variation is summarized. This matters because PCA based on covariance is about variation around the feature means, not variation around zero. If the data are not centered, the matrix \(X^\mathsf{T}X/n\) includes both variation around the means and a contribution from the means themselves. A feature's baseline level can then influence directions that would otherwise describe co-variation.
Scaling changes the relative weight assigned to columns. Covariance-based PCA gives greater influence, in general, to features with larger numerical variance. Standardization expresses each column in units of its own sample standard deviation, so each contributes variance 1. This can be sensible when scales differ for unimportant measurement reasons, but it can be unhelpful when large variance is meaningful or when low-variance columns are dominated by noise.
Worked Example: Centering Removes Baseline Differences
Suppose three hypothetical samples have values \(1,2,3\) for feature A and \(101,102,103\) for feature B. Both features rise together, but B has a much larger baseline. Find the centered columns and explain what would happen without centering.
Means and centered values. The means are \(\bar{x}_A=(1+2+3)/3=2\) and \(\bar{x}_B=(101+102+103)/3=102\). Subtracting these gives centered columns \((-1,0,1)\) for both features.
Covariance after centering. Each centered column has sample variance \([(-1)^2+0^2+1^2]/(3-1)=1\), and their sample covariance is also \(1\). Thus the covariance matrix is \(\begin{pmatrix}1&1\\1&1\end{pmatrix}\). The two centered features show identical variation.
Why not omit centering? The uncentered cross-product matrix divided by \(n\) is \(\begin{pmatrix}14/3&614/3\\614/3&31214/3\end{pmatrix}\). It reflects the large mean of feature B as well as the changes across samples. PCA of this matrix would not be analyzing variation around the observed feature means. This illustrates why ordinary covariance-based PCA centers columns first.
Calculating standardized values
Standardization uses the sample standard deviation, with denominator \(n-1\), to put a feature's centered observations on a common scale. The resulting values are dimensionless. A value of \(1\) means the observation is one sample standard deviation above that feature's sample mean; it does not mean that the original feature was measured in the same units as another feature.
Worked Example: Standardizing Two Features
For three hypothetical samples, feature A has values \(2,4,6\), and feature B has values \(10,20,30\). Calculate their sample means, standard deviations, and standardized columns.
Feature A. Its mean is \(\bar{x}_A=(2+4+6)/3=4\). Its sample standard deviation is \(s_A=\sqrt{[(2-4)^2+(4-4)^2+(6-4)^2]/2}=\sqrt{4}=2\). The centered values are \((-2,0,2)\), so dividing by 2 gives standardized values \((-1,0,1)\).
Feature B. Its mean is \(\bar{x}_B=(10+20+30)/3=20\). Its sample standard deviation is \(s_B=\sqrt{[(10-20)^2+(20-20)^2+(30-20)^2]/2}=\sqrt{100}=10\). The centered values are \((-10,0,10)\), so dividing by 10 also gives \((-1,0,1)\).
The standardized columns are identical, so their correlation is 1. In this particular data set, the columns differ only by a multiplicative factor. Standardization removes that scale difference, while centering removes their different means.
A profile matrix where the first component changes
The next example shows that scaling can change the leading principal direction, not merely the numerical units of a fixed direction. For a centered matrix, a unit vector \(a\) defines a linear combination of features. Its sample variance is \(a^\mathsf{T}Sa\). The first principal direction maximizes this variance subject to \(a^\mathsf{T}a=1\); equivalently, it is a unit eigenvector of \(S\) for its largest eigenvalue. Standardized PCA carries out this comparison using \(R\) instead.
Worked Example: Covariance PCA and Correlation PCA Differ
Consider four hypothetical samples and two centered features, A and B. Their columns are \(x=(-1,-1,1,1)\) and \(y=(-10(1+\sqrt{3})/2,\;10(\sqrt{3}-1)/2,\;10(1-\sqrt{3})/2,\;10(1+\sqrt{3})/2)\), approximately \((-13.660,3.660,-3.660,13.660)\). Both columns sum to zero. Compare the first directions from covariance-based and standardized PCA.
Covariances and standard deviations. The squared length of \(x\) is 4. The values of \(y\) are \(10q\), where \(q\) is the combination of two orthogonal, equal-length centered patterns with coefficients \(1/2\) and \(\sqrt{3}/2\). Thus \(y^\mathsf{T}y=400\), and \(x^\mathsf{T}y=20\). With \(n-1=3\), the sample covariance matrix is
The sample standard deviations are \(s_A=\sqrt{4/3}\) and \(s_B=\sqrt{400/3}\). The correlation is \((20/3)/\{ \sqrt{4/3}\sqrt{400/3}\}=1/2\), so the correlation matrix is \(\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix}\).
First direction without scaling. The eigenvalues of \(S\) are \(4/3\) times the eigenvalues of \(\begin{pmatrix}1&5\\5&100\end{pmatrix}\). The larger eigenvalue of that inner matrix is \((101+\sqrt{9901})/2\), approximately \(100.252\), so the largest eigenvalue of \(S\) is approximately \(133.669\). From the first row of the eigenvector equation, the ratio of the A coordinate to the B coordinate is \(5/(100.252-1)\), approximately \(0.0504\). A unit first direction is therefore approximately \((0.0503,0.9987)\): covariance-based PCA is dominated by feature B, which has much greater variance.
First direction after scaling. The correlation matrix has eigenvalues \(1.5\) and \(0.5\). Its leading unit eigenvector is \((1/\sqrt{2},1/\sqrt{2})\), approximately \((0.7071,0.7071)\). After standardization the two features have equal variance, and the first direction weights them equally. The different directions arise from a change in the matrix whose variance is being maximized, not from a computational discrepancy.
Choosing a scaling scheme and checking it
The choice should follow the question and the meaning of the columns. If features are comparable measurements and larger variation is scientifically relevant, centering without scaling may preserve an important signal. If numerical scales differ for reasons unrelated to the intended comparison, standardization can prevent a high-variance column from dominating just because of its units. In genomic analyses, this decision depends on what each column represents and how it was measured; treating every feature as equally variable is a substantive choice.
Before fitting PCA, verify that the intended samples and features are in the matrix, column means are handled as planned, and the chosen scaling has actually been applied. For standardized data, check that each retained column has positive sample variance and that the resulting columns have variance 1, up to rounding. A constant column has \(s_j=0\), so division by \(s_j\) is undefined; remove it or handle it explicitly rather than silently attempting standardization. Very small \(s_j\) can also make scaling unstable by magnifying tiny fluctuations.
Means and standard deviations are sensitive to outliers. An extreme observation can shift a column mean or inflate its standard deviation, changing all its centered or standardized values. Inspect suspicious ranges and plots before interpreting PCA. If the data are transformed or filtered, document the operation and apply the same preprocessing parameters when projecting new samples. In predictive settings, estimate preprocessing parameters from the training data, not from held-out data, to avoid information leakage.
PCA is a descriptive linear calculation: the algebra of centering and scaling does not require normally distributed observations. However, an unstable sample covariance matrix or a small number of unusual samples can make estimated directions unreliable. Interpret patterns as associations in the analyzed profiles, not as evidence that one feature causes changes in another.
| Choice | Matrix analyzed | When it is useful | Main caution |
|---|---|---|---|
| Center columns only | Sample covariance matrix \(S\) | Feature scales are comparable, or variance magnitude is meaningful | High-variance features may dominate |
| Center and divide by sample standard deviation | Sample correlation matrix \(R\) | Equal variance per feature is a useful comparison | Low-variance noise may be amplified |
| Center and use scientifically chosen scale factors | Covariance matrix of the deliberately rescaled columns | Known units or measurement design call for a specific scale | Results depend on the chosen factors and their justification |
Common mistake and limitations
- Wrong reasoning: “PCA is invariant to units, so scaling cannot change the answer.” Correction: multiplying a column by a constant changes its covariance with other columns and its variance. Standardization changes the matrix used for PCA and can rotate the principal directions.
- Wrong reasoning: “Standardizing is always fair because every feature then has variance 1.” Correction: equal variance is an analytical choice, not a guarantee of equal information. A nearly constant noisy feature can be given the same variance as a reliable feature after standardization.
- Wrong reasoning: “The sample mean is zero, so centering and scaling are interchangeable.” Correction: centering subtracts a mean; scaling divides by a standard deviation. Centered covariance PCA and standardized correlation PCA use different matrices unless the feature variances happen to make them equivalent up to a common factor.
No scaling rule can determine whether a feature's variance is signal, nuisance variation, or measurement noise. Nor does standardization remove outliers or correct a poorly chosen sample definition. Report which columns were centered, how they were scaled, and how zero-variance features were handled so that the PCA input is reproducible and its interpretation is clear.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For values \(3,5,7\), what are the sample mean and sample standard deviation, and what are the standardized values?
- Why can an uncentered cross-product matrix reflect feature means as well as variation around those means?
- If two centered features have covariance 6 and sample standard deviations 2 and 3, what is their correlation?
- What happens if a feature has sample standard deviation zero and someone attempts to standardize it?
Solutions. (1) The mean is \((3+5+7)/3=5\). The sample standard deviation is \(\sqrt{[(-2)^2+0^2+2^2]/2}=\sqrt{4}=2\), giving standardized values \((-1,0,1)\). (2) Writing each observation as its centered value plus its column mean shows that the cross-products include terms involving the means; they are not solely cross-products of deviations from the means. (3) The correlation is \(6/(2\cdot3)=1\). (4) Division by zero is undefined. The constant column has no observed variation, so it cannot be standardized; it must be removed or otherwise explicitly handled.