Pitfalls of Clustering Principal Component Scores

Why PCA Scores Can Mislead Clustering

Principal component analysis (PCA) is often used before clustering a genomic profile matrix: projecting profiles onto a few components reduces the number of coordinates, and may suppress noise or redundant variation. But PCA chooses directions to capture total variance, not directions that best separate clusters. A component explaining little overall variance can still distinguish groups of interest; conversely, a high-variance component can reflect within-group variation or technical effects rather than group structure.

There is a second choice after projection: how to scale the component scores. Clustering the raw scores uses their original Euclidean geometry. Dividing scores by their standard deviations changes that geometry, giving relatively more weight to lower-variance components. Neither choice is automatically correct. The appropriate representation depends on the scientific target and on what variation should count as similarity.

Definition: Let \(X\) be an \(n \times p\) matrix of centered profiles, with rows for samples and columns for features. For PCA based on the sample covariance matrix, let \(v_j\) be the unit eigenvector for component \(j\), with eigenvalue \(\lambda_j\), ordered so \(\lambda_1 \geq \lambda_2 \geq \cdots \geq 0\). The score of sample \(i\) on component \(j\) is \(z_{ij}=x_i^\mathsf{T}v_j\). The sample variance of that score is \(\lambda_j\). Clustering the first \(q\) raw scores uses the vectors \(z_i^{(q)}=(z_{i1},\ldots,z_{iq})\); clustering standardized scores instead uses \(z_{ij}/\sqrt{\lambda_j}\) for components with \(\lambda_j>0\).

This setup assumes the same centered, preprocessed features are used for all samples and that the PCA directions are orthonormal. If features were standardized before PCA, the PCA geometry is based on those standardized features; it is not the same as PCA on the raw feature covariance matrix. A zero eigenvalue gives a constant score and cannot be divided by its standard deviation. No normality assumption is needed for these geometric statements.

What Projection Does to the Clustering Objective

For a partition \(C_1,\ldots,C_K\), the K-means within-cluster sum of squares is the sum, over clusters, of squared Euclidean distances from each point to its cluster mean. Orthogonal PCA coordinates make the effect of truncation explicit: squared distance is the sum of the squared differences in each component. Retaining only the first \(q\) components simply removes the terms from components \(q+1,\ldots,p\).

W_{\mathrm{full}}(C) = \sum_{k=1}^{K}\sum_{i\in C_k}\sum_{j=1}^{p} (z_{ij}-\bar{z}_{kj})^2 = W_q(C)+ \sum_{k=1}^{K}\sum_{i\in C_k}\sum_{j=q+1}^{p} (z_{ij}-\bar{z}_{kj})^2,

Here \(\bar{z}_{kj}\) is the mean score on component \(j\) among samples in cluster \(C_k\), and \(W_q(C)\) is the objective in the retained raw-score space. Every omitted term is nonnegative, so \(W_q(C)\leq W_{\mathrm{full}}(C)\) for a fixed partition. However, the optimal partition can change when the objective is changed: the partition minimizing \(W_q\) need not minimize \(W_{\mathrm{full}}\). Nor does a small omitted fraction of total variance guarantee that the omitted terms are small for every partition.

Standardizing component scores changes the objective in another way. For a fixed partition and positive eigenvalues, it is

W_{\mathrm{standardized}}(C) = \sum_{k=1}^{K}\sum_{i\in C_k}\sum_{j=1}^{q} \frac{(z_{ij}-\bar{z}_{kj})^2}{\lambda_j}.

Thus, a component with small \(\lambda_j\) receives a large weight relative to one with high variance. This may be useful when equal weighting of component-wise standardized deviations is intended, but it is not a neutral adjustment. It can amplify low-variance noise and make small score differences influential.

Key takeaway: PCA ranks directions by total variance. K-means minimizes within-cluster squared distance in the chosen coordinates. A variance-explained threshold alone does not establish that the resulting cluster partition preserves the structure relevant to the analysis.

Worked Comparisons

Worked Example: Two Partitions in Raw and Standardized Scores

Consider four hypothetical profiles whose scores on two PCA components are \((10,1)\), \((8,-1)\), \((-8,-1)\), and \((-10,1)\). The scores have mean zero on each component. Their cross-product sum is \(10-8+8-10=0\), so their sample covariance is zero. The sums of squared scores are \(328\) and \(4\), giving sample variances \(328/3\approx109.3\) for PC1 and \(4/3\approx1.333\) for PC2. PC1 therefore has much greater variance.

Compare two \(K=2\) partitions. The PC1 sign partition groups the two negative PC1 scores together and the two positive scores together. In each cluster the PC1 values differ by 2, so their squared deviations from their mean sum to 2; across both clusters the PC1 contribution is 4. Each cluster also contains PC2 scores \(1\) and \(-1\), whose squared deviations from their mean sum to 2. The total PC2 contribution is 4. Thus the full two-component objective is \(4+4=8\).

The PC2 sign partition groups the two scores with PC2 equal to 1 and the two with PC2 equal to \(-1\). PC2 has no within-cluster variation. The PC1 values in those groups are \((10,-10)\) and \((8,-8)\), giving squared-deviation sums of \(200\) and \(128\), respectively. Its full raw-score objective is therefore \(200+128=328\). For these points, the raw full-score objective strongly favors the PC1 sign partition.

Now standardize each component by its sample standard deviation. The PC1 sign partition has standardized objective

\frac{4}{328/3}+\frac{4}{4/3} = \frac{3}{82}+3 \approx 3.037.

For the PC2 sign partition, the standardized objective is

\frac{328}{328/3}+\frac{0}{4/3}=3.

The standardized objective slightly favors the PC2 sign partition, whereas the raw-score objective favors the PC1 sign partition. The calculations do not establish which partition is scientifically meaningful; they show that score scaling can change the criterion and its preferred partition.

Worked Example: A Low-Variance Component Carries the Targeted Separation

Suppose four hypothetical profiles have scores \((-10,-1)\), \((10,-1)\), \((-10,1)\), and \((10,1)\). Assume, for illustration, that an independently defined biological subtype is encoded by the sign of PC2. The score means and sample covariance are zero. The component sample variances are \(400/3\approx133.3\) and \(4/3\approx1.333\).

With \(K=2\), splitting by the sign of PC1 gives two clusters, each with PC2 values \(-1\) and \(1\). PC1 has no within-cluster variation, and PC2 contributes 2 per cluster, so \(W_{\mathrm{full}}=4\). The same partition is available if only PC1 is retained, with objective 0 in that one-dimensional space; the omitted PC2 contribution restores the full-space objective of 4.

Splitting instead by the subtype, which is the sign of PC2, gives two clusters. Each contains PC1 values \(-10\) and \(10\), with mean zero and squared deviations summing to \(100+100=200\). Across the two clusters, the PC1 contribution is 400 and PC2 contributes zero, so \(W_{\mathrm{full}}=400\). Raw K-means strongly prefers the PC1 split under its squared-distance objective, despite the stipulated subtype labels following PC2.

This example does not imply that the subtype partition is the best solution to raw K-means. It demonstrates that a low-variance component can encode a target of interest that the raw Euclidean objective does not prioritize. Retaining PC1 alone makes that distinction invisible; including PC2 does not force K-means to select it.

Worked Example: Whitening Reverses a Distance Comparison

Suppose two retained components have sample variances \(\lambda_1=16\) and \(\lambda_2=1\). Consider three hypothetical score points \(A=(0,0)\), \(B=(4,1)\), and \(C=(1,2)\). In raw scores, the squared distances from \(A\) are

d^2(A,B)=4^2+1^2=17,\qquad d^2(A,C)=1^2+2^2=5.

So \(C\) is closer to \(A\) than \(B\) is. After standardizing scores by their component standard deviations, the squared distances are

d^2_{\mathrm{standardized}}(A,B)=\frac{4^2}{16}+\frac{1^2}{1}=2,\qquad d^2_{\mathrm{standardized}}(A,C)=\frac{1^2}{16}+\frac{2^2}{1}=4.0625.

Now \(B\) is closer to \(A\). The arithmetic shows a reversal, not an error: standardization reduces the influence of PC1 and makes PC2 count more. A distance-based algorithm can consequently produce a different partition after this transformation.

Diagnostics, Alternatives, and Limitations

Treat the number of components and score scaling as clustering choices, not automatic consequences of PCA. A scree plot or cumulative variance percentage describes total variation retained, but does not certify that a partition is preserved. Useful checks include comparing partitions across a reasonable range of retained dimensions and comparing raw with deliberately chosen rescaled scores. When a plausible low-variance component separates samples, inspect its loadings and scores in context rather than discarding it solely because its eigenvalue is small. Conversely, check whether a component with a large eigenvalue is dominated by a technical gradient or another variation that should not define similarity.

The introductory material on internal and external cluster validation provides ways to assess partitions; here, the key diagnostic is whether conclusions depend on the PCA representation. If labels or an external criterion are available, compare them with partitions without treating agreement as proof of a causal or biological mechanism. Without labels, inspect cluster profiles in the original features and assess whether the groups are interpretable and stable to reasonable representation choices. A visually separated plot of the first two components is not, by itself, evidence that the clusters are well separated in all features.

Clustering original centered profiles with Euclidean distance preserves all PCA directions and their original variance contributions. Clustering standardized original features instead changes feature weights before or during distance calculation. Clustering raw PCA scores after retaining all nonzero components is geometrically equivalent to clustering the same centered profiles in the original Euclidean space, because PCA is an orthogonal rotation. By contrast, truncating components discards directions, and whitening all components weights directions inversely by variance. Hierarchical clustering and Gaussian mixture models also depend on representation: their distances, linkage criteria, or covariance structures do not make omitted information reappear automatically.

Representation	What it emphasizes	Important caution
All raw PCA scores	Original Euclidean geometry of centered profiles	High-variance directions contribute more to squared distance
First \(q\) raw scores	Distances after projection onto the retained subspace	Omitted directions cannot affect the partition
Standardized retained scores	Equal marginal variance for retained components	Low-variance components receive larger relative weight

No representation is universally best. PCA directions are global linear summaries: a cluster distinction that is nonlinear or localized may not align with any leading component. Standardization can overemphasize components dominated by measurement noise, while truncation can remove weak but structured signals. Preprocessing choices such as feature scaling affect both the PCA directions and the resulting score variances. For these reasons, report the features and scaling used for PCA, the number of components retained, and whether score standardization was applied.

Common Pitfalls

“The first components explain most variance, so their clusters must be best.” Explained variance concerns reconstruction of total variation, not cluster separation or scientific relevance. Check whether discarded directions contain structure relevant to the objective.
“Standardizing scores only changes units.” It changes relative distance weights. The worked distance comparison reverses because component-specific division by standard deviations changes the geometry.
“A low-variance component is unimportant.” Low variance means small total spread, not absence of a meaningful pattern. A small but coherent group difference can occur in such a direction.
“A partition found in a PC plot is the partition in the original data.” A plot usually shows a projection. Omitted coordinates may alter pairwise distances and cluster objectives.

Practical rule: State what variation the clustering is intended to respect, then justify the PCA representation against that goal. Compare reasonable retention and scaling choices, and interpret any resulting partition in the original feature context.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

For a fixed partition, why can the raw-score objective using \(q\) components not exceed the full-score objective?
A discarded component has small total variance but separates a prespecified biological subgroup. Does the variance ranking alone justify dropping it?
What happens to a component's relative weight in Euclidean distance when its scores are divided by their standard deviation?
When is clustering all nonzero raw PCA scores geometrically equivalent to clustering centered profiles?

Solutions. (1) The full objective is the retained objective plus the sum of nonnegative omitted within-cluster squared deviations, so it cannot be smaller. (2) No. Variance ranking measures total spread, not relevance to the subgroup; evaluate whether the signal is reliable and whether the subgroup separation is the intended target. (3) Its squared deviations are divided by its variance, so low-variance components receive greater relative weight than under raw-score Euclidean distance. (4) When PCA is an orthogonal rotation of the same centered profiles and all nonzero components are retained; orthogonal transformations preserve Euclidean distances.