Combining Cluster and PCA Features

Why Combine Cluster and PCA Features?

Cluster membership and principal-component (PC) scores describe different kinds of structure in a predictor set. A cluster indicator identifies a record’s group, which can capture a difference in level between groups. A PC score locates the record along a continuous direction of variation in the predictors. A model with both can therefore represent group shifts and within-group trends at the same time.

That combination is not automatically better. Cluster indicators and component scores may encode overlapping information, and a large set of derived features can make a model unnecessarily complex. The practical questions are whether the combined design matrix contains distinct information, whether its coefficients can be estimated, and whether its predictions improve on data not used to fit the model.

Definition: A combined cluster-and-PCA model is a predictive model whose inputs include one or more indicators for cluster membership and one or more PC scores. For a linear regression model with an intercept, one possible specification is \(Y=\beta_0+\sum_{g=2}^{K}\beta_g I_g+\sum_{j=1}^{r}\gamma_j Z_j+\varepsilon\), where \(K\) is the number of clusters, \(I_g\) indicates membership in cluster \(g\), \(Z_j\) is the \(j\)th PC score, and \(\varepsilon\) is the model error.

With \(K\) clusters, reference coding uses \(K-1\) indicators and an intercept: one cluster is the reference group. The cluster coefficients then represent conditional shifts relative to that group, holding the included PC scores fixed. The PC coefficients describe changes associated with a one-unit change in the corresponding score, holding cluster membership fixed. These are predictive associations within the specified model, not necessarily causal effects.

A PC score is a linear combination of the original predictor values after the chosen centering and scaling. Cluster membership is a categorical label produced by a clustering rule. Thus, PC scores are continuous global summaries, while cluster indicators distinguish groups. Their distinct roles explain why combining them can help, but do not guarantee that they provide independent information.

Specify the Combined Model Carefully

Before fitting, decide which cluster is the reference category and which PC scores to include. If there are \(K\) clusters and \(r\) scores, the model above has \(1+(K-1)+r=K+r\) regression coefficients, counting the intercept. This is a parameter count, not a guarantee that all coefficients are estimable: the columns of the model matrix must also be linearly independent.

Use the same defined cluster and PCA representations for the records to which the fitted model is applied. In particular, a cluster label is a category, not a numerical scale: cluster 3 is not “three times” cluster 1. Likewise, a component’s sign is arbitrary, so a PC coefficient’s direction is meaningful only in relation to the chosen sign convention for its score.

Worked Example: Constructing a Combined Prediction

A hypothetical portfolio model has four risk clusters, with cluster A as the reference. It uses PC1 and PC2. Suppose its fitted linear predictor for a response measured in percentage points is \(\hat{Y}=0.12+0.03I_B+0.05I_C-0.01I_D+0.02Z_1-0.01Z_2\). Find the prediction for a record in cluster C with PC scores \(Z_1=0.8\) and \(Z_2=-0.5\), and count the fitted coefficients.

Method. Substitute the record’s reference-coded indicators and scores. A cluster C record has \(I_B=0\), \(I_C=1\), and \(I_D=0\). Therefore,

\hat{Y}=0.12+0.03(0)+0.05(1)-0.01(0)+0.02(0.8)-0.01(-0.5).

The PC contributions are \(0.02(0.8)=0.016\) and \(-0.01(-0.5)=0.005\). Thus, \(\hat{Y}=0.12+0.05+0.016+0.005=0.191\), or 0.191 percentage points under the stated response scale.

There are \(1+3+2=6\) coefficients: an intercept, three cluster indicators, and two PC coefficients. The common mistake is to include four cluster indicators as well as an intercept. Since the four indicators sum to one for every record, doing so would create exact linear dependence. Reference coding avoids that particular redundancy. The arithmetic checks because the included cluster contribution is 0.05 and the two score contributions sum to 0.021, giving \(0.12+0.05+0.021=0.191\).

Redundancy and Identifiability

A model matrix has full column rank when no column is an exact linear combination of the others. Full column rank is needed for unique ordinary least-squares coefficient estimates. When columns are redundant, fitted values may still be determined, but there can be multiple coefficient vectors that produce those same fitted values. This makes individual coefficients unstable or uninterpretable, and some software will drop a column or report an aliased coefficient.

A common source of exact redundancy is including all cluster indicators with an intercept. Another can occur when the cluster labels are determined by a component score that takes only a small number of values in the data. More generally, using all PCs alongside all original predictors creates exact linear dependence when the PCA transformation is full rank: the scores are an invertible linear transformation of the centered, scaled original predictors. Using only some PCs alongside original predictors need not create exact dependence, but it can still create substantial overlap.

Result: If a nonzero vector \(q\) satisfies \(Xq=0\), where \(X\) is the model matrix, then coefficient vectors \(\beta\) and \(\beta+cq\) give identical fitted values for any scalar \(c\), because \(X(\beta+cq)=X\beta+cXq=X\beta\). The coefficients are not uniquely identified by the data.

This identity explains the issue rather than merely labeling it. Least squares chooses coefficients to minimize the sum of squared residuals. If moving along the direction \(q\) changes no fitted value, it changes no residual and cannot distinguish one coefficient vector from another. A model may therefore predict adequately while its separate cluster and PC coefficients lack a unique interpretation.

Worked Example: Detecting an Exact Redundancy

A small hypothetical training set has a PC score \(Z\), a high-cluster indicator \(H\), and response values as shown. The clustering rule puts records with \(Z=1\) in the high cluster and records with \(Z=-1\) in the low cluster. Compare a regression using an intercept and \(Z\) with one that also includes \(H\).

Record	\(Z\)	\(H\)	\(Y\)
1	-1	0	4
2	-1	0	6
3	1	1	10
4	1	1	12

Step 1: Check the feature relationship. For all four records, \(H=(Z+1)/2\). For \(Z=-1\), this gives \(H=0\); for \(Z=1\), it gives \(H=1\). Thus the high-cluster indicator is an exact linear combination of the intercept column and the PC-score column.

Step 2: Fit the model with the intercept and \(Z\). The mean response at \(Z=-1\) is \((4+6)/2=5\), and at \(Z=1\) it is \((10+12)/2=11\). The line through these two group means has slope \((11-5)/(1-(-1))=3\) and intercept \(5-3(-1)=8\). Its predictions are 5, 5, 11, and 11. The residuals are \(-1,1,-1,1\), so the sum of squared errors is \(1+1+1+1=4\).

Step 3: Add \(H\). The new model is \(\hat{Y}=b_0+b_1Z+b_2H\). Substituting \(H=(Z+1)/2\) gives \(\hat{Y}=(b_0+b_2/2)+(b_1+b_2/2)Z\). Only these two combined quantities affect predictions; the three coefficients cannot be uniquely determined. For example, \((b_0,b_1,b_2)=(8,3,0)\) and \((7,2,2)\) both give predictions 5 at \(Z=-1\) and 11 at \(Z=1\).

The added indicator does not improve the fitted values: the sum of squared errors remains 4. The trap is to interpret a fitted coefficient for \(H\) as a separate cluster effect without checking the feature relationship. Here there is no unique separate effect to interpret. The arithmetic checks because both coefficient choices reduce to the same line, with intercept 8 and slope 3.

Test Whether the Combination Helps Prediction

A combined model can be estimable and still fail to improve prediction. Compare candidate models on records not used to fit them, using the same response scale and the same prediction metric. For a validation set with \(n\) observed outcomes \(y_i\) and predictions \(\hat{y}_i\), the root mean squared error is

\operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}.

RMSE gives greater weight to larger errors because each error is squared. A lower validation RMSE indicates better performance on that set under this metric, but does not establish that the model will always perform better. Compare models on the same records; otherwise, a difference may reflect different validation cases rather than different features.

The most useful comparison is not simply “combined versus not combined.” Consider a baseline with PC scores, a model with cluster indicators, and a model with both, when those are plausible candidates. Check whether the combined model’s extra terms contribute useful predictive structure, whether its design matrix is redundant, and whether the validation improvement is large enough to justify the added complexity. A lower training error alone is not sufficient: adding predictors cannot increase the minimum training sum of squared errors for ordinary least squares, but that fact does not ensure better predictions on new records.

Worked Example: Comparing PC-Only and Combined Predictions

A hypothetical analyst evaluates two fitted models on four validation records. The PC-only model predicts using \(\hat{Y}=110+18Z\). The combined model uses \(\hat{Y}=109+18Z+2I_B\), where \(I_B=1\) for cluster B and 0 otherwise. The validation outcomes, PC scores, and cluster memberships are listed below. Calculate both validation RMSEs and state which model has the lower RMSE.

Record	\(Y\)	\(Z\)	Cluster
1	80	-1.5	A
2	100	-0.5	A
3	120	0.5	B
4	140	1.5	B

PC-only model. Its predictions are \(110+18(-1.5)=83\), \(110+18(-0.5)=101\), \(110+18(0.5)=119\), and \(110+18(1.5)=137\). The prediction errors, defined as observed minus predicted, are \(-3,-1,1,3\). Their squared values are \(9,1,1,9\), so the sum is 20. The RMSE is \(\sqrt{20/4}=\sqrt{5}=2.236\), rounded to three decimals.

Combined model. For clusters A, \(I_B=0\); for clusters B, \(I_B=1\). The predictions are \(109+18(-1.5)=82\), \(109+18(-0.5)=100\), \(109+18(0.5)+2=120\), and \(109+18(1.5)+2=138\). The errors are \(-2,0,0,2\), whose squared values sum to \(4+0+0+4=8\). Thus the RMSE is \(\sqrt{8/4}=\sqrt{2}=1.414\), rounded to three decimals.

The combined model has the lower validation RMSE for these four records: 1.414 rather than 2.236. It improves the observed validation error by capturing a pattern represented by the cluster term in this example. The common trap is to claim that combining features is generally superior from one comparison. This result supports the combined model on this validation set, not a universal conclusion. The final check is that both RMSEs use the same four outcomes: squared-error totals 8 and 20 divided by 4 give the reported values.

A Practical Diagnostic Sequence

State the candidate representations.
List the cluster indicators and PC scores in each candidate model. Identify the reference cluster and the number of scores so the feature set is unambiguous.

Check the model matrix.
Confirm that the indicators use reference coding when an intercept is present. Look for exact linear dependence and unusually overlapping representations before interpreting individual coefficients.

Fit and inspect the candidate models.
Check whether the model can estimate its coefficients and whether the added terms change fitted predictions. A coefficient reported as aliased or dropped is a signal to investigate the design matrix, not evidence of a zero effect.

Compare predictive performance fairly.
Calculate the same metric on the same validation records for each candidate. Consider whether the improvement justifies the added terms, and avoid treating training fit as a substitute for validation performance.

The central judgment is about complementary predictive information. Cluster features can represent differences among groups; PC scores can represent smooth movement through a lower-dimensional predictor space. A useful combined model retains both kinds of signal without confusing duplicated columns for additional information. If the combined model is redundant, simplify its representation. If it is identifiable but does not improve validation performance, prefer the simpler candidate unless another clearly stated objective justifies the extra complexity.

Key takeaway: Cluster indicators and PC scores can complement one another, but their combination must be checked for rank redundancy and evaluated on common validation data. Count the parameters, verify that coefficients are estimable, and judge the added features by predictive performance rather than by training fit alone.

Check Your Understanding

For each problem, use the stated model and data to check coding, redundancy, or predictive performance.

A model has an intercept, five cluster levels represented by reference coding, and three PC scores. How many regression coefficients does it estimate if the design matrix has full column rank? (Answer: 8 coefficients.)
A binary cluster indicator satisfies \(H=(Z+1)/2\) in every record. For the model \(\hat{Y}=b_0+b_1Z+b_2H\), what two combinations of coefficients determine the fitted line, and why are the three coefficients not uniquely identified? (Answer: The fitted intercept is \(b_0+b_2/2\) and the fitted slope is \(b_1+b_2/2\); the three coefficients are not unique because \(H\) is a linear combination of the intercept and \(Z\) columns.)
On five validation records, one model has a squared-error total of 45 and another has a squared-error total of 20. Calculate both RMSEs to three decimals and identify the lower one. (Answer: \(\sqrt{45/5}=3.000\) and \(\sqrt{20/5}=2.000\); the second model has the lower RMSE.)
A candidate model includes an intercept and indicators for all three clusters. The three indicator columns sum to one for every record. What is the model-matrix problem, and what standard coding change resolves it? (Answer: Exact linear dependence makes the coefficients nonunique; use an intercept and indicators for only two clusters, treating the third as the reference.)
A full set of PC scores is included together with every centered and scaled original predictor used to calculate those scores. Under a full-rank PCA transformation, what is the concern? (Answer: The two feature sets span the same predictor information and are exactly linearly dependent when both are included, so the coefficient representation is redundant.)

What You'll Learn