Why Create Predictors Without Using the Outcome?
A cluster or principal-component score is constructed from predictor data without using the response variable. That can seem counterintuitive: if the goal is prediction, why create a feature without looking at what must be predicted? The reason is that the new feature can give a model a more useful representation of the predictors. A grouping can summarize a region of predictor space, while a component score can summarize a shared pattern among several variables.
Neither construction, however, is designed to find outcome signal. A cluster algorithm seeks groups according to a measure of predictor similarity; principal component analysis (PCA) seeks directions that account for variation among predictors. A feature can therefore be meaningful as a summary of \(X\) and still be irrelevant to \(Y\). Its predictive value must be judged by how well a model predicts outcomes, not by how clearly the unsupervised method separates or compresses the predictors.
How a Derived Feature Can Help
A derived feature may help when it makes a pattern easier for the chosen prediction model to express. For example, a simple linear model may struggle with a sharp change between two regions of predictor space. If an unsupervised grouping happens to separate those regions, a cluster indicator can allow the model to use different average predictions across them. Similarly, several highly redundant variables may share a dominant direction of variation; a component score can represent that shared direction with one predictor.
This is about the interaction between representation and model form. A transformation does not create new information about a record: a cluster assignment or score is calculated from information already in \(X\). It can nonetheless change the forms of relationship that a particular model can represent efficiently. A sufficiently flexible model using the original variables might already capture the same structure.
Worked Examples
Worked Example: A Cluster Feature Represents a Step Change
A hypothetical equipment warranty analyst records a numeric operating index \(x\) for four units and their subsequent repair costs \(y\), in hundreds of dollars. The observed pairs are \((-2,1),(-1,1),(1,5),(2,5)\). Suppose a two-cluster procedure based only on \(x\) assigns the first two units to one cluster and the last two to another. Compare the training squared error from a model that predicts one overall mean, a straight-line model, and cluster-specific means.
Method. First calculate the overall response mean: \((1+1+5+5)/4=12/4=3\). Predicting 3 for every unit gives squared error \((1-3)^2+(1-3)^2+(5-3)^2+(5-3)^2=4+4+4+4=16\).
For the straight-line least-squares model, the mean of \(x\) is \((-2-1+1+2)/4=0\), and the mean of \(y\) is 3. The slope is \[ \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} = \frac{(-2)(-2)+(-1)(-2)+(1)(2)+(2)(2)}{4+1+1+4} = \frac{12}{10}=1.2. \] The intercept is \(\bar{y}-1.2\bar{x}=3\). Thus the fitted values at \(x=-2,-1,1,2\) are \(0.6,1.8,4.2,5.4\). The residuals are \(0.4,-0.8,0.8,-0.4\), so the squared error is \(0.16+0.64+0.64+0.16=1.60\).
Each cluster has two records. The first cluster mean is \((1+1)/2=1\), and the second is \((5+5)/2=5\). Predicting each record with its cluster mean gives squared error \(0+0+0+0=0\) on these four observations. In this example, the cluster feature represents a step change that a straight line does not fit exactly. The cluster was formed from \(x\), not from the repair costs; the observed alignment with costs is a separate fact that would need to persist in new data to support a predictive claim.
Check. The cluster means reproduce all four observed costs, so zero training squared error is consistent. Common mistake avoided: Treating the zero training error as proof of future improvement. These four records show a possible mechanism, not validation of predictive performance.
Worked Example: A Component Compresses Redundant Predictors
A hypothetical analyst has two centered measurements, \(x_1\) and \(x_2\), from four applicants: \((1,1),(2,2),(-1,-1),(-2,-2)\). A continuous response \(y\) is \(1,2,-1,-2\), respectively. Use a component score to show how the predictor information can be represented with one feature.
Method. The sample means of both predictors are zero. Using the sample covariance denominator \(n-1=3\), each variance is \((1+4+1+4)/3=10/3\), and their covariance is also \(10/3\). Thus the covariance matrix is \[ S=\begin{pmatrix}10/3&10/3\\10/3&10/3\end{pmatrix}. \] Its eigenvalues are \(20/3\) and \(0\): the direction \((1,1)/\sqrt{2}\) captures all observed predictor variation, while the perpendicular direction captures none. The first component score is therefore \(z=(x_1+x_2)/\sqrt{2}\).
The four scores are \(\sqrt{2},2\sqrt{2},-\sqrt{2},-2\sqrt{2}\). In each case \(y=z/\sqrt{2}\), so a linear prediction using just \(z\) fits all four responses exactly. In this constructed data, the score compresses two perfectly redundant measurements into one without losing their observed variation or response relationship.
Check. The component score is a linear combination of the original predictors, and the stated relationship \(y=z/\sqrt{2}\) holds for all four records. Common mistake avoided: Concluding that compression must improve prediction. Here it preserves the information, but a model using either original predictor could also fit the response exactly.
Worked Example: High Predictor Variance Can Miss the Response
A hypothetical analyst observes four centered predictor pairs \((x_1,x_2)=(-2,-0.1),(-1,0.1),(1,0.1),(2,-0.1)\), and the response is \(y=x_2\). Calculate the variance proportions and assess what happens if only the first principal component is retained.
Method. The sample means of both predictors are zero. The sum of squared \(x_1\) values is \(4+1+1+4=10\); the sum of squared \(x_2\) values is \(0.01+0.01+0.01+0.01=0.04\). The cross-product sum is \((-2)(-0.1)+(-1)(0.1)+(1)(0.1)+(2)(-0.1)=0.2-0.1+0.1-0.2=0\). The sample covariance matrix is therefore diagonal, with entries \(10/3\) and \(0.04/3\). Its principal directions are the coordinate axes, and the first component is \(x_1\).
The first component’s proportion of total predictor variance is \(10/(10+0.04)=10/10.04\approx0.9960\), or 99.60%. But the response is \(x_2\). Its mean is zero, and its cross-product with \(x_1\) is zero, so the least-squares slope for predicting \(y\) from \(x_1\) is \(0/10=0\). The resulting constant prediction of zero has squared error \(0.04\). Using \(x_2\) instead predicts every response exactly, with squared error zero.
Check. The retained component explains nearly all predictor variation, yet the discarded direction contains the response in this example. Common mistake avoided: Using a high explained-variance proportion as though it were evidence of high predictive value.
Why These Results Hold
The PCA example follows from what PCA optimizes. For centered predictor data with sample covariance matrix \(S\), a unit-length direction \(w\) produces the score \(Xw\). Its sample variance is \(w^\mathsf{T}Sw\). PCA selects directions that maximize this variance, with later directions constrained to be perpendicular to earlier ones. In particular, the first direction is an eigenvector of \(S\) with the largest eigenvalue.
This criterion contains the predictors, but not the response. If \(Y\) varies strongly along a direction with little variation in \(X\), variance-maximizing PCA may leave that direction out when only a few components are retained. Conversely, a direction can explain substantial variation in \(X\) while having little relationship with \(Y\). The calculations above show both possibilities.
A cluster assignment has a related limitation. A clustering method groups records according to predictor similarity under its chosen distance and settings. It does not choose groups to make outcomes similar or predictions accurate. Cluster-based features are most plausible when predictor regions correspond to response differences that the prediction model would otherwise represent poorly. That correspondence must be assessed using outcome-based predictive evidence.
When Derived Features Are Unlikely to Help
Unsupervised predictors are less promising when the structure they summarize is unrelated to the response, when the original prediction model already captures the relevant pattern, or when a compressed representation discards response-relevant variation. They can also add complexity without useful signal: a feature may be unstable, hard to interpret, or based on distinctions that do not matter for the prediction task.
A useful way to reason about a proposed feature is to ask three questions:
- What structure does it summarize? State whether it represents a group of similar records, a shared direction of variation, or something else precise.
- Why might that structure relate to the response? This is a substantive prediction argument, not a consequence of clustering or PCA itself.
- Does it improve predictions? Compare models using appropriate held-out predictive evidence. A better clustering objective or a larger variance proportion does not answer this question.
The same transformation can help one model and be redundant for another. A step-like response pattern may be awkward for a simple linear predictor but straightforward for a more flexible model. A component score may reduce redundant inputs, yet offer little benefit when the original variables are already handled effectively. These are reasons to treat unsupervised features as candidates to assess, not automatic improvements.
Common Traps
- Confusing discovery with prediction. Finding distinct groups in \(X\) does not show that the groups differ in \(Y\).
- Equating explained variance with predictive value. PCA’s variance criterion concerns \(X\), while prediction concerns \(Y\).
- Assuming a derived feature adds information. A score or cluster assignment is calculated from the original predictors; its potential benefit is a more useful representation for a particular model.
- Judging by training fit alone. A more elaborate set of derived predictors can fit the development data better without improving predictions on new records.
Check Your Understanding
For each situation, distinguish the unsupervised objective from the predictive question and calculate any requested quantities.
- A two-cluster solution groups hypothetical policy records using two rating variables, but no claim about claim frequency has been checked. What does the clustering establish, and what remains to be assessed? (Answer: It establishes a grouping under the chosen predictor-based clustering method; whether cluster membership helps predict claim frequency remains to be assessed using predictive evidence.)
- Four centered observations have predictor pairs \((1,0),(0,1),(-1,0),(0,-1)\). Using sample covariance with denominator 3, calculate the covariance matrix and total sample variance. (Answer: The covariance matrix is \(\begin{pmatrix}2/3&0\\0&2/3\end{pmatrix}\), and total sample variance is \(4/3\); there is no unique preferred PCA direction.)
- A proposed derived score is \(q=(-x_1+3x_2)/2\). For records \((1,0),(0,1),(-1,-1),(0,0)\), calculate the score for each record. (Answer: \(-0.5,\ 1.5,\ -1,\ 0\), respectively.)
- Two orthogonal components explain 92% and 8% of predictor variance. The response is strongly associated with the second component and weakly with the first. What is the risk of retaining only the first? (Answer: The model may discard the response-relevant direction; the 92% variance proportion alone does not justify excluding the second component.)
- A cluster feature reduces training squared error from 40 to 12, but no held-out results are available. Is predictive improvement established? (Answer: No. The training error is lower, but predictive improvement on new records has not been established.)