Why Clusters Can Add Structure to a Prediction Model
A linear predictor imposes a particular shape on the relationship between predictors and a response. For example, a model with one quantitative predictor \(x\) and an intercept predicts \(\beta_0+\beta_1x\): as \(x\) changes, the fitted mean follows one straight line. A response may instead change sharply between regions, remain nearly flat within a region, or follow a pattern that one straight line cannot represent well.
A cluster feature offers another way to represent this structure. Clustering groups records using their predictor values, without using the response to form the groups. A supervised model can then use cluster membership as a categorical predictor. Its fitted values can differ between clusters, allowing the model to represent local shifts in the response. This can improve fit, but a smaller training error alone does not establish better prediction on new records.
With this specification, the fitted value is constant within each cluster. The intercept is the fitted value for the reference cluster; each \(\gamma_j\) is the difference between the fitted value for cluster \(j\) and that reference value. The cluster labels themselves have no numerical ordering or response meaning. Their usefulness comes from membership, not from treating labels such as “1” and “2” as quantitative values.
Why Least Squares Produces Cluster Means
The cluster-indicator model is flexible enough to estimate one fitted value per cluster. To see what those values are, let \(y_i\) be the observed response and let \(m_j\) be the fitted value for every observation in cluster \(j\). Under squared-error loss, the part of the objective for that cluster is \(\sum_{i:C_i=j}(y_i-m_j)^2\).
Solving this first-order condition gives \(m_j=\bar{y}_j\), the observed response mean within cluster \(j\). The second derivative is \(2n_j>0\) when the cluster contains \(n_j>0\) observations, so this stationary point minimizes the sum of squared errors. Thus, for an intercept-plus-cluster-indicators model fitted by least squares, each cluster's fitted value is its training response mean. This result assumes a nonempty cluster and this model specification; it does not say that the mean is automatically the best prediction under a different loss function, such as absolute error.
A Step Pattern That a Straight Line Misses
The first example compares a straight-line model with cluster indicators for a hypothetical data set. The two clusters are formed from predictor values alone: cluster A contains \(x=0\) and \(x=1\), and cluster B contains \(x=2\) and \(x=3\). The response values are then used to fit and compare the supervised models.
Worked Example: A Local Shift in the Response
A hypothetical service-planning team records a workload score \(x\) and a follow-up cost \(y\), in thousands of dollars, for eight cases. Two cases occur at each predictor value. Predictor-based clustering assigns \(x=0,1\) to cluster A and \(x=2,3\) to cluster B. The responses are \(0,0\) at \(x=0\), \(0,0\) at \(x=1\), \(4,4\) at \(x=2\), and \(4,4\) at \(x=3\). Compare an intercept-plus-linear-\(x\) model with an intercept-plus-cluster-indicators model using training SSE.
Method and linear fit. For ordinary least squares with an intercept, the slope is \(S_{xy}/S_{xx}\), and the intercept is \(\bar{y}-\hat{\beta}_1\bar{x}\). Across the eight records, \(\bar{x}=1.5\) and \(\bar{y}=2\). The sum of squared deviations in \(x\) is \(S_{xx}=2(2.25+0.25+0.25+2.25)=10\). The cross-product sum is \(S_{xy}=2(3+1+1+3)=16\), since the centered \(x\) values at \(0,1,2,3\) are \(-1.5,-0.5,0.5,1.5\), and the corresponding response deviations from 2 are \(-2,-2,2,2\). Therefore, \(\hat{\beta}_1=16/10=1.6\) and \(\hat{\beta}_0=2-(1.6)(1.5)=-0.4\).
The linear-model predictions at \(x=0,1,2,3\) are \(-0.4,1.2,2.8,4.4\). The residuals for one case at each of these predictor values are \(0-(-0.4)=0.4\), \(0-1.2=-1.2\), \(4-2.8=1.2\), and \(4-4.4=-0.4\). Each occurs twice, so
Cluster-indicator fit. The mean response in cluster A is \((0+0+0+0)/4=0\), and the mean in cluster B is \((4+4+4+4)/4=4\). Every training response equals its cluster mean, so \(\mathrm{SSE}_{\mathrm{cluster}}=0\). The cluster specification fits this training pattern better: its SSE is 0 rather than 6.4. This is a training-fit comparison, not a guarantee that the cluster model will have lower error on new cases.
Common mistake avoided: Do not conclude that cluster indicators are always superior because they reduce training SSE. A model with more flexibility can fit training data at least as closely; predictive value must be assessed on data not used to fit the model.
Representing Curvature With Local Predictions
Cluster indicators can also approximate a curved relationship. The model does not estimate a smooth curve: it assigns a separate fitted value to each cluster. If clusters correspond to neighboring regions of predictor space, those values can follow a curved pattern across the regions. The approximation depends on the clusters and does not imply that the response is constant throughout a real population region.
Worked Example: A Symmetric Nonlinear Pattern
Suppose five hypothetical sites have predictor values \(x=-2,-1,0,1,2\) and response values \(y=4,1,0,1,4\). A predictor-only clustering assigns \(-2,-1\) to cluster L, \(0\) to cluster M, and \(1,2\) to cluster R. Compare training SSE for an intercept-plus-linear-\(x\) model and a model using the three cluster indicators.
Linear fit. The mean predictor is \(\bar{x}=0\), and the mean response is \(\bar{y}=(4+1+0+1+4)/5=2\). By symmetry, \(S_{xy}=(-2)(2)+(-1)(-1)+0(-2)+1(-1)+2(2)=0\), while \(S_{xx}=4+1+0+1+4=10\). Thus the fitted slope is \(0/10=0\), and the intercept is 2. The fitted value is 2 for every site, so
Cluster fit. The response mean in L is \((4+1)/2=2.5\), the mean in M is 0, and the mean in R is \((1+4)/2=2.5\). The left-cluster squared errors are \((4-2.5)^2+(1-2.5)^2=2.25+2.25=4.5\). The middle-cluster squared error is \((0-0)^2=0\). The right-cluster squared errors are \((1-2.5)^2+(4-2.5)^2=2.25+2.25=4.5\). Therefore \(\mathrm{SSE}_{\mathrm{cluster}}=4.5+0+4.5=9\), which is less than 14 for the linear fit.
The three cluster predictions reproduce the broad low-middle, high-ends pattern better than one straight line does. The cluster model still makes a piecewise-constant prediction, not a smooth quadratic prediction. Its lower training SSE does not by itself establish that it will predict better beyond these five records.
Compare Fit Carefully and Check the Arithmetic
SSE is the sum of squared residuals, \(\sum_i(y_i-\hat{y}_i)^2\). It is measured in squared response units, so it is useful for comparing models fitted to the same response observations, but its magnitude depends on the response scale and number of records. If models use different records or different outcome definitions, their SSEs are not a like-for-like comparison. For predictive assessment, use a validation design and a metric suited to the response and decision. Training SSE describes fit to the training observations; it is not an out-of-sample performance estimate.
A useful arithmetic check is to square residuals before summing them. For example, if a six-record linear fit has residuals \(10/3,-2/3,-2/3,-8/3,-8/3,10/3\), then
The squared residuals sum to \(112/3\), not \(97/3\). In a comparison where the cluster model's training SSE is 17.5 on those same six responses, the cluster model's training SSE is 17.5 rather than 37.333. The difference is not a predictive-performance conclusion; that requires an appropriate validation comparison as well.
Use the predictor information specified for clustering, rather than using response values to create membership. This keeps the feature construction unsupervised.
Fit the simpler model and the model with cluster indicators to the same training records and response. For least-squares cluster indicators, the fitted value in each cluster is its training response mean.
Calculate training fit accurately, then assess predictive performance on validation data. Apply the training-defined clustering rule to new records rather than refitting clusters using those records.
A cluster feature is most useful when it contributes a stable local distinction that the existing model does not capture. It may simply duplicate information already represented by the original predictors, or it may fit accidental training variation. In practice, retain the simpler specification as a baseline, and judge any improvement using a validation strategy consistent with the intended prediction setting. A lower training SSE is evidence of closer in-sample fit only.
Check Your Understanding
Work through these original problems, checking both the model interpretation and the arithmetic.
- A squared-error model has two clusters. Cluster A has responses \(2\) and \(4\), while cluster B has responses \(5\) and \(8\). Find each cluster's fitted value and the total training SSE for an intercept-plus-cluster-indicators model. (Answer: fitted values 3 and 6.5; total SSE \(6.5\).)
- For five records, a linear model has residuals \(2,-1,-1,1,-1\). Calculate its SSE. A cluster-indicator model on the same records has SSE 3. Which model has lower training SSE, and what does this establish? (Answer: linear SSE \(8\); the cluster model has lower training SSE, but this alone does not establish better out-of-sample prediction.)
- In a model with an intercept and three cluster levels, cluster A is the reference. The fitted intercept is 12, and the coefficients for clusters B and C are \(-3\) and \(5\). State the fitted value in each cluster and interpret the coefficient for C. (Answer: A: 12, B: 9, C: 17; the fitted value for C is 5 units above that for A.)
- A validation set contains two records with observed responses 3 and 7. A cluster model predicts 4 and 5; a linear model predicts 2 and 8. Calculate each model's validation SSE and RMSE, using RMSE equal to the square root of SSE divided by the number of records. (Answer: cluster SSE \(5\), RMSE \(\sqrt{2.5}\approx1.5811\); linear SSE \(2\), RMSE \(1\); the linear model is better by these validation metrics.)
- A team forms clusters using predictor values, then uses response values from both training and validation records to decide the cluster boundaries before comparing validation errors. Identify the concern and state the appropriate correction. (Answer: validation information has influenced feature construction; determine the clustering rule using training data only, then apply that fixed rule to validation records.)