Why Cluster Granularity Is a Prediction Choice
A clustering divides records into groups based on their predictor values. When cluster membership is then used as a predictor, choosing the number of clusters \(k\) also chooses how finely the downstream model can distinguish records. A small \(k\) gives a compact summary; a larger \(k\) can represent more distinctions, but may create small groups and a more complex model.
The right number of clusters is therefore not determined by the clustering algorithm alone. A partition that is compact in predictor space is not necessarily useful for predicting the response. Compare candidate values of \(k\) by fitting the same downstream model and assessing its predictive performance on the same validation records. The choice should also account for model complexity, group sizes and whether the assignments are reproducible under modest changes to the training data.
The clustering is an unsupervised transformation: its assignment rule is determined from predictor data, not from the response. The predictive model, however, is evaluated against the response. Keep those purposes distinct. A clustering criterion such as within-cluster sum of squares measures properties of the predictor-space partition; it does not directly measure predictive accuracy.
A Consistent Comparison Process
Use the same outcome, validation records, downstream model specification and performance metric for every candidate \(k\). If these change between candidates, a difference in score cannot be attributed cleanly to the cluster count. The candidate values should be chosen to represent meaningfully different levels of granularity, not just to search for a numerically attractive score.
Choose candidate cluster counts and a downstream performance metric appropriate to the response. State any operational constraints, such as a minimum acceptable cluster size, before comparing results.
For each candidate, derive cluster membership and include it in the same downstream model, with all other modeling choices held constant.
Use the same held-out observations and metric. For a lower-is-better metric such as RMSE or log loss, a smaller value indicates better predictive performance on those observations.
Consider the number of model terms, the sizes of the resulting groups and whether assignments are sensitive to reasonable perturbations of the training data.
Select the candidate that offers a useful and credible predictive gain without unacceptable complexity or instability. If scores are close, a predeclared preference for the simpler option can make the decision consistent.
Predictive Metrics and Model Complexity
For a quantitative response, one common comparison metric is root mean squared error. Given \(n\) validation observations, actual responses \(y_j\), and predictions \(\hat{y}_j\), it is the square root of the mean squared prediction error:
For a binary response, log loss is one possible metric. If \(y_j\) is 1 for an event and 0 otherwise, and \(\hat{p}_j\) is the predicted event probability, the average log loss is:
Both metrics assess predictions against outcomes; their scales and interpretation differ, so use the metric suited to the task rather than comparing their numerical values directly. A metric that averages over records can also conceal a small group with poor predictions. That is one reason to inspect group sizes and, where relevant, group-specific performance alongside the overall score.
With a categorical cluster predictor having \(k\) observed levels, a standard reference-coded regression model uses \(k-1\) indicator coefficients for that predictor. It also has an intercept and any other model terms. Thus, increasing \(k\) can add degrees of freedom to the downstream model.
More coefficients do not guarantee a better model on new records. A finer partition may fit particular patterns in the development data that do not persist, while a small cluster may provide little information for estimating its effect. These are reasons to consider validation results and group sizes together, not reasons to impose one universal maximum \(k\).
Worked Example: Comparing RMSE Under a Minimum-Size Rule
A hypothetical insurer uses cluster membership as a categorical predictor of annual claim cost in a regression model. The same 1,000 validation records are scored for each candidate, and the validation sum of squared prediction errors is 10,000 for the model without clusters, 9,600 for \(k=2\), 9,100 for \(k=4\), and 8,750 for \(k=8\). A policy set before fitting requires every cluster to contain at least 20 training records. The observed training cluster sizes are 540 and 460 for \(k=2\); 330, 260, 240 and 170 for \(k=4\); and 220, 180, 150, 130, 120, 95, 94 and 11 for \(k=8\). Which eligible clustered model should be selected by validation RMSE?
Method. Calculate validation RMSE as the square root of the sum of squared prediction errors divided by the number of validation records. Then apply the stated size rule and compare the eligible candidates. This avoids selecting the most granular model just because it has the smallest raw error.
For the model without clusters, the mean squared error is \(10{,}000/1{,}000=10\), so RMSE is \(\sqrt{10}=3.1623\). For \(k=2\), it is \(9{,}600/1{,}000=9.6\), so RMSE is \(\sqrt{9.6}=3.0984\). For \(k=4\), it is \(9{,}100/1{,}000=9.1\), so RMSE is \(\sqrt{9.1}=3.0166\). For \(k=8\), it is \(8{,}750/1{,}000=8.75\), so RMSE is \(\sqrt{8.75}=2.9580\). The RMSE values are rounded to four decimal places.
All four \(k=4\) groups meet the minimum size of 20. For \(k=8\), one group has only 11 training records, so that candidate fails the predeclared rule despite its lower validation RMSE. Of the eligible clustered models, \(k=4\) has the smaller RMSE. It is selected over \(k=2\); the model without clusters remains a useful baseline.
Check. The \(k=4\) group sizes sum to \(330+260+240+170=1{,}000\), and the \(k=8\) sizes also sum to 1,000. The selection follows the stated rule rather than an after-the-fact preference for a particular score.
Consider Stability as Well as Score
A validation score is an estimate based on particular records. When two candidates have similar scores, the more detailed clustering may not provide a reliably better prediction. One useful diagnostic is assignment stability: do nearby versions of the training data produce similar groupings of the same records?
Cluster labels themselves have no fixed meaning. For example, a group called “cluster 1” in one fit could be called “cluster 3” in another. A simple label-invariant comparison looks at pairs of records: for each pair, ask whether the two records are placed together or separately in each fit. The fraction of pairs for which that together/separate decision agrees is a basic measure of pairwise agreement. It is a diagnostic, not a complete or chance-adjusted stability measure; in particular, it should not be interpreted without considering \(k\).
Stability and predictive performance answer different questions. Stable assignments can still fail to help predict the response. Conversely, a candidate with a modestly better validation score may be less appealing if small perturbations lead to materially different assignments. Evaluate both in context and explain the trade-off.
Worked Example: Trading a Small RMSE Gain Against Complexity
A hypothetical equipment-repair model is evaluated on the same six records for two cluster counts. With \(k=2\), its validation RMSE is 4.00 and it uses \(k-1=1\) cluster coefficient. With \(k=3\), its RMSE is 3.92 and it uses \(k-1=2\) cluster coefficients. To assess stability, each clustering is also refit on a modestly perturbed training sample and applied to the same six records. For \(k=2\), the first fit groups \(\{1,2,3\}\) and \(\{4,5,6\}\), while the second groups \(\{1,2,4\}\) and \(\{3,5,6\}\). For \(k=3\), the first fit groups \(\{1,2\}\), \(\{3,4\}\), \(\{5,6\}\), while the second groups \(\{1,2\}\), \(\{3,5\}\), \(\{4,6\}\). Compare the stability and state a selection under a predeclared rule that requires an RMSE reduction of at least 0.10 to justify the additional cluster coefficient.
Method. There are \(\binom{6}{2}=6(5)/2=15\) record pairs. Compare the together/separate decision for each pair across the two fits, and then apply the stated predictive-gain rule.
For \(k=2\), the pairs placed together in the first fit are \(12,13,23,45,46,56\). In the second fit they are \(12,14,24,35,36,56\). Two pairs, \(12\) and \(56\), are together in both fits. The union contains \(6+6-2=10\) pairs that are together in at least one fit, so the other \(15-10=5\) pairs are separate in both. Agreement is therefore \((2+5)/15=7/15=0.4667\), rounded to four decimals.
For \(k=3\), the together-pairs in the first fit are \(12,34,56\); in the second they are \(12,35,46\). Only pair \(12\) is together in both. The union contains \(3+3-1=5\) pairs, leaving \(15-5=10\) pairs separate in both. Agreement is \((1+10)/15=11/15=0.7333\), rounded to four decimals. On this comparison, the \(k=3\) assignments are more stable.
The validation RMSE reduction from \(k=2\) to \(k=3\) is \(4.00-3.92=0.08\), which is less than the required 0.10. The stated rule therefore selects \(k=2\), despite the higher pairwise agreement for \(k=3\). This does not establish that \(k=2\) is universally preferable; it applies the specified decision rule to the given evidence.
Check. The pair counts use all 15 distinct pairs, and the two reported agreement fractions are between 0 and 1. The selection follows the 0.10 threshold rather than treating stability as a substitute for predictive performance.
Common Traps in Selecting \(k\)
- Choosing the smallest clustering objective. A compact partition in predictor space is not automatically the partition that best predicts the response. Use the downstream validation metric for predictive comparisons.
- Assuming more clusters must help. More groups can add model coefficients, and a small group may yield an uncertain estimate. A lower training error alone is not evidence of better generalization.
- Comparing unlike models. Changing the validation records, metric or other model terms between candidates makes the score comparison difficult to interpret.
- Ignoring the reference level. A \(k\)-level cluster factor generally adds \(k-1\), not \(k\), coefficients in a reference-coded model with an intercept.
- Treating one score as definitive. Small differences may be inconclusive. Consider their practical size, group sizes and assignment stability before claiming that one \(k\) is clearly superior.
A disciplined comparison makes the reason for choosing \(k\) explicit: for example, a meaningful improvement in validation RMSE, subject to a minimum group size, or a preference for the smaller candidate when the scores are close. There is no universal penalty or threshold that applies to every response and use case; the rule should fit the modeling objective and be set before inspecting the final comparison.
The approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus: explain the predictive objective, compare candidates on a relevant metric, and justify the selected level of complexity with evidence rather than relying on cluster count alone.
Check Your Understanding
For each problem, use the stated comparison rule and show the key calculation.
- A regression model has 800 validation records and a sum of squared prediction errors of 7,200. Calculate its validation RMSE. (Answer: 3.0000)
- A reference-coded regression includes a cluster factor with seven observed groups and an intercept. How many coefficients represent the cluster factor? (Answer: 6)
- Two binary-response models are scored on the same 80 validation records. Their total individual log losses are 56 and 60. What is the first model’s mean log loss, and which model has the lower mean log loss? (Answer: 0.7000; the first model)
- For 8 records, two clustering fits agree on the together/separate status of 18 of the 28 distinct pairs. What is their pairwise agreement, rounded to four decimals? (Answer: 0.6429)
- Two eligible candidates have validation RMSEs of 5.10 at \(k=3\) and 5.04 at \(k=5\). A predeclared rule requires an improvement of at least 0.08 to justify moving to the larger \(k\). Which candidate does the rule select? (Answer: \(k=3\), because the reduction is 0.06)