Why Use Policyholder Clusters?
A policyholder portfolio may contain patterns that are not obvious from any single predictor. For example, a group with relatively high annual mileage and frequent hard braking may differ from a group with low mileage and few braking events. Clustering can summarize such patterns by assigning records to groups based on their predictor values, without using the outcome being modeled.
Those groups can then be considered as a candidate feature in a predictive model. This is a question to test, not an assumption: a cluster label may summarize useful structure, or it may add no predictive value beyond the original features. The labels are exploratory segments produced by an algorithm. They do not establish that policyholders belong to permanent or naturally distinct customer types.
Scale Predictors Before Measuring Distance
Many clustering methods use distances between records. In \(k\)-means clustering, the usual distance is Euclidean distance: the square root of the sum of squared differences across the selected features. If one feature is recorded in large numerical units and another in small units, the first can dominate the distance even when it is not substantively more important.
A common preparation for numeric features is standardization. For a feature \(j\), let \(x_{ij}\) be its value for record \(i\), and let \(\bar{x}_j\) and \(s_j\) be the mean and standard deviation calculated for that feature in the data used to form the clusters. Its standardized value is \(z_{ij}=(x_{ij}-\bar{x}_j)/s_j\). A standardized value of 1 means one standard deviation above the feature mean; a value of \(-1\) means one standard deviation below it.
Here, \(d^2(i,c)\) is the squared Euclidean distance from record \(i\) to cluster center \(c\), and \(\mu_{cj}\) is the center's coordinate for feature \(j\) on the standardized scale. Squared distance is used in the \(k\)-means objective; comparing squared distances gives the same nearest center as comparing distances because the square root is increasing.
Standardization is not a guarantee that all features deserve equal influence. It puts features on a scale measured in standard deviations, but the choice of features and their substantive relevance still matter. Features that are duplicates or strongly related can effectively give one concept extra influence. Identifiers should not be clustered as measurements, and a response such as future claim cost should not be used to create groups that will later be presented as outcome-independent predictors. Categorical predictors also need deliberate treatment; arbitrary numeric codes for categories do not create meaningful distances.
Worked Example: Why Standardization Changes a Cluster Assignment
A hypothetical insurer is exploring two numeric predictors: annual mileage in thousands of miles and hard-braking events per year. The clustering reference data have means of 10 and 5, respectively, and standard deviations of 4 and 2. Two candidate cluster centers are \((-1,-1)\) and \((1,1)\) on the standardized scale. Assign a policyholder with mileage 9.2 and 6.6 braking events to the nearer center. Name the method and identify the scaling mistake this avoids.
Method. Standardize both features using the stated means and standard deviations, then compare squared Euclidean distances to the two centers. The policyholder's standardized mileage is \((9.2-10)/4=-0.2\). The standardized braking value is \((6.6-5)/2=0.8\). Thus the point is \((-0.2,0.8)\).
To center \((-1,-1)\), the squared distance is \((-0.2+1)^2+(0.8+1)^2=0.8^2+1.8^2=0.64+3.24=3.88\). To center \((1,1)\), it is \((-0.2-1)^2+(0.8-1)^2=(-1.2)^2+(-0.2)^2=1.44+0.04=1.48\). Since \(1.48<3.88\), assign the policyholder to the second center.
Check. The same centers correspond to original-scale values \((6,3)\) and \((14,7)\). If raw squared distances were used instead, the distances would be \((9.2-6)^2+(6.6-3)^2=10.24+12.96=23.20\) and \((9.2-14)^2+(6.6-7)^2=23.04+0.16=23.20\), a tie. Raw-unit distance therefore does not produce the standardized assignment. The common mistake is to use unscaled values in a distance calculation even though the features have different units and spreads.
What \(k\)-Means Optimizes
For a chosen number \(K\) of clusters, \(k\)-means assigns each observation to one cluster and represents each cluster by its coordinate-wise mean, called its centroid. On standardized features, it seeks assignments and centroids that minimize the total within-cluster sum of squared distances:
In this expression, \(C_c\) is the set of records assigned to cluster \(c\), and \(\mu_{cj}\) is the mean standardized value of feature \(j\) among those records. The objective favors groups whose observations lie relatively close to their centroids. It does not guarantee that the groups correspond to real-world customer types, or that a unique best segmentation exists.
The centroid is the mean because the mean minimizes the sum of squared deviations. For one feature in one cluster, hold the assignments fixed and let \(a\) be a proposed center. The sum of squared deviations is \(g(a)=\sum_{i\in C_c}(z_i-a)^2\). Differentiating gives \(g'(a)=-2\sum_{i\in C_c}(z_i-a)\). At a minimum, \(g'(a)=0\), so \(a=(1/|C_c|)\sum_{i\in C_c}z_i\), the cluster mean. Since \(g''(a)=2|C_c|>0\) for a nonempty cluster, this stationary point is a minimum. Applying this argument separately to each feature gives the coordinate-wise centroid.
This optimization is generally carried out by repeatedly assigning records to their nearest current centroid and recomputing the centroids. The result can depend on the initial centroids, so a single run should not be treated as definitive. The method is most natural for numeric features and distance structures where mean-based centers are sensible; it can be a poor description when groups are irregularly shaped or dominated by unusual observations.
Worked Example: Describe Two Clusters in Original Units
Suppose a clustering run on standardized mileage and hard-braking events produces the following assignments for four hypothetical policyholders: cluster A contains \((-1,-1)\) and \((-1,1)\); cluster B contains \((1,1)\) and \((1,3)\). The reference means and standard deviations are 10 thousand miles and 4 thousand miles for mileage, and 5 events and 2 events for braking. Calculate the centroids, translate them back to the original units, and give a restrained description of the groups.
Method. Compute the mean of each standardized coordinate within each cluster. Cluster A's mileage coordinate is \((-1-1)/2=-1\), and its braking coordinate is \((-1+1)/2=0\). Its centroid is therefore \((-1,0)\). Cluster B's mileage coordinate is \((1+1)/2=1\), and its braking coordinate is \((1+3)/2=2\), giving centroid \((1,2)\).
To return to original units, use \(x=\bar{x}+zs\). Cluster A's centroid corresponds to mileage \(10+(-1)(4)=6\) thousand miles and braking \(5+0(2)=5\) events. Cluster B corresponds to mileage \(10+1(4)=14\) thousand miles and braking \(5+2(2)=9\) events.
Interpretation. In this small example, A is centered on lower mileage and average braking relative to the reference data; B is centered on higher mileage and higher braking. These statements describe the observed features and computed centers. They do not show that every member of A or B has those values, that the groups are stable, or that membership causes different claim outcomes.
Test Membership as a Candidate Predictor
After forming clusters without using the response, a model can include membership as a categorical predictor. For \(K\) clusters, a regression model commonly represents membership with \(K-1\) indicator variables and a reference cluster. The cluster identifiers “1,” “2,” and “3” are labels, not measurements: entering them as a single numeric predictor would impose an unsupported ordering and equal spacing.
The useful question is incremental predictive value. Fit a baseline model using the chosen existing predictors, then compare it with a model that also includes cluster membership. Compare predictions on the same validation records using a metric appropriate to the response and decision. For a continuous cost target, an error metric such as RMSE may be relevant; for a binary target, use a metric suited to probabilities or the intended decision. The metric, validation design, and response definition must be held consistent in the comparison.
A favorable change in one validation result is evidence to consider, not proof that clustering has discovered a durable segment. Check whether cluster sizes are large enough to interpret, whether the group descriptions make sense in original units, and whether the apparent gain is material for the intended use. Small groups or labels that are difficult to explain may create operational burden without reliable predictive benefit.
Worked Example: Does Membership Improve Validation RMSE?
On the same four hypothetical validation policies, a baseline cost model has prediction errors (observed minus predicted), in thousands of dollars, of \(2,-1,0,1\). A model that also uses cluster membership has errors \(1,-1,1,0\). Calculate both RMSE values, compare them, and state what the comparison supports. The fitted predictions and cluster assignments are supplied; the task is not to refit either model.
Method. For \(n\) validation observations with errors \(e_i\), RMSE is \(\sqrt{\sum_i e_i^2/n}\). The baseline sum of squared errors is \(2^2+(-1)^2+0^2+1^2=4+1+0+1=6\). Its mean squared error is \(6/4=1.5\), so its RMSE is \(\sqrt{1.5}=1.2247\) thousand dollars, rounded to four decimal places.
For the model with membership, the sum of squared errors is \(1^2+(-1)^2+1^2+0^2=1+1+1+0=3\). Its mean squared error is \(3/4=0.75\), so its RMSE is \(\sqrt{0.75}=0.8660\) thousand dollars, rounded to four decimal places. The reduction is \(1.2247-0.8660=0.3587\) thousand dollars, using the rounded values.
Conclusion and trap. For these supplied predictions, adding membership is associated with lower RMSE on this validation set. This result supports further consideration, not a claim of universal improvement or meaningful customer types. A common error is to conclude that the feature is useful just because the model fit improves on the data used to fit it; this calculation concerns validation predictions and does not establish performance on every future portfolio.
Practical Checks and Common Traps
Use predictors available for the intended modeling task. Exclude identifiers and avoid using the response to construct groups that are then described as unsupervised segments.
Standardize numeric features when their units or spreads differ, and retain the means and standard deviations so values can be interpreted and new records assigned consistently.
Review cluster sizes, centers on both standardized and original scales, and whether a few unusual records dominate the interpretation. Treat the clustering as one plausible summary, not a discovered truth.
Encode membership categorically, compare a model with and without it on the same validation basis, and decide whether any improvement is useful enough to justify the added complexity.
Two errors are particularly tempting. First, a numeric cluster code is not an ordered risk score: cluster 3 is not inherently “more” than cluster 2. Second, cluster descriptions are based on centers and feature distributions, not on every individual in the group. Use qualified language such as “the cluster is centered on higher mileage,” rather than assigning a fixed behavioral label to all of its members.
Check Your Understanding
Work through each item, keeping the distinction between cluster construction and predictive evaluation clear.
- A numeric feature has reference mean 40 and standard deviation 10. What is the standardized value of an observation equal to 55? (Answer: 1.5)
- Two standardized centers are \((0,0)\) and \((2,0)\). A record is at \((1.4,0.5)\). Calculate the squared distances and choose a cluster. (Answer: distances 2.21 and 0.61; assign to the second center)
- Two cluster centers have standardized mileage coordinates \(-0.5\) and \(1.5\). Mileage has reference mean 12 thousand miles and standard deviation 4 thousand miles. Give both centers in original mileage units. (Answer: 10 thousand miles and 18 thousand miles)
- A model has validation errors \(1,-2,1\), in thousands of dollars. Calculate RMSE to four decimal places. (Answer: \(\sqrt{2}=1.4142\) thousand dollars)
- A fitted model uses cluster codes 1, 2, and 3 as one numeric predictor. Identify the modeling concern and a suitable correction. (Answer: the model imposes an arbitrary order and spacing on labels; encode membership as a categorical feature, typically using indicator variables)