Using K-Means to Explore Risk Profiles
An insurer may want to explore whether policy profiles with similar numerical features form recognizable groups. For example, a hypothetical portfolio might be described by claim frequency and average claim severity. K-means can group profiles that are close together in those features, but the result depends on how the features are measured and scaled. A cluster is a description of similarity in the chosen inputs; it is not, by itself, a conclusion about causation, profitability, or an individual policy’s future risk.
The method is useful when the observations and features are numerical and distance is a meaningful way to describe similarity. In the examples below, each observation is a hypothetical territory profile, and the features are its claim frequency and average claim severity. The calculations illustrate the method, not actual insurance data.
In practice, the iterative algorithm alternates between assigning each observation to its nearest current center and recalculating each center as the mean of its assigned observations. The final result can depend on the starting centers; a run reaches a solution for its initialization, but that solution is not guaranteed to be the global minimum. Here, the assignments and recalculations will be shown explicitly so that the resulting groups can be checked.
Scaling and Interpreting Actuarial Features
Features measured on different scales can have very different numerical ranges. If severity is measured in dollars while frequency is a small rate, unscaled Euclidean distances may be dominated by severity simply because its numbers are larger. One common remedy is to standardize each feature using a stated reference mean and standard deviation:
The reference values might come from the analysis data or from a defined benchmark. They must be stated and applied consistently. After clustering standardized features, a center can be returned to the original units by multiplying by the feature’s standard deviation and adding its reference mean. This makes the center easier to describe, but it does not change what the algorithm was fitted to.
A centroid describes the average feature values of the observations assigned to that cluster. It does not necessarily describe every member well: profiles can be spread around the center, and observations near a boundary can be nearly as close to another cluster. In addition, k-means gives each row equal weight unless a weighted method is specified. If rows represent territories of very different sizes, that distinction matters when describing the result.
Worked Example: Clustering Hypothetical Territory Profiles
A hypothetical insurer summarizes six territory profiles by claim frequency per 100 policy-years and average claim severity in thousands of dollars. For this illustration, standardize frequency using reference mean 4 and standard deviation 2, and severity using reference mean 10 and standard deviation 5. Use \(K=2\). Suppose the initial groups are A, B, C and D, E, F.
| Profile | Frequency | Average severity ($000s) | Standardized coordinates |
|---|---|---|---|
| A | 2 | 5 | (-1, -1) |
| B | 2 | 10 | (-1, 0) |
| C | 4 | 5 | (0, -1) |
| D | 6 | 10 | (1, 0) |
| E | 6 | 15 | (1, 1) |
| F | 8 | 15 | (2, 1) |
Step 1: Check the standardization. For A, the standardized frequency is \((2-4)/2=-1\), and standardized severity is \((5-10)/5=-1\), giving \((-1,-1)\). For F, the corresponding calculations are \((8-4)/2=2\) and \((15-10)/5=1\), giving \((2,1)\). The other rows use the same formula.
Step 2: Calculate the initial centers. The mean of A, B, C is \(((-1)+(-1)+0)/3=-2/3\) in the first coordinate and \(((-1)+0+(-1))/3=-2/3\) in the second. The mean of D, E, F is \((1+1+2)/3=4/3\) and \((0+1+1)/3=2/3\). Thus the centers are \(\boldsymbol{\mu}_1=(-2/3,-2/3)\) and \(\boldsymbol{\mu}_2=(4/3,2/3)\).
Step 3: Check the assignments. The table gives squared distances to both centers. Using squared distance avoids taking square roots and gives the same nearest-center assignment as distance.
| Profile | Squared distance to \(\boldsymbol{\mu}_1\) | Squared distance to \(\boldsymbol{\mu}_2\) | Nearest center |
|---|---|---|---|
| A | 2/9 | 74/9 | 1 |
| B | 5/9 | 53/9 | 1 |
| C | 5/9 | 41/9 | 1 |
| D | 29/9 | 5/9 | 2 |
| E | 50/9 | 2/9 | 2 |
| F | 89/9 | 5/9 | 2 |
For example, profile C’s squared distance to center 1 is \((0+2/3)^2+(-1+2/3)^2=4/9+1/9=5/9\). Its squared distance to center 2 is \((0-4/3)^2+(-1-2/3)^2=16/9+25/9=41/9\). The table shows that every profile remains in its initial group, so recalculating the centers does not change them. The assignments have converged for this initialization.
Step 4: Calculate WCSS and interpret the centers. For cluster 1, the squared distances from A, B, C to its center sum to \(2/9+5/9+5/9=12/9=4/3\). For cluster 2, the sum is \(5/9+2/9+5/9=12/9=4/3\). The total WCSS is \(4/3+4/3=8/3\). The first center corresponds to frequency \(4+2(-2/3)=8/3\) per 100 policy-years and severity \(10+5(-2/3)=20/3\) thousand dollars. The second corresponds to frequency \(4+2(4/3)=20/3\) and severity \(10+5(2/3)=40/3\) thousand dollars.
The centers therefore describe one group with lower average frequency and severity than the other in this particular feature space. They do not establish that the groups have different expected profitability, nor do they show why the observed profiles differ. The common mistake to avoid is interpreting a cluster label as a validated risk class rather than a summary of similarity in the inputs.
Why the Center Is the Mean
The centroid is not an arbitrary summary: for a fixed cluster, it is the point that minimizes the sum of squared Euclidean distances to the observations. To see why, let \(\bar{\mathbf{x}}\) be the component-wise mean of the \(m\) vectors in a cluster, and consider any proposed center \(\mathbf{c}\). Write each difference as \(\mathbf{x}_i-\mathbf{c}=(\mathbf{x}_i-\bar{\mathbf{x}})+(\bar{\mathbf{x}}-\mathbf{c})\). Expanding the squared norms and summing gives
The cross term is zero because \(\sum_{i=1}^m(\mathbf{x}_i-\bar{\mathbf{x}})=\mathbf{0}\). The first term does not depend on \(\mathbf{c}\), and the second is nonnegative and equals zero when \(\mathbf{c}=\bar{\mathbf{x}}\). Therefore the mean minimizes the sum of squared distances. This argument relies on squared Euclidean distance; it does not establish that the mean is the appropriate center for every possible distance or clustering method.
Worked Example: Assigning a New Hypothetical Profile
Use the two centers from the territory example without refitting them. A new hypothetical profile has frequency 4 per 100 policy-years and average severity of $10,000. Assign it to the nearer center in standardized coordinates.
Step 1: Standardize the new profile. Its frequency coordinate is \((4-4)/2=0\), and its severity coordinate is \((10-10)/5=0\). The point is \((0,0)\).
Step 2: Compare squared distances. To center 1, the squared distance is \((0+2/3)^2+(0+2/3)^2=4/9+4/9=8/9\). To center 2, it is \((0-4/3)^2+(0-2/3)^2=16/9+4/9=20/9\).
Conclusion. Since \(8/9<20/9\), assign the new profile to cluster 1. This is a nearest-center assignment using the fitted centers; it does not mean the new profile was included when those centers were calculated. The common mistake is to compare distances in raw units after fitting on standardized features. The assignment must use the same feature transformation as the fit.
Worked Example: What an Unweighted Center Represents
Suppose a hypothetical cluster contains three territory profiles: R has frequency 2 and severity $10,000; S has frequency 4 and severity $20,000; T has frequency 10 and severity $30,000. For the frequency feature, use reference mean 4 and standard deviation 2; for severity in thousands of dollars, use reference mean 20 and standard deviation 10. Find the standardized center and interpret it in original units. Then compare it with an exposure-weighted average if the territories have 100, 100, and 10,000 policy-years, respectively.
Step 1: Standardize the profiles. R becomes \(((2-4)/2,(10-20)/10)=(-1,-1)\). S becomes \((0,0)\). T becomes \(((10-4)/2,(30-20)/10)=(3,1)\).
Step 2: Find the k-means center. Taking the component-wise means gives frequency coordinate \((-1+0+3)/3=2/3\) and severity coordinate \((-1+0+1)/3=0\). The standardized center is \((2/3,0)\).
Step 3: Convert to original units. The frequency represented by the center is \(4+2(2/3)=16/3\), or about 5.333 claims per 100 policy-years. Severity is \(20+10(0)=20\), or $20,000. Directly averaging the original values gives \((2+4+10)/3=16/3\) for frequency and \((10+20+30)/3=20\) thousand dollars for severity, as expected for this linear standardization.
Step 4: Compare the exposure-weighted averages. The frequency average weighted by policy-years is \((100(2)+100(4)+10{,}000(10))/10{,}200=100{,}600/10{,}200\), approximately 9.863 claims per 100 policy-years. The weighted severity average is \((100(10)+100(20)+10{,}000(30))/10{,}200=303{,}000/10{,}200\), approximately $29.706 thousand. These differ substantially from the unweighted centroid.
The k-means center describes the equal-weight average of the three profile rows. Policy-year-weighted averages answer a different question, and standard k-means does not automatically use those weights. The common mistake is to interpret the centroid as an exposure-weighted portfolio average without checking how the observations entered the analysis.
Describing Clusters With Appropriate Caution
A useful cluster description names the features and units, reports the center or relevant summaries, and acknowledges that membership is relative to the chosen inputs, scaling, value of \(K\), and fitted solution. For example, a defensible statement is: “In this illustration, one group has a lower mean claim frequency and lower mean severity than the other, under the stated standardization.” It would not be justified to call it a “safe” group or to infer that its policies should receive a particular price from clustering alone.
A center alone can hide substantial variation. Before using cluster labels in a business discussion, consider the number of profiles in each cluster, the spread of profiles around each center, and whether individual profiles lie close to competing centers. Also check that the numerical features have clear meanings and units. A label encoded as a number is not automatically a quantitative measure, and including an arbitrary code in Euclidean distance can produce misleading groupings.
Check Your Understanding
For each problem, use the stated feature coordinates and assumptions.
- A profile has frequency 3 claims per 100 policy-years and severity $25,000. Standardize frequency using mean 5 and standard deviation 2, and severity in thousands using mean 20 and standard deviation 10. What are its coordinates? (Answer: \((-1,0.5)\))
- A standardized profile is \((1,0)\). Its two centers are \((0,0)\) and \((2,1)\). Which center is nearer, and what are the two squared distances? (Answer: Center 1; squared distances 1 and 2.)
- A cluster contains standardized points \((-1,0)\), \((1,2)\), and \((0,1)\). Find its centroid. (Answer: \((0,1)\))
- A cluster contains points \((0,0)\), \((2,0)\), and \((1,0)\). Find its center and WCSS. (Answer: Center \((1,0)\); WCSS \(=2\).)
- Using frequency reference mean 5 and standard deviation 2, and severity reference mean $20,000 and standard deviation $10,000, standardize a profile with frequency 7 and severity $15,000. Compare its squared distances to centers \((0,0)\) and \((2,-1)\). (Answer: The profile is \((1,-0.5)\); both squared distances are 1.25, so it is tied and a tie-handling rule is needed.)