What a Cluster Can—and Cannot—Tell You
Clustering can summarize how observations differ across their recorded features. For example, a hypothetical insurer might group policyholders using annual mileage and the share of driving done at night. The groups may have clearly different feature profiles. That is a description of the predictor data; by itself, it does not show that the groups differ in future claims.
This distinction matters whenever a cluster label is used to discuss risk. A cluster is formed from similarities in the variables supplied to the clustering procedure, not from an outcome unless that outcome is explicitly included. Even if a cluster later turns out to have a higher event rate, that association must be assessed using outcome observations, with attention to how those observations were obtained. This distinction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
A profile is easiest to interpret when the analyst states both the feature and its units. If a cluster has higher average mileage and lower average nighttime driving, that describes its members on those two features. It does not, without outcome evidence, mean that the cluster has higher or lower claim risk.
Profiles, Scaling, and Centroids
For a numeric feature, the cluster mean is a basic profile summary. When features were standardized before clustering, the standardized mean describes a cluster’s location relative to the overall feature scale. If a feature \(X_j\) was transformed using its mean \(\bar{x}_j\) and standard deviation \(s_j\), then a standardized value is \(z_{ij}=(x_{ij}-\bar{x}_j)/s_j\). A standardized cluster mean can be converted back to the original units by multiplying by \(s_j\) and adding \(\bar{x}_j\).
For example, a standardized mean of \(-0.5\) means the cluster average is half of one standard deviation below the centering mean. It does not mean the feature itself is negative. Always identify whether a reported profile is standardized or in original units. Also remember that the clusters themselves may depend on scaling: a feature measured in large numerical units can dominate a distance calculation if features are not put on comparable scales.
The reason the mean is the centroid under squared-error distance follows directly from minimization. Fix a cluster with \(n_g\) feature vectors \(\mathbf{z}_1,\ldots,\mathbf{z}_{n_g}\), and consider the sum of squared distances from a proposed center \(\mathbf{c}\). The vectors here are in the space actually used for clustering, such as standardized feature space.
Differentiating with respect to \(\mathbf{c}\) gives \(-2\sum_i\mathbf{z}_i+2n_g\mathbf{c}\). Setting this equal to zero gives \(\mathbf{c}=n_g^{-1}\sum_i\mathbf{z}_i\). The second derivative matrix is \(2n_g I\), which is positive definite when \(n_g>0\), so this solution minimizes the sum of squared distances. This justification applies to squared Euclidean distance; it does not establish that a mean is the right summary for every distance measure or feature type.
Worked Example: Translate Standardized Profiles Into Risk-Feature Units
A hypothetical portfolio is clustered using two standardized features: annual mileage and the proportion of driving done at night. In the full development data, annual mileage has mean 10,000 miles and standard deviation 2,000 miles; nighttime-driving proportion has mean 0.20 and standard deviation 0.10. Cluster A has standardized feature means \((-0.8, 0.6)\), and Cluster B has means \((0.8,-0.6)\). Cluster A contains 30 observations and Cluster B contains 20. Translate the profiles into original units and state what can be concluded.
Method. For each feature, convert its standardized mean \(z\) to original units using \(x=\bar{x}+zs\). This translation describes the feature averages; it does not use or predict an outcome.
Cluster A, mileage. The mean is \(10{,}000+(-0.8)(2{,}000)=10{,}000-1{,}600=8{,}400\) miles per year.
Cluster A, nighttime proportion. The mean is \(0.20+(0.6)(0.10)=0.20+0.06=0.26\), or 26%.
Cluster B, mileage. The mean is \(10{,}000+(0.8)(2{,}000)=10{,}000+1{,}600=11{,}600\) miles per year.
Cluster B, nighttime proportion. The mean is \(0.20+(-0.6)(0.10)=0.20-0.06=0.14\), or 14%.
The two cluster profiles differ by \(11{,}600-8{,}400=3{,}200\) miles per year and \(0.26-0.14=0.12\), or 12 percentage points, in nighttime-driving proportion. The cluster sizes sum to \(30+20=50\), consistent with the stated total. These are feature differences only. The calculation supplies no evidence that either group has a higher claim rate.
Common mistake avoided: Reading the standardized value \(0.6\) as a 60% nighttime-driving rate would confuse standard-deviation units with the original proportion. The correct original-unit value for Cluster A is 26%.
Assignments Describe Similarity in the Chosen Feature Space
An assignment says which cluster an observation belongs to under a particular clustering rule. It is not a rank of risk, and the number in the cluster label has no quantitative meaning. In the following example, assignments are made by choosing the nearer of two fixed centroids using Euclidean distance in standardized feature space. The features are numeric and standardized using the stated means and standard deviations.
Worked Example: Assign a New Observation Without Reading Risk Into the Label
A hypothetical driving profile has annual mileage 12,000 and nighttime-driving proportion 0.18. Standardization uses mileage mean 10,000 and standard deviation 2,000, and nighttime proportion mean 0.15 and standard deviation 0.10. The two fixed cluster centroids in standardized coordinates are \(C_1=(-1,0)\) and \(C_2=(1,0)\). Assign the observation using squared Euclidean distance, and explain the limit of that result.
Step 1: Standardize the observation. Mileage becomes \((12{,}000-10{,}000)/2{,}000=1\). Nighttime proportion becomes \((0.18-0.15)/0.10=0.3\). Thus its standardized coordinates are \((1,0.3)\).
Step 2: Calculate the squared distance to \(C_1\). The differences are \(1-(-1)=2\) and \(0.3-0=0.3\). The squared distance is \(2^2+0.3^2=4+0.09=4.09\).
Step 3: Calculate the squared distance to \(C_2\). The differences are \(1-1=0\) and \(0.3-0=0.3\). The squared distance is \(0^2+0.3^2=0.09\).
Since \(0.09<4.09\), the observation is assigned to Cluster 2 under this rule. The assignment means it is nearer to \(C_2\) in the specified standardized feature space. It does not mean the observation has a particular probability of a claim. That would require an outcome-based analysis.
Check. The observation has the same first standardized coordinate as \(C_2\), and its second coordinate is 0.3 from both centroids. Its squared distance to \(C_2\) must therefore be smaller by the squared difference in the first coordinate, \(2^2=4\); indeed \(4.09-0.09=4\).
When Clusters Provide Outcome Evidence
To investigate whether clusters are associated with a later outcome, the outcome must be observed and summarized separately. For a binary outcome \(Y\), define \(Y=1\) for the event of interest and \(Y=0\) otherwise. In an evaluation sample, the observed event rate in cluster \(g\) is the number of events in that cluster divided by the number of evaluated observations assigned to that cluster. A pooled rate provides a useful reference, but it does not replace a decision-relevant measure of predictive performance.
The timing and use of the outcome matter. If the outcome was not used to create the clusters, comparing cluster-specific outcome rates can reveal an association. If the rates are examined on the same observations used to invent or refine the grouping, the result is exploratory: repeated choices based on those outcomes can make apparent differences look more convincing than they are. A separate evaluation sample assigned using a fixed clustering rule gives stronger evidence about whether the association carries beyond the data that motivated the grouping.
Even an observed difference in a separate sample does not establish causation. Nor does a difference alone show that cluster membership improves decisions: assess the outcome in context, compare with an appropriate baseline, and account for uncertainty and the consequences of errors. A cluster can be well separated on its input features yet show little or no outcome differentiation. Conversely, a modest-looking feature separation can coincide with different event rates, but that relationship must be demonstrated rather than inferred from a cluster plot or centroid table.
Worked Example: Compare Cluster Event Rates With the Pooled Rate
A hypothetical, independent evaluation sample contains 120 accounts. A clustering rule was fixed before these outcomes were examined and uses only predictor features. Cluster A contains 40 accounts, of which 12 have a later event. Cluster B contains 80 accounts, of which 6 have a later event. Calculate the two event rates, the pooled rate, and the rate differences. State what the results support.
Cluster A rate. There are 12 events among 40 accounts, so the rate is \(12/40=0.30\), or 30%.
Cluster B rate. There are 6 events among 80 accounts, so the rate is \(6/80=0.075\), or 7.5%.
Pooled rate. There are \(12+6=18\) events among \(40+80=120\) accounts, giving \(18/120=0.15\), or 15%. The counts reconcile: the non-events are \(40-12=28\) in A and \(80-6=74\) in B, for \(28+74=102\); \(18+102=120\).
Compare rates. Cluster A is \(0.30-0.15=0.15\), or 15 percentage points, above the pooled rate. Cluster B is \(0.075-0.15=-0.075\), or 7.5 percentage points below it. The A-to-B rate ratio is \(0.30/0.075=4\).
Because the clustering rule was fixed and the outcomes come from an independent evaluation sample, these calculations provide evidence of an outcome association in this sample: the observed rate in A is four times the rate in B. They do not by themselves establish causation, quantify uncertainty, or prove that using cluster membership improves a particular decision. Those conclusions require further evaluation appropriate to the intended use.
Common mistake avoided: Dividing A’s 12 events by all 120 accounts gives \(12/120=0.10\), but that is not Cluster A’s event rate. The denominator for a cluster-specific rate is the number of evaluated accounts assigned to that cluster: \(12/40=0.30\). The pooled rate, using all accounts, is separately \(18/120=0.15\).
A Practical Interpretation Checklist
When reading a cluster result, keep the description of the input features separate from any claim about a response or future event. A profile table should identify which variables were used, their units, any standardization, and the number of observations in each group. If a later outcome is reported, check how the rate was calculated and whether the clustering rule was fixed independently of the evaluation outcomes.
- Describe, do not rename: say “higher average mileage and lower nighttime-driving proportion,” not “safer drivers,” unless outcome evidence supports that wording.
- Check the scale: standardized means are measured in standard deviations; convert them before presenting original-unit interpretations.
- Check the denominator: a cluster’s event rate uses that cluster’s event count divided by that cluster’s evaluated size.
- Keep labels neutral: Cluster 2 is not inherently riskier than Cluster 1; cluster numbers can be permuted.
- Separate association from prediction: rate differences are outcome evidence, but their usefulness depends on evaluation, uncertainty, and the decision being made.
Check Your Understanding
For each problem, distinguish what the feature information establishes from what any outcome information establishes.
- A feature has overall mean 50 and standard deviation 10. A cluster’s standardized mean is \(-0.5\). Find its mean in original units. (Answer: 45.)
- An observation has standardized coordinates \((0.2,-0.4)\). Two centroids are \((-0.8,-0.4)\) and \((0.6,0.1)\). Using squared Euclidean distance, which centroid is nearer? (Answer: The second centroid; squared distances are 1.00 and 0.41.)
- Cluster A has 25 observations and 5 events; Cluster B has 75 observations and 15 events. Find both cluster rates and the pooled rate. Do the rates distinguish the clusters on this outcome? (Answer: Both cluster rates and the pooled rate are 0.20; these rates show no difference between the clusters.)
- In a separate evaluation sample, Cluster A has 60 observations and 9 events, while Cluster B has 40 observations and 2 events. Find the two rates, pooled rate, and A-minus-B rate difference. (Answer: A is 0.15, B is 0.05, pooled is 0.11, and the difference is 0.10, or 10 percentage points; this is an observed association, not proof of causation.)
- A fixed clustering rule is evaluated on 100 accounts. Cluster A has 50 accounts and 10 events; Cluster B has 50 accounts and 5 events. Compute the cluster rates, pooled rate, and rate ratio A-to-B. (Answer: A is 0.20, B is 0.10, pooled is 0.15, and the rate ratio is 2; the rates indicate an outcome association in this evaluation sample.)