Clustering as Exploratory Group Discovery
Sometimes an analysis begins without a response variable to predict. An insurer might have customer records containing several attributes but no agreed customer categories, or a health analyst might want to explore whether service locations have distinct operating profiles. Clustering is a way to group observations according to similarities in their recorded features. It can help organize a data set and suggest patterns worth investigating, but the resulting groups are not automatically real-world types, risk classes, or predictions.
The central question is not simply, “What are the clusters?” It is: Which observations were grouped together, on the basis of which information, and what conclusions does that grouping support? A cluster is an exploratory summary of the data under a chosen analysis. Its meaning depends on the observations, the features, and the choices used to define similarity.
“Unsupervised” does not mean that the analysis has no choices or assumptions. The analyst still decides which records and features to use and what sort of similarity is relevant. Different choices can produce different groupings. Those choices are addressed in detail when preparing inputs and selecting a clustering approach. For now, the key point is that a cluster is a relationship among observations under a particular analysis, not a natural category discovered with certainty.
Clusters Are Not Known Classes
A known class is a category supplied independently of the clustering process. For example, a data set might include a verified indicator of whether a policy lapsed, or a recorded category of claim type. If an analysis uses such a label as the target to be predicted, it is a supervised classification task. Clustering instead groups observations without using a known target class as the answer.
A data set can contain both features and a known outcome. An analyst may form clusters using selected features and then examine how the known outcome is distributed across those clusters. That later comparison can describe an association in the data. It does not change the original clustering into a supervised classification model, provided the outcome was not used to create the groups. Nor does an association establish that group membership caused the outcome.
| Question | Clustering | Classification |
|---|---|---|
| What is supplied as the target? | No known target class is used to form the groups | A known class or outcome is supplied as the target |
| What is produced? | An exploratory assignment of observations to groups | A rule or model for assigning observations to known classes |
| What does a group label mean? | It identifies a group in this particular result | It refers to a defined class, such as “lapsed” or “not lapsed” |
| What can be checked? | Whether the grouping is interpretable and useful for the stated exploratory purpose | How well predicted classes agree with known labels on appropriate evaluation data |
Cluster identifiers such as 1, 2, and 3 are names, not measurements. Cluster 3 is not necessarily “more” of anything than cluster 1. The identifiers can be permuted without changing which observations are grouped together. A label such as “high-service group” may be useful after examining the group’s characteristics, but it is an analyst’s description, not the original numeric cluster identifier.
Worked Example: Interpreting a Grouping of Service Locations
A hypothetical insurer groups six service locations using two recorded features: monthly claim inquiries and average handling time in minutes. The clustering result assigns Locations A, B, and C to Cluster 1, and Locations D, E, and F to Cluster 2. The values are:
| Location | Monthly inquiries | Average handling time (minutes) |
|---|---|---|
| A | 20 | 5 |
| B | 22 | 6 |
| C | 24 | 7 |
| D | 80 | 15 |
| E | 84 | 16 |
| F | 86 | 17 |
Step 1: Identify what formed the groups. The stated inputs are monthly inquiries and average handling time. No outcome such as customer satisfaction, expense, or claim severity was supplied as a target. Thus, this is an exploratory grouping of locations by the two recorded features.
Step 2: Summarize the supplied groups. For Cluster 1, the mean inquiry count is \((20+22+24)/3=66/3=22\), and the mean handling time is \((5+6+7)/3=18/3=6\) minutes. For Cluster 2, the mean inquiry count is \((80+84+86)/3=250/3=83.33\) inquiries, and the mean handling time is \((15+16+17)/3=48/3=16\) minutes. The inquiry mean for Cluster 2 is rounded to two decimal places.
Step 3: State a defensible interpretation. In this result, Cluster 1 contains locations with lower inquiry counts and shorter handling times than the locations in Cluster 2. It is reasonable to describe the groups as lower-volume, shorter-handling-time locations and higher-volume, longer-handling-time locations, respectively.
Step 4: State what does not follow. The result does not establish that higher inquiry volume causes longer calls. It does not show that Cluster 2 locations have worse service or higher costs, because those outcomes were not provided. It also does not establish that every location in a future period will belong to one of these same groups.
Method and trap avoided. The method is to interpret a supplied cluster assignment using the features that formed it, then keep conclusions within that scope. A common mistake is to turn a descriptive contrast into a causal or quality claim. The computed feature means support a contrast in these recorded features, not a claim about service quality or cause.
What Cluster Results Can and Cannot Answer
Clustering can help an analyst explore structure in unlabeled observations. It can provide a compact way to describe groups with different feature profiles, identify records that do not resemble the larger groups, and generate hypotheses for later analysis. These are exploratory uses: they help decide what to investigate, rather than proving why a pattern occurred.
The result alone cannot establish that the groups are objectively correct or that they will be stable in another data set. It cannot establish causation. It does not, by itself, predict a future response or show that acting on a group label will improve an outcome. Those claims require separate evidence and methods suited to the question. If the business question is whether a policy will lapse, for example, an exploratory grouping is not a substitute for evaluating a model that predicts lapse using known outcomes.
The grouping also reflects the information made available to the analysis. If an important feature is absent, the clusters cannot organize observations by that feature. If features encode a process or measurement convention, the clusters may partly reflect those conventions. A clear report therefore states the population being grouped, the features considered, the purpose of the exploration, and the limits of interpretation.
Why Cluster Numbers Do Not Carry Meaning
This property follows directly from what a clustering result represents. Suppose \(n\) observations are assigned to \(k\) groups. Write \(z_i\) for the group identifier assigned to observation \(i\), where \(z_i\) is one of \(1,\ldots,k\). The grouping is the collection of sets \(C_j=\{i:z_i=j\}\), one set for each group identifier \(j\).
Now choose any one-to-one relabeling \(\pi\) of the identifiers and define a new assignment \(z_i'=\pi(z_i)\). An observation is assigned to \(j\) in the original result exactly when it is assigned to \(\pi(j)\) in the relabeled result. Thus the set of observations in each group is unchanged except for its name. The partition of observations is the same, even though the numeric identifiers may differ.
The third equality holds because \(\pi\) is one-to-one: \(\pi(z_i)=\pi(j)\) if and only if \(z_i=j\). Therefore, the numbers attached to the groups do not rank the groups or describe their characteristics. Interpretation comes from examining group membership and the recorded features, not from the identifier itself.
Worked Example: A Cluster Identifier Is Not a Risk Class
A hypothetical analysis groups 100 insurance accounts using account age and number of service contacts. It assigns 60 accounts to Cluster 1 and 40 to Cluster 2. After the grouping is formed, the analyst checks a separately recorded indicator for whether an account renewed: 48 of the 60 accounts in Cluster 1 renewed, and 24 of the 40 accounts in Cluster 2 renewed. The analyst asks whether Cluster 1 can be called the “renewal class.”
Step 1: Calculate the observed renewal proportions. In Cluster 1, \(48/60=0.80\), or 80%. In Cluster 2, \(24/40=0.60\), or 60%. Across both groups there are \(48+24=72\) renewals among \(60+40=100\) accounts, so the overall renewal proportion is \(72/100=0.72\), or 72%.
Step 2: Distinguish the inputs from the later comparison. The clusters were formed using account age and service contacts. Renewal was checked afterward. The renewal proportions describe how the observed outcome varies across these two groups; they do not show that renewal was the target used to create the clusters.
Step 3: Decide what the numbers support. In this hypothetical data set, the observed renewal proportion is 20 percentage points higher in Cluster 1 than in Cluster 2, since \(0.80-0.60=0.20\). This is a descriptive association in the records. It does not show that Cluster 1 is a class defined by renewal, or that membership caused renewal.
Step 4: Check the arbitrary labels. If Cluster 1 and Cluster 2 were swapped, the membership and the two proportions would stay the same, but the group numbers would reverse. The label “Cluster 1” therefore cannot itself mean “renewed.” Calling it the renewal class would confuse an exploratory group with a known outcome class.
Method and trap avoided. The method is to calculate outcome summaries within groups while preserving the distinction between the grouping inputs and the separately observed outcome. The trap is treating a higher observed outcome rate as proof that a cluster is a supervised class or that it predicts individual outcomes.
Using Clustering Responsibly
In actuarial work, an exploratory grouping may help organize a portfolio, summarize operational patterns, or identify questions for subsequent analysis. Before using a result, state what it is intended to support. A group description may help stakeholders discuss patterns in the data, but decisions about pricing, underwriting, claims, or customer treatment need additional validation and consideration of applicable constraints.
In particular, clustering does not automatically make a decision fair or appropriate. A group can reflect historical practices, uneven data quality, or features that act as proxies for sensitive characteristics. If cluster membership is proposed for a consequential decision, the analyst should examine how the grouping was formed, who may be affected, and whether the proposed use is justified. Exploratory usefulness is not, by itself, evidence of fairness or suitability for action.
A clear communication distinguishes three statements: what the grouping did, what patterns appear in the observed features, and what remains unknown. For example, “These locations were grouped using inquiry volume and handling time; the two groups have different averages on those features” is a descriptive statement. “The grouping identifies poor service locations” adds a judgment not established by those facts. The careful distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Worked Example: Choosing a Defensible Business Conclusion
A hypothetical claims team groups 50 claim offices using only the monthly number of open claims and the average age of open claims. One resulting group contains 15 offices. Its average number of open claims is 120, compared with 70 in the other 35 offices. A manager proposes: “The 15 offices are inefficient, so reducing their staffing should improve performance.” Evaluate the statement using only the information supplied.
Step 1: Identify the supported comparison. The group contains 15 of the 50 offices, so its share of the offices is \(15/50=0.30\), or 30%. Its reported average open-claim count is 120, compared with 70 for the other group. The difference is \(120-70=50\) open claims per office on average.
Step 2: Describe the result within scope. The clustering used open-claim count and average claim age. The stated summary supports saying that the 15-office group has a higher average open-claim count than the other group. The given information does not provide the groups’ average claim ages, so no numerical comparison on that feature can be calculated here.
Step 3: Identify the unsupported parts. “Inefficient” is not established by open-claim counts alone. The information does not state office staffing, claim complexity, incoming workload, service standards, or an efficiency measure. It also provides no comparison of outcomes before and after a staffing change. Therefore, it cannot show that reducing staffing would improve performance.
Step 4: Replace the conclusion with a defensible next question. A suitable conclusion is: “This exploratory grouping identifies 15 offices with a higher average open-claim count in the supplied data. Further analysis is needed to assess workload, staffing, claim complexity, and service outcomes before considering an operational change.”
Method and trap avoided. The method is to report the supported feature contrast, identify the missing evidence, and phrase a next-step question without asserting cause. The trap is treating a cluster profile as a performance measure and then treating that measure as proof of a policy’s effect.
Common Mistakes
- Calling a cluster a known class. A cluster is formed from similarity in selected features; a class is a known category. They are not interchangeable.
- Reading an ordering into cluster numbers. Cluster 2 is not necessarily larger, riskier, or more important than Cluster 1. The identifiers can be swapped without changing the grouping.
- Claiming causation from a profile. Differences between groups are descriptive unless a separate design and analysis support a causal conclusion.
- Assuming the grouping is universally true. A cluster result depends on the records, features, and analytical choices. It is not a permanent or objective taxonomy merely because software returns it.
- Using exploratory groups as predictions without evaluation. A group’s observed outcome rate does not, by itself, establish predictive performance for future observations.
Check Your Understanding
For each problem, distinguish what the grouping establishes from what would require additional evidence.
- A hypothetical analyst groups 30 repair depots without using a known outcome. Later, the analyst finds that 8 of 10 depots in one group met a service target. Is that group a known service-target class? State the observed proportion and explain. (Answer: No. The observed proportion is \(8/10=0.80\), or 80%; it is an outcome summary for an exploratory group, not a class used to form it.)
- A clustering output gives group identifiers 1 and 2. An analyst swaps every 1 and 2 while leaving memberships otherwise unchanged. Has the underlying partition changed? (Answer: No. Only the identifiers have been relabeled; the same observations remain grouped together.)
- A hypothetical insurer groups 40 regions using claim frequency. Group A has 12 regions with a mean of 25 claims per region; Group B has 28 regions with a mean of 10 claims per region. By how much does the mean differ, and does this establish that Group A is less efficient? (Answer: The difference is \(25-10=15\) claims per region. No; claim frequency alone does not establish efficiency.)
- A health analyst forms clusters using age and recorded service use, then observes that one cluster has a higher hospitalization proportion. Can the analyst conclude that cluster membership caused hospitalization? (Answer: No. The comparison is an observed association; clustering alone does not establish causation.)
- A manager wants to use an exploratory group formed from customer contact counts to decide which customers should receive a price increase. Name two questions that should be answered before treating the group as a basis for that action. (Answer: For example, whether the grouping is appropriate and stable for the intended use, and whether the features or resulting decisions create fairness or other suitability concerns. The exploratory grouping alone does not justify the price action.)