Start With the Grouping Question
Clustering groups observations according to patterns in selected variables. That makes choosing the variables a substantive decision, not merely a step in preparing a data frame. If the columns do not describe the differences relevant to the purpose, the resulting groups may be mathematically well formed but answer the wrong question.
Begin by stating what one row represents and what kind of groups would be useful. A row might represent a policyholder, a claim, a clinic or a month of operations. The same column can be appropriate for one question and misleading for another. For example, location may be central when studying geographic service patterns but an unwanted source of separation when the goal is to compare operating profiles independent of geography.
A clustering analysis does not have a response variable in the same sense as a supervised prediction model. Nevertheless, an outcome may be present in the source data, perhaps because it will later be used to profile or evaluate the groups. Its presence in the data frame does not make it a suitable clustering input. The selection should follow the question and the information that would legitimately be available for that question.
A Structured Review of Candidate Columns
Before selecting variables, inspect both their recorded values and their meaning. Column names alone are not enough: a field called “status” could describe a meaningful operating condition, a future outcome, or an internal processing step. For each candidate, ask whether it describes the observation, whether it is measured at the relevant time, and whether the difference it represents is intended to shape the groups.
Write a sentence such as, “Each row is one clinic, and the goal is to identify clinics with similar service and capacity profiles.” This anchors every later inclusion decision.
Check for row numbers, customer keys, claim numbers and other fields whose purpose is to identify records. A distinct-value count can flag them for review, but meaning—not the count alone—determines whether a column is an identifier.
Identify results that occur after the intended observation time, such as a later renewal decision or final claim payment. Decide whether the task is exploratory grouping or instead explicitly organized around that outcome.
Consider reporting date, operating region, recording channel, data-system changes and other fields that might make groups reflect how or where records were collected rather than the intended subject.
For each field, note whether it is included, excluded or retained for a deliberate comparison, and state the reason. This makes the analysis auditable and easier to explain.
A simple diagnostic is the number of distinct observed values. If a data set has \(n\) rows and a column has \(U\) distinct values, its distinct-value proportion is \(U/n\). A value near 1 may indicate a unique record key, especially when the field is named “ID” or “number.” It is a warning to investigate, not an automatic exclusion rule: a genuine measurement can also be unique in a small sample.
In R, a first inspection can include the column names, variable types, a compact summary and the number of distinct values in each column. These checks help reveal unexpected codes, near-unique fields or timing variables. They do not establish what a field means; that requires knowledge of how and when it was recorded.
Why Identifiers and Outcomes Need Special Treatment
An identifier distinguishes records, but distinction is not the same as useful similarity information. For instance, sequential policy numbers may reflect the order in which records were created. If supplied as a clustering input, those numbers can influence the grouping even though being close in sequence does not mean two policyholders have similar risk or service needs. The method has no way to know that a column is “only an ID” unless it is left out.
Outcomes require a different caution. Suppose the purpose is to discover profiles among accounts before their renewal decisions are known. If renewal status is included, the groups are formed using a field that directly states that later result. It then becomes misleading to present the resulting groups as an outcome-independent discovery of account profiles.
The reasoning is direct. Let \(X\) be the selected predictor columns and let a clustering procedure produce groups \(C=A(X)\). If a known outcome \(Y\) is appended, the input becomes \([X,Y]\), so the result is \(A([X,Y])\). This is a different analysis: its groups may depend on \(Y\), and there is no general rule that they must match \(A(X)\). Thus, when the goal is to discover profiles without using the outcome, leave \(Y\) out of the clustering inputs and examine it separately afterward if appropriate.
Time, location, demographic attributes and process fields are not automatically inappropriate. Their role depends on the stated purpose. A regional service-planning analysis may intentionally include region; a comparison intended to find service profiles independent of region may not. Likewise, demographic variables can be relevant to a legitimate analysis, but their use calls for an explicit rationale and attention to fairness, privacy and potential proxies for protected or disadvantaged groups. Excluding a sensitive field does not guarantee that other fields cannot encode similar differences.
Worked Example: Selecting Inputs for Account Profiles
A hypothetical insurer has six account records. The purpose is to identify account profiles using information available just before renewal, not to group accounts by the renewal result. The data contain account number, years with the insurer, annual premium, prior-year claim count and next renewal status. The distinct counts are 6, 5, 4, 3 and 2, respectively. Select appropriate candidate inputs and explain the exclusions.
Method. Define the time point and purpose first, then check whether each field describes an account at that time or labels a record or later result.
Step 1: Check the identifier. Account number has \(U=6\) distinct values among \(n=6\) records, so its distinct-value proportion is \(6/6=1.000\). Its purpose is to identify records, not describe account profiles. Exclude it.
Step 2: Check the potential profile variables. Years with the insurer, annual premium and prior-year claim count are all available before the renewal decision. If the intended profiles concern tenure, price and prior claims, these are defensible candidate inputs. Their distinct-value proportions are \(5/6=0.8333\), \(4/6=0.6667\) and \(3/6=0.5000\). None is automatically an identifier; the values and subject matter support their use as profile characteristics.
Step 3: Check the outcome. Next renewal status has two distinct values, but the key issue is not its low distinct count. It is a later result that the analysis explicitly aims not to use when forming profiles. Exclude it from the inputs and, if relevant, inspect renewal status separately after grouping.
Decision and check. Use years with the insurer, annual premium and prior-year claim count as candidate inputs; exclude account number and next renewal status. The decision matches the stated time point and prevents both a record label and the later result from defining the profiles. The common mistake is to include every column except the account number, which would let renewal status shape an analysis intended to be independent of renewal.
Worked Example: Avoiding a Reporting-Time Grouping
A hypothetical health network wants to group 12 clinics by operating capacity and service volume. Each row is one clinic. Candidate fields include clinic code (12 distinct values), staffed beds (5 distinct values), annual visits (8 distinct values), region (3 distinct values) and reporting month (4 distinct values, with three clinics recorded in each month). The reporting months differ because data were extracted on staggered dates; the goal is not to group clinics by extraction schedule. Decide which fields to retain as candidates.
Method. Use the purpose to distinguish operational characteristics from record labels and collection circumstances.
Step 1: Remove the clinic code. Its distinct-value proportion is \(12/12=1.000\), and its function is identifying each clinic. It does not describe operating capacity or volume, so exclude it.
Step 2: Retain the operational candidates. Staffed beds and annual visits directly describe the capacity and volume named in the question. Their distinct-value proportions are \(5/12=0.4167\) and \(8/12=0.6667\). These values do not prove relevance by themselves; their operational meanings do.
Step 3: Assess region against the stated goal. Region has three values, so each region contains some of the 12 clinics. If the purpose were to discover geographic service types, region might be intentional. Here, the question is about capacity and service volume, and geography is not part of that definition. Exclude region for this analysis, while documenting that choice.
Step 4: Exclude the reporting month. Its distinct-value proportion is \(4/12=0.3333\), with three clinics in each month. The month records staggered extraction timing, not an intended clinic characteristic. Including it could make groups partly reflect data collection circumstances. Exclude it.
Decision and check. Staffed beds and annual visits are the candidate inputs; clinic code, region and reporting month are excluded for the stated purpose. The common mistake is to treat every non-unique field as meaningful. Reporting month is not unique, but it still records an unintended difference.
Worked Example: Keeping an Intake Analysis Separate From Claim Outcomes
A hypothetical claims team wants exploratory groups of newly reported claims for planning intake resources. There are 10 claims. Candidate fields are claim number (10 distinct values), injury category (4), initial severity band (3), reporting channel (2), final paid amount (7) and later fraud-referral decision (2). The team wants groups based on circumstances known at intake, not final outcomes or internal process decisions. Which fields should be candidate inputs?
Method. Apply the observation-time rule: a candidate must describe the claim as known at intake and be relevant to the stated intake-planning purpose.
Step 1: Exclude the claim number. It has \(10/10=1.000\) distinct values and identifies records. It is not a claim circumstance.
Step 2: Assess intake characteristics. Injury category and initial severity band are available at intake and describe the reported claim. Reporting channel is also available then, but the team’s stated goal is to group by claim circumstances, not by the way the organization received the report. For this purpose, retain injury category and initial severity band as candidates and exclude reporting channel. If channel were part of a separate analysis of intake operations, the decision could differ.
Step 3: Exclude later fields. Final paid amount is determined after claim development, and the fraud-referral decision is a later internal process result. Neither is part of the information set the team intends to use at intake. Their distinct counts—\(7/10=0.700\) and \(2/10=0.200\)—do not make them appropriate inputs.
Decision and check. Candidate inputs are injury category and initial severity band. Exclude claim number, reporting channel, final paid amount and later fraud-referral decision, with the reasons recorded. The common mistake is to retain final paid amount because it is informative about claim size. It may be informative for a different question, but it is not available at the defined intake time and would change this analysis’s purpose.
Make the Selection Defensible
A good variable-selection decision can be explained in plain language: “We included these fields because they describe the intended profile at the specified time; we excluded these fields because they identify records, occur later, or represent variation outside the question.” This explanation is more useful than a rule such as “keep every numeric column” or “remove every field with many values.”
- Use the unit consistently. A row-level identifier may be essential for joining or auditing records while still being inappropriate as a clustering input.
- Do not confuse prediction with grouping. A column can be useful for predicting an outcome yet inappropriate when the goal is outcome-independent exploratory grouping.
- Do not infer meaning from a distinct count alone. A low-cardinality field can encode a consequential process difference, while a high-cardinality measurement can be substantively meaningful.
- Make intentional differences explicit. If location, time or demographic attributes are part of the research question, state that choice and its implications rather than treating those variables as neutral by default.
- Keep selection distinct from preparation. Once the candidate fields have been chosen, their data types, missingness and scales still require attention before a clustering method can use them. Those are separate preparation decisions.
For an actuarial analysis, this discipline supports interpretable findings and reduces the risk of presenting administrative artifacts or future results as meaningful segments. The variable-selection rationale should be aligned with the SOA Exam PA (Predictive Analytics) syllabus: define the question, use the data deliberately, and communicate what the analysis can and cannot establish.
Check Your Understanding
For each situation, decide which fields are suitable candidate inputs for the stated clustering purpose and explain the key exclusion. Answers are provided for checking.
- A hypothetical portfolio has 20 accounts. Account key has 20 distinct values, tenure has 6, service contacts in the prior year has 5, and a later cancellation indicator has 2. The goal is to explore account profiles before cancellation. Which fields are candidate inputs? (Answer: Tenure and prior-year service contacts; exclude the account key as an identifier and the later cancellation indicator as an outcome.)
- A team wants to group 15 service centers by staffing and workload. Center code has 15 distinct values, staffing level has 5, workload has 9, and extraction month has 5, with three centers recorded per month. Extraction month reflects staggered data collection and is not part of the goal. Which fields should be candidates? (Answer: Staffing level and workload; exclude center code and extraction month.)
- A study aims to group hypothetical claims by conditions known when reported. For 30 claims, claim ID has 30 distinct values, reported cause has 4, initial reserve band has 5, final settlement amount has 18, and later litigation status has 2. Which fields are candidates, and why? (Answer: Reported cause and initial reserve band; exclude claim ID as an identifier and final settlement amount and litigation status because they are later outcomes.)
- A hypothetical analysis seeks geographic patterns in service access across 24 clinics. Region has 4 values, annual visits has 10, and clinic ID has 24 distinct values. Is region automatically inappropriate because it could create geographic differences? (Answer: No. Region is intentional for a geographic-pattern question; annual visits may also be relevant if the purpose supports it, while clinic ID remains an identifier and should be excluded.)
- A customer-profile analysis excludes a protected demographic field but includes postal area and product type, both of which may reflect demographic or access differences. Is the exclusion alone enough to conclude the inputs are free from unintended group differences? (Answer: No. Other fields can act as proxies, so their meaning and potential fairness implications must also be reviewed.)