Evaluate Risk Segments With PCA And Clustering

From Risk Profiles to Defensible Segments

A portfolio may contain many measured characteristics: annual mileage, vehicle age, prior incidents, or a building’s age and construction features. Risk segmentation seeks groups of observations with similar predictor profiles. Principal component analysis (PCA) can summarize correlated predictors, while clustering can group observations in the resulting feature space. Neither method, by itself, establishes that the groups predict future loss.

A defensible analysis therefore has distinct stages: define and scale the predictors, inspect the PCA summary, compare plausible clusterings, and then describe the resulting groups in data not used to construct them. The final step tests whether a description carries over to held-out observations. It does not turn an unsupervised segmentation into a validated prediction model.

Definition: PCA forms new, uncorrelated linear combinations of centered predictors, ordered by the amount of predictor variation they explain. Clustering partitions observations into groups according to a chosen measure of similarity. In risk segmentation, the predictors define the groups; an outcome such as future claim cost should not be used to construct the segments if it is later to be compared across them.

Scale Predictors Before Comparing Profiles

A distance-based method can be dominated by a predictor whose units or numerical range are large. For predictor $j$, standardize an observation $x_{ij}$ using a mean $\bar{x}_j$ and standard deviation $s_j$ estimated from the analysis sample:

z_{ij}=\frac{x_{ij}-\bar{x}_j}{s_j}.

Each standardized predictor has mean zero and standard deviation one in the sample used to calculate those quantities. This makes a one-standard-deviation difference comparable across predictors; it does not imply that the predictors have equal practical importance. For later observations, keep the training means and standard deviations fixed. Recomputing them using held-out data changes the coordinate system and allows information from that data into the analysis.

A predictor with zero standard deviation cannot be standardized by this formula and contributes no variation in that sample; remove it from the PCA and distance calculation. Scaling is also a modeling decision: correlation-based PCA standardizes predictors, whereas covariance-based PCA uses centered values in their original units. When predictors have different units or ranges, correlation-based PCA is often the more interpretable starting point.

Worked Example: Standardizing a Hypothetical Risk Profile

A hypothetical motor portfolio uses annual mileage and prior-incident count as predictors. In the training sample, mileage has mean 12,000 miles and standard deviation 4,000 miles; incident count has mean 1 and standard deviation 1. A new profile has 16,000 miles and 2 incidents. Find its standardized coordinates and compare its distance from the training mean before and after scaling. The figures are illustrative, not company data.

Method. Apply the training-sample standardization formula separately to each predictor. The unscaled difference from the mean is 4,000 miles and 1 incident, so its raw Euclidean distance is:

\sqrt{(16{,}000-12{,}000)^2+(2-1)^2} =\sqrt{16{,}000{,}001} \approx 4{,}000.0001.

The standardized mileage is $(16{,}000-12{,}000)/4{,}000=1$, and the standardized incident count is $(2-1)/1=1$. The standardized distance is therefore:

\sqrt{1^2+1^2}=\sqrt{2}\approx 1.4142.

The raw distance is numerically almost entirely determined by mileage’s units; the standardized distance expresses the profile as one standard deviation above the mean on each predictor. Common mistake avoided: treating the raw distance as evidence that mileage is thousands of times more important. This example uses the training means and standard deviations, as a held-out profile must.

PCA: Directions of Variation and Their Scores

Let $Z$ be the matrix of centered, scaled predictors and let $R$ be their sample correlation matrix. A principal component is a linear combination $Z a_k$, where the loading vector $a_k$ has length one. The first component is the direction with the largest sample variance; each later component has the largest remaining variance subject to being orthogonal to the earlier directions.

Key result: The loading vectors $a_k$ are the eigenvectors of $R$, and their eigenvalues $\lambda_k$ are the variances of the corresponding component scores. The proportion of total standardized predictor variation explained by component $k$ is $\lambda_k/\sum_j\lambda_j$. For $p$ standardized predictors, $\sum_j\lambda_j=p$.

This result follows from maximizing the variance of a unit-length linear combination. For any vector $a$ with $a'a=1$, the variance of $Za$ is $a'Ra$. With a constraint that $a$ has unit length, the Lagrange condition for maximizing this quantity is $Ra=\lambda a$. Thus a maximizing direction is an eigenvector, and its variance is its eigenvalue. Repeating the maximization in directions orthogonal to those already chosen gives the remaining principal components.

A large eigenvalue means that the component captures substantial variation among the predictors; it does not mean that it captures variation in the outcome of interest. Loadings indicate how the original predictors contribute to a component. Their signs are arbitrary: changing every sign in an eigenvector changes the score signs but not the PCA solution. A component may be difficult to interpret when its loadings combine predictors in ways that do not describe a clear profile.

Worked Example: Reading a Two-Predictor PCA Summary

For a hypothetical training portfolio, suppose the correlation matrix of two standardized risk predictors is:

R= \begin{pmatrix} 1 & 0.8\\ 0.8 & 1 \end{pmatrix}.

Find the principal directions, the variance explained by each, and the component scores for a profile with standardized coordinates $z=(1,0)$. Interpret the first component cautiously.

Method. The eigenvectors proportional to $(1,1)$ and $(1,-1)$ are perpendicular. Normalizing each to length one gives $a_1=(1,1)/\sqrt{2}$ and $a_2=(1,-1)/\sqrt{2}$. Multiplication by $R$ gives:

Ra_1=1.8a_1,\qquad Ra_2=0.2a_2.

The eigenvalues are 1.8 and 0.2. Their sum is 2, as required for two standardized predictors. The first component explains $1.8/2=0.90$, or 90%, of their total standardized variation; the second explains $0.2/2=0.10$, or 10%.

The profile’s first score is $z'a_1=(1+0)/\sqrt{2}=0.7071$. Its second score is $z'a_2=(1-0)/\sqrt{2}=0.7071$. Thus the profile has a positive score on both the shared-variation direction and the contrast direction. The first component represents a direction in which both predictors rise together, consistent with their positive correlation. It does not establish that either direction is associated with future losses. Check: the two component variances sum to 2, matching the trace of $R$.

Clustering and the Choice of Geometry

After choosing the predictor representation—scaled original variables, selected component scores, or both—choose a clustering method and a distance measure. K-means partitions observations into a prespecified number $K$ of groups to minimize the sum of squared Euclidean distances to each group’s centroid. It is sensitive to scale and can be sensitive to initialization; a reported solution should be checked for stability across starts. Its centroid-based objective tends to favor compact groups.

Hierarchical clustering builds a sequence of nested groupings. In agglomerative clustering, each observation begins alone and groups are successively merged according to a linkage rule. Single linkage uses the closest pair of observations between groups; complete linkage uses the farthest pair; average linkage uses the average pairwise distance. Cutting the resulting hierarchy at a chosen level produces a partition. Different linkage rules can produce different group shapes, and single linkage can create long, chained groups.

PCA and clustering answer different questions. PCA summarizes directions of variation; clustering divides observations into groups. Using only leading components can discard lower-variance directions that separate groups. Conversely, retaining many components can preserve noise. Compare a clustering on scaled predictors with one on a defensible PCA representation, and explain what was retained or discarded. Cluster labels such as “1” and “2” have no inherent order or risk meaning.

Worked Example: Comparing K-Means With Single Linkage

Suppose four observations have one retained component score, respectively $0, 2, 4,$ and $7$. Compare a two-cluster k-means solution with a two-cluster single-linkage cut. Use squared Euclidean distance for k-means and ordinary distance for linkage.

K-means method. In one dimension, a globally optimal partition can be checked by considering the three possible splits between ordered values. Splitting after 0 gives a singleton with within-group sum of squares 0 and a group $\{2,4,7\}$ with mean $13/3$. Its sum of squares is:

$$ (2-13/3)^2+(4-13/3)^2+(7-13/3)^2 =49/9+1/9+64/9=38/3. $$

Splitting after 2 gives groups $\{0,2\}$ and $\{4,7\}$. Their means are 1 and 5.5, and their sums of squares are $(0-1)^2+(2-1)^2=2$ and $(4-5.5)^2+(7-5.5)^2=4.5$, for a total of 6.5. Splitting after 4 gives $\{0,2,4\}$, whose mean is 2 and whose sum of squares is $4+0+4=8$, plus singleton $\{7\}$ with sum of squares 0. Thus the minimum is 6.5, with groups $\{0,2\}$ and $\{4,7\}$.

Single-linkage method. The successive nearest gaps are $2$ between 0 and 2, $2$ between 2 and 4, and $3$ between 4 and 7. After the first merge, the distance from $\{0,2\}$ to 4 is 2, so the next merge makes $\{0,2,4\}$. Cutting the hierarchy to leave two groups gives $\{0,2,4\}$ and $\{7\}$. The partitions differ because the linkage procedure can join a chain of nearby points, whereas k-means minimizes within-group squared distances. Common mistake avoided: claiming the methods must agree because they used the same scores. The partition depends on the clustering objective, not just the input coordinates.

Check Whether the Description Carries Over

A segment is constructed from predictor profiles, so evaluating it on the same observations can make the groups look more distinct than they will elsewhere. A useful descriptive check holds out observations before fitting any transformations or clusters. Estimate scaling parameters, PCA loadings, the chosen representation, and cluster centers using the training data only. For k-means, assign each held-out observation to its nearest frozen center under the same scaled representation. For hierarchical clustering, an out-of-sample assignment rule must be specified; a hierarchy alone does not automatically define one.

Then summarize held-out characteristics by assigned segment: counts, predictor summaries, or outcomes not used to form the groups. A difference in held-out average loss is a descriptive observation, not proof that the segments predict future loss or that the difference is statistically reliable. Strong predictive claims require an appropriate supervised evaluation, including a defined target, comparison method, and performance assessment; in-sample cluster separation is not that assessment.

Worked Example: Describing Frozen Segments in Held-Out Data

A hypothetical training sample has produced two k-means centers in the scaled two-predictor space: $A=(-1,-1)$ and $B=(1,1)$. The training means, standard deviations, PCA decisions if used, and centers are now frozen. Four held-out records have scaled predictor coordinates and observed claim payments as shown below. Assign each record to its nearest center and calculate the mean payment in each assigned group.

Record	Scaled coordinates	Payment
1	(-0.5, -1.0)	$200
2	(0.8, 1.0)	$600
3	(-1.2, -0.7)	$400
4	(1.1, 0.6)	$800

Method. Compare squared Euclidean distances; taking square roots would not change which center is nearer. Record 1 has squared distances to $A$ and $B$ of $(0.5)^2+0^2=0.25$ and $(1.5)^2+2^2=6.25$, so it joins $A$. Record 2 has distances $1.8^2+2^2=7.24$ and $0.2^2+0^2=0.04$, so it joins $B$. Record 3 has distances $0.2^2+0.3^2=0.13$ and $2.2^2+1.7^2=7.73$, so it joins $A$. Record 4 has distances $2.1^2+1.6^2=6.97$ and $0.1^2+0.4^2=0.17$, so it joins $B$.

Group $A$ contains records 1 and 3; its mean held-out payment is $(200+400)/2=\$300$. Group $B$ contains records 2 and 4; its mean is $(600+800)/2=\$700$. The observed descriptive difference is $\$700-\$300=\$400$. This small example checks how a frozen segmentation describes these four held-out records. It does not show that the segmentation predicts future payment, establish a causal explanation, or quantify uncertainty. Check: each record was assigned using predictors and centers only, not its payment.

Common Traps in an Integrated Segmentation Case

Clustering on incompatible raw scales. For example, a 4,000-mile difference and a one-incident difference give a raw distance of about 4,000, even when both are one standard deviation. Standardize where scale comparability is appropriate.
Re-estimating preprocessing on held-out records. This changes the fitted coordinate system. Apply the training-sample means, standard deviations, and PCA loadings unchanged.
Choosing components solely by explained variance. Explained variance concerns predictor variation, not outcome relevance or cluster quality. Examine whether discarded directions matter to the segmentation question.
Calling an in-sample outcome gap predictive validation. If outcomes were not used to construct clusters, an in-sample gap is still only a description of those same records. Use held-out data for a separate descriptive check, and do not overstate its evidence.
Giving cluster labels substantive meaning automatically. Describe groups using their predictor profiles and the specified outcome summaries; a cluster number is only an arbitrary label.

Key takeaway: A careful risk-segmentation analysis states its scaling and PCA choices, compares clustering methods that fit the feature geometry, and applies frozen rules to held-out observations. Held-out group summaries can show whether a description carries over; they do not, on their own, establish predictive value.

Check Your Understanding

Work through each question, keeping the distinction between predictor construction and outcome description explicit.

A training sample gives a predictor mean of 50 and standard deviation 10. A held-out observation has value 65. What standardized value should be used if the training transformation is frozen? (Answer: 1.5)
A correlation-based PCA has three predictors and eigenvalues 2.1, 0.6, and 0.3. What fraction of total standardized predictor variation does the first component explain? (Answer: 0.7000, or 70%)
Two one-dimensional component scores are 2 and 6. If they form a k-means cluster with two members, what is the centroid and the within-cluster sum of squares? (Answer: centroid 4; within-cluster sum of squares 8)
Two cluster centers are $(0,0)$ and $(2,0)$. A held-out observation has coordinates $(1.2,0.5)$. Using squared Euclidean distance, which center is nearer? (Answer: $(2,0)$; squared distances are 1.69 and 0.89)
Frozen segments have held-out payments of $100 and $300 in group A, and $500, $700, and $900 in group B. Find each group mean and their difference. What conclusion is justified? (Answer: group A mean $200; group B mean $700; difference $500. This is a held-out descriptive difference only, not proof of predictive value.)

What You'll Learn