Diagnosing Error Changes Across Risk Segments

Why Segment-Level Error Diagnosis Matters

A cluster or principal component can add useful structure to a prediction model without improving predictions equally for every record. An overall score may improve because a large segment gains substantially, even while a smaller segment experiences larger errors. For an actuarial analysis, that difference matters: average performance alone may not reveal where predictions have become less reliable.

The diagnostic task is to compare models on the same evaluation records, using the same outcome definition and evaluation weights. Then examine both aggregate errors and errors within meaningful segments. A segment may be a risk band defined in advance, or a group based on a cluster assignment or a range of a component score. If the segments come from a derived-feature mapping, use a consistent mapping for both models so that the comparison is about prediction errors, not changing group membership.

Definition: For record \(j\), let \(y_j\) be the observed outcome and \(\hat{y}_j\) the prediction. Define the signed residual as \(e_j=y_j-\hat{y}_j\). A positive residual means the prediction is too low; a negative residual means it is too high. For a segment \(g\) with \(n_g\) records, its mean signed error is \(\bar{e}_g=\sum_{j\in g}e_j/n_g\), its mean absolute error is \(\operatorname{MAE}_g=\sum_{j\in g}|e_j|/n_g\), and its root mean squared error is \(\operatorname{RMSE}_g=\sqrt{\sum_{j\in g}e_j^2/n_g}\).

Mean signed error describes direction of error, while MAE and RMSE describe its magnitude. Positive and negative residuals can cancel in a mean; they cannot cancel in MAE or RMSE. RMSE gives relatively greater influence to large residuals because it squares them. The appropriate metric depends on the prediction target and the cost of errors; a segment comparison should use the same metric as the aggregate comparison, while retaining other metrics when they reveal a different feature of performance.

How Segment Errors Combine

For unweighted records, overall mean squared error is a size-weighted average of segment mean squared errors. Overall RMSE is the square root of that weighted average, not the average of segment RMSE values. If \(n\) is the total number of records, then:

\operatorname{MSE} =\frac{\sum_g n_g\operatorname{MSE}_g}{n}, \qquad \operatorname{RMSE} =\sqrt{\frac{\sum_g n_g\operatorname{RMSE}_g^2}{n}}.

This identity explains how a model can improve overall while worsening in one segment: the gain in one or more other segments can outweigh the loss after accounting for segment sizes. It also shows why averaging the segment RMSE values gives the wrong aggregate result. For weighted evaluation, replace counts with the sum of record weights in each segment and use weighted mean squared errors.

Justification: Within segment \(g\), \(\operatorname{MSE}_g=\sum_{j\in g}e_j^2/n_g\), so \(\sum_g n_g\operatorname{MSE}_g=\sum_g\sum_{j\in g}e_j^2\). Every record belongs to exactly one segment, so the right-hand side is the sum of squared errors across all \(n\) records. Dividing by \(n\) gives overall MSE. Taking the square root gives overall RMSE. Because the square root is nonlinear, averaging segment RMSE values does not generally give overall RMSE.

Worked Example: An Overall Gain Hides a Segment-Level Loss

A hypothetical insurer compares a baseline model with a model that adds a cluster-derived predictor. On the same ten validation records, four records are in segment A and six are in segment B. Residuals are observed outcome minus prediction. The baseline residuals are \((1,1,-1,-1)\) in A and \((3,3,3,-3,-3,-3)\) in B. The augmented-model residuals are \((0,0,2,-2)\) in A and \((2,2,2,-2,-2,-2)\) in B. Determine whether aggregate RMSE improves and identify any segment that worsens.

Method. Square each residual, calculate each segment's mean squared error, then combine segment MSE values using their record counts. This avoids treating the cluster label as a performance metric in itself.

For the baseline, segment A has squared errors \(1,1,1,1\), so \(\operatorname{MSE}_A=4/4=1\) and \(\operatorname{RMSE}_A=1\). Segment B has six squared errors of \(9\), so \(\operatorname{MSE}_B=54/6=9\) and \(\operatorname{RMSE}_B=3\). Overall, \(\operatorname{MSE}=(4(1)+6(9))/10=58/10=5.8\), giving \(\operatorname{RMSE}=\sqrt{5.8}\approx2.4083\).

For the augmented model, segment A has squared errors \(0,0,4,4\), so \(\operatorname{MSE}_A=8/4=2\) and \(\operatorname{RMSE}_A=\sqrt{2}\approx1.4142\). Segment B has six squared errors of \(4\), so \(\operatorname{MSE}_B=24/6=4\) and \(\operatorname{RMSE}_B=2\). Overall, \(\operatorname{MSE}=(4(2)+6(4))/10=32/10=3.2\), giving \(\operatorname{RMSE}=\sqrt{3.2}\approx1.7889\).

Conclusion and check. Aggregate RMSE falls from about \(2.4083\) to \(1.7889\), but segment A's RMSE rises from \(1\) to about \(1.4142\). The improvement in the larger, initially less accurate segment B outweighs the deterioration in A. The common mistake is to report only the aggregate gain and conclude that every segment benefits; the segment calculations show that conclusion is not justified.

Separate Error Magnitude from Error Direction

A segment's mean signed error can reveal systematic underprediction or overprediction that a magnitude metric does not explain. Since \(e_j=y_j-\hat{y}_j\), a positive segment mean indicates average underprediction; a negative mean indicates average overprediction. An aggregate mean can conceal opposing segment biases, just as aggregate error magnitude can conceal unequal performance.

Worked Example: Opposing Segment Biases Cancel in the Aggregate

A hypothetical model has six validation records split evenly into segments X and Y. Its residuals are \(2,2,2\) in X and \(-2,-2,-2\) in Y. A revised model has residuals \(1,1,1\) in X and \(-3,-3,-3\) in Y. Compare overall and segment-level bias and error magnitude.

Original model. Segment X has mean signed error \((2+2+2)/3=2\), MAE \(=2\), and RMSE \(=\sqrt{(4+4+4)/3}=2\). Segment Y has mean signed error \((-2-2-2)/3=-2\), MAE \(=2\), and RMSE \(=2\). Across all six records, the signed errors sum to zero, so overall mean signed error is \(0/6=0\). Overall MAE and RMSE are both \(2\).

Revised model. Segment X has mean signed error \(1\), MAE \(1\), and RMSE \(1\). Segment Y has mean signed error \(-3\), MAE \(3\), and RMSE \(3\). Overall mean signed error is \((3-9)/6=-1\); overall MAE is \((3+9)/6=2\); and overall RMSE is \(\sqrt{(3(1^2)+3(3^2))/6}=\sqrt{30/6}=\sqrt{5}\approx2.2361\).

Conclusion and check. The revision improves X but worsens Y. Aggregate MAE stays at \(2\), while aggregate RMSE rises from \(2\) to about \(2.2361\), and overall signed error changes from zero to \(-1\). The original model's zero aggregate bias did not mean that either segment was unbiased: its underprediction in X and overprediction in Y cancelled exactly. The correction is to report segment bias alongside a magnitude measure.

Use the Evaluation Weights That Match the Objective

Some evaluations give every record equal weight; others use exposure or another prespecified weight. If \(w_j\) is a nonnegative evaluation weight, weighted MSE in segment \(g\) is \(\sum_{j\in g}w_je_j^2/\sum_{j\in g}w_j\). Its weighted RMSE is the square root of that quantity. The aggregate uses the same formula over all records. These formulas require a positive total weight; a segment with total weight zero has no defined weighted error measure.

The denominator matters. For instance, if the objective is error per exposure unit, a record with twice the exposure should contribute twice the weight of an otherwise comparable record. If the intended objective instead gives each policyholder equal influence, exposure weighting changes the question being answered. State the weighting rule and apply it consistently to both models and all reported segments.

Worked Example: Exposure Weighting Changes the Segment Diagnosis

A hypothetical portfolio has two segments, R and S, with three records in each. Every R record has weight 2 and every S record has weight 1. Baseline residuals are \((1,1,1)\) in R and \((3,3,3)\) in S. After adding a PCA score as a predictor, residuals are \((2,0,0)\) in R and \((2,2,2)\) in S. Calculate the weighted aggregate RMSE and weighted segment RMSE values for each model.

Method. For each segment, divide its weighted sum of squared residuals by its total weight. The total portfolio weight is \(3(2)+3(1)=9\).

For the baseline, R's weighted squared-error sum is \(2(1+1+1)=6\), with total weight \(6\). Thus its weighted MSE is \(6/6=1\) and weighted RMSE is \(1\). S's weighted squared-error sum is \(1(9+9+9)=27\), with total weight \(3\). Its weighted MSE is \(27/3=9\) and weighted RMSE is \(3\). Overall weighted MSE is \((6+27)/9=33/9\), so overall weighted RMSE is \(\sqrt{33/9}\approx1.9149\).

For the augmented model, R's weighted squared-error sum is \(2(4+0+0)=8\), giving weighted MSE \(8/6=4/3\) and weighted RMSE \(\sqrt{4/3}\approx1.1547\). S's weighted squared-error sum is \(1(4+4+4)=12\), giving weighted MSE \(12/3=4\) and weighted RMSE \(2\). Overall weighted MSE is \((8+12)/9=20/9\), so overall weighted RMSE is \(\sqrt{20/9}\approx1.4907\).

Conclusion and check. Weighted aggregate RMSE improves from about \(1.9149\) to \(1.4907\), while segment R worsens from \(1\) to about \(1.1547\). The result is consistent with the weighted aggregation identity: the weighted segment MSEs, not segment RMSEs, combine using segment weight totals. The common mistake is to average \(1\) and \(3\), or \(1.1547\) and \(2\), without accounting for the segment weights.

Interpret Segments as Diagnostics, Not Established Populations

A cluster assignment is a feature constructed from observed predictor patterns. It is not, by itself, evidence that the records form a natural or permanent population. A segment-level error difference says that the evaluated records assigned to that segment had a particular error pattern under the stated evaluation procedure. It does not establish why that pattern occurred, that the cluster caused it, or that the same pattern will hold in a different sample or period.

Use a disciplined comparison:

Fix the comparison.
Use the same evaluation records, target definition, segment assignments, weights, and metric for the baseline and augmented models.

Report segment context.
Show the number of records and, when relevant, total exposure or total evaluation weight. A large error change in a very small segment requires cautious interpretation.

Compare more than one aspect of error.
Use an error-magnitude measure such as RMSE or MAE, and inspect mean signed error when underprediction or overprediction matters.

Check the aggregate arithmetic.
Confirm that segment MSE values combine to the reported overall MSE using record counts or evaluation weights.

Describe the evidence narrowly.
State which evaluated segment improved or worsened under which metric; do not treat a derived segment label as a causal explanation or a proven population category.

The same caution applies to segments formed from PCA scores. A range of a component score can be useful for locating error patterns, but the range is an analytical partition, not automatically a meaningful group in the real world. Segment definitions should be clear enough that the reader can understand who is being compared, and conclusions should stay tied to the evaluation data and chosen metric. These checks are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Key takeaway: Overall improvement and segment-level deterioration can coexist. Compare errors on the same records, aggregate squared errors using the appropriate counts or weights, inspect signed error as well as magnitude, and interpret cluster- or PCA-based segments as diagnostic partitions rather than established populations.

Check Your Understanding

For each problem, compare the specified aggregate and segment results. Use residuals defined as observed outcome minus prediction.

A baseline model has residuals \((1,-1)\) in segment A and \((2,-2,2)\) in segment B. An augmented model has residuals \((0,3)\) in A and \((1,-1,1)\) in B. Calculate both overall RMSE values and identify the segment that worsens. (Answer: Baseline RMSE \(=\sqrt{14/5}\approx1.6733\); augmented RMSE \(=\sqrt{12/5}\approx1.5492\); overall RMSE improves, but A worsens from \(1\) to \(\sqrt{4.5}\approx2.1213\).)
A model has residuals \((2,2,-1,-1)\) in segment X and \((-3,-1,2,2)\) in segment Y. Calculate each segment's mean signed error and the overall mean signed error. (Answer: X has mean \(0.5\), Y has mean \(0\), and the overall mean is \(2/8=0.25\).)
Four records have residuals \((1,3)\) in segment P and \((2,2)\) in segment Q. Each P record has weight 2, and each Q record has weight 1. Calculate weighted segment MSEs and overall weighted RMSE. (Answer: P weighted MSE \(=(2+18)/4=5\); Q weighted MSE \(=(4+4)/2=4\); overall weighted MSE \(=(20+8)/6=14/3\), so weighted RMSE \(=\sqrt{14/3}\approx2.1602\).)
In one segment, a baseline model has residuals \((-2,2)\), while an augmented model has residuals \((1,1)\). Compare mean signed error and RMSE. (Answer: Baseline mean signed error \(0\) and RMSE \(2\); augmented mean signed error \(1\) and RMSE \(1\). RMSE improves, but average underprediction increases.)
A team reports that a cluster-based segment has higher validation RMSE than the other segments and concludes that the cluster represents a permanently higher-risk population. Give two checks or qualifications needed before making a defensible statement. (Answer: Confirm the segment's record count or total weight and that the models use the same evaluation records and fixed segment assignments; describe the observed validation error difference without claiming the cluster establishes a permanent population or causal explanation.)

What You'll Learn