Why Diagnose Errors by Segment?
A model can have an acceptable portfolio-level error and still perform poorly for a particular part of the portfolio. For example, many small errors in one territory can outweigh a few large errors in another when errors are averaged across all policies. Conversely, a striking result in a very small risk segment may be driven by only a few observations. Looking at validation errors by segment helps reveal these differences, but a segment result needs its sample size and context to be interpreted correctly.
The analysis below assumes predictions have already been evaluated on validation policies that were not used to fit the model. Each policy contributes one observed target and one prediction, measured on the same scale. Examples use hypothetical claim costs stated in thousands of dollars. The approach applies to other numerical targets, provided the error measure matches the target and the decision being considered.
RMSE gives more influence to large errors because it squares each residual before averaging. Mean residuals can cancel: predictions that are too low and too high by similar amounts may produce a mean close to zero even when individual errors are large. Use the two summaries to answer different questions. RMSE describes error magnitude; mean residual indicates the direction of average error.
How Segment Results Reconcile to Overall RMSE
Suppose the segments form a non-overlapping partition of the validation policies, so every policy belongs to exactly one segment. Let \(\operatorname{SSE}_g=\sum_{i\in g}e_i^2\), the sum of squared errors in segment \(g\). Adding the segment sums of squared errors gives the portfolio sum of squared errors. The overall RMSE therefore follows from the segment RMSEs as shown below.
The second equality follows directly from the definition: \(\operatorname{RMSE}_g^2=\operatorname{SSE}_g/n_g\), so \(\operatorname{SSE}_g=n_g\operatorname{RMSE}_g^2\). Summing this identity across segments and dividing by the total number of policies gives the overall mean squared error; taking its square root gives overall RMSE. Thus, it is the squared RMSE that averages across segments, weighted by segment size. RMSE itself is not the size-weighted average of segment RMSEs.
Worked Example: Reconciling Territory RMSEs
A hypothetical insurer evaluates claim-cost predictions on five validation policies. Costs and residuals are in thousands of dollars. In the East territory, the residuals are \(+6,-6,+6\). In the West territory, they are \(+20,-20\). Calculate each territory’s RMSE and the overall RMSE. State what the overall result does and does not show.
Method. Calculate the sum of squared residuals and the number of policies in each territory, then pool the squared errors before taking the square root.
For East, the squared residuals are \(6^2=36\), \((-6)^2=36\), and \(6^2=36\). Thus \(\operatorname{SSE}_{E}=36+36+36=108\), with \(n_E=3\). The mean squared error is \(108/3=36\), so \(\operatorname{RMSE}_{E}=\sqrt{36}=6\) thousand dollars.
For West, the squared residuals are \(20^2=400\) and \((-20)^2=400\). Thus \(\operatorname{SSE}_{W}=400+400=800\), with \(n_W=2\). The mean squared error is \(800/2=400\), so \(\operatorname{RMSE}_{W}=\sqrt{400}=20\) thousand dollars.
Across all policies, \(\operatorname{SSE}_{\mathrm{all}}=108+800=908\) and \(n_{\mathrm{all}}=3+2=5\). Therefore,
The portfolio RMSE is approximately \(13.476\) thousand dollars. The West RMSE is much higher in this validation sample, but the overall value alone hides that contrast. Common mistake: taking the unweighted average \((6+20)/2=13\) as overall RMSE. This is incorrect because it gives each territory equal weight regardless of its number of policies and averages RMSE rather than mean squared error. Pooling the squared errors gives the correct value, approximately \(13.476\). This small example signals a diagnostic difference; it does not establish that the same difference will persist in a larger portfolio.
Separate Error Magnitude from Direction
Segmented RMSE is useful for identifying where large errors occur, but it does not indicate whether predictions tend to be too high or too low. The mean residual supplies that directional information. It is important to report both when systematic underprediction or overprediction matters. A near-zero mean residual is not evidence of accurate predictions if positive and negative residuals offset one another.
Worked Example: Large Errors in a Small Risk Segment
In a hypothetical validation sample, a low-risk segment contains 100 policies, each with a residual of \(+2\) thousand dollars. A high-risk segment contains four policies with residuals \(+20,-20,+20,-20\) thousand dollars. Calculate both segment RMSEs, the portfolio RMSE, and the high-risk segment’s mean residual. Then explain why its RMSE deserves cautious interpretation.
Method. Use squared residuals for RMSE and signed residuals for mean residual. Compare segment sizes as well as the resulting error measures.
In the low-risk segment, each squared residual is \(2^2=4\). The sum of squared errors is \(100(4)=400\), so \(\operatorname{RMSE}_{L}=\sqrt{400/100}=\sqrt{4}=2\) thousand dollars.
In the high-risk segment, each squared residual is \(20^2=400\), including the negative residuals because squaring removes the sign. Its sum of squared errors is \(4(400)=1{,}600\), so \(\operatorname{RMSE}_{H}=\sqrt{1{,}600/4}=\sqrt{400}=20\) thousand dollars. Its mean residual is \((20-20+20-20)/4=0\) thousand dollars. That zero indicates no average signed error in these four observations, not small individual errors.
For the full sample, the sum of squared errors is \(400+1{,}600=2{,}000\), and the number of policies is \(100+4=104\). Hence,
The portfolio RMSE is approximately \(4.385\) thousand dollars, much closer to the low-risk RMSE than to the high-risk RMSE because the low-risk segment contains most of the policies. The high-risk result is based on only four policies, so it could be sensitive to the particular cases in this sample. Do not dismiss it: the errors are large and may warrant investigation. But do not treat this estimate alone as a stable description of all high-risk policies. Common mistake: reporting the high-risk mean residual of zero as proof that predictions are accurate. The correction is to examine signed bias and error magnitude separately; here the mean residual is zero while RMSE is \(20\) thousand dollars.
Look at Intersections Without Losing Perspective
A territory summary and a risk summary each combine policies across the other characteristic. As a result, both marginal summaries can hide a pattern that appears only when territory and risk are considered together. Cross-classification can help detect that pattern. It also creates smaller groups, so each cell’s policy count must be reported alongside its error.
For a partition into territory-by-risk cells, calculate the error measure in each cell. Then aggregate cells to check territory totals, risk totals, and the portfolio total by summing their squared errors and counts. When examining separate segmentations, such as territory alone and risk alone, do not add those counts together: the same policies appear in both analyses.
Worked Example: Finding a Pattern in Territory-by-Risk Cells
A hypothetical validation set is split into North and South territories and low- and high-risk groups. The validation summaries below give the number of policies and sum of squared residuals in each cell. Claim costs and residuals are in thousands of dollars. Calculate each cell RMSE, the two territory RMSEs, the two risk-group RMSEs, and the overall RMSE. Interpret the pattern with appropriate caution.
| Cell | Policies | Sum of squared residuals |
|---|---|---|
| North, low risk | 20 | 800 |
| North, high risk | 5 | 500 |
| South, low risk | 5 | 500 |
| South, high risk | 20 | 800 |
Method. Divide each cell’s sum of squared residuals by its policy count and take the square root. For a territory or risk total, add the relevant cell sums and counts first; do not average the cell RMSEs.
The North, low-risk cell has \(\operatorname{RMSE}=\sqrt{800/20}=\sqrt{40}\approx6.325\) thousand dollars. North, high risk has \(\sqrt{500/5}=\sqrt{100}=10\) thousand dollars. South, low risk also has \(\sqrt{500/5}=10\) thousand dollars, and South, high risk has \(\sqrt{800/20}=\sqrt{40}\approx6.325\) thousand dollars.
North contains \(20+5=25\) policies and has total squared error \(800+500=1{,}300\). Its RMSE is \(\sqrt{1{,}300/25}=\sqrt{52}\approx7.211\) thousand dollars. South also has 25 policies and total squared error \(500+800=1{,}300\), so its RMSE is also approximately \(7.211\) thousand dollars.
The low-risk total is 25 policies and \(800+500=1{,}300\) in squared error, giving \(\sqrt{1{,}300/25}\approx7.211\) thousand dollars. The high-risk total is also 25 policies and \(500+800=1{,}300\), giving the same RMSE. Across all four cells, the total is \(50\) policies and \(2{,}600\) in squared error, so overall RMSE is \(\sqrt{2{,}600/50}=\sqrt{52}\approx7.211\) thousand dollars.
The territory and risk summaries are identical, yet the cell results differ: each territory has higher RMSE in one risk group, and that contrast reverses in the other territory. This is a pattern worth investigating, not a settled conclusion. Each high-risk or low-risk cell with five policies is small, and the table gives no evidence that the cell differences are precise or persistent. Common mistake: conclude that territory and risk are irrelevant because their separate RMSEs are equal. The correction is to inspect the joint cells when the question calls for it, while clearly labelling small cells as uncertain.
A Disciplined Diagnostic Process
A segment analysis is most useful when it is planned and reported with care. Define segments using information available for the intended prediction task, and avoid repeatedly searching for ever-smaller groups until an extreme result appears. More subdivisions create more opportunities for chance differences. If a segment finding will affect a modeling or business decision, it should be checked with suitable additional evidence rather than treated as conclusive from one small validation group.
Specify whether the concern is error magnitude, directional bias, or both. State the target scale and the unit of analysis, such as one policy.
Use interpretable categories, such as territory or risk band, and state how each policy is assigned. For a cross-classification, report the joint cells as well as relevant marginal summaries.
For each group, report its count, error measure, and—when relevant—mean residual. Confirm that partition counts and sums of squared errors reconcile to the overall result.
Describe the size of the observed difference, the number of policies behind it, and the limitations of the evidence. Do not infer a stable problem from a handful of cases or declare equal performance solely from equal marginal averages.
The reported metric should match the question. RMSE is sensitive to large residuals, so it is useful when unusually costly misses matter, but one extreme policy can have substantial influence. Mean residual reveals directional bias but allows cancellation. A segment can therefore have a near-zero mean residual and a large RMSE, or a noticeable mean residual without the largest RMSE. State what the metric measures instead of describing all error summaries simply as “accuracy.”
Finally, segment diagnostics describe validation performance, not a guarantee about future performance. A segment difference can guide further review, but its interpretation depends on sample size, the way the segments were formed, and the costs of the errors. A clear report separates the observed calculation from the conclusion: for example, “the high-risk validation segment had higher RMSE, based on four policies” is more defensible than “the model systematically fails for high-risk policies.”
Check Your Understanding
For each problem, assume the stated groups contain validation policies and the residual is observed target minus prediction. Report RMSE in the same units as the residual.
- Segment A has four policies and a sum of squared residuals of 400. Segment B has one policy and a sum of squared residuals of 900. Calculate each segment RMSE and the overall RMSE. (Answer: Segment A \(=10\), Segment B \(=30\), overall \(=\sqrt{260}\approx16.1245\).)
- Three residuals, in thousands of dollars, are \(+12,-8,+4\). Calculate the mean residual and RMSE. What does their difference illustrate? (Answer: Mean residual \(=8/3\approx2.6667\) thousand dollars; RMSE \(=\sqrt{224/3}\approx8.64099\) thousand dollars; the signed average is smaller because positive and negative residuals can offset.)
- Group A has 40 policies and RMSE 5; Group B has 10 policies and RMSE 15. Calculate the pooled RMSE. (Answer: \(\sqrt{(40(5^2)+10(15^2))/50}=\sqrt{65}\approx8.0623\).)
- A small segment has three residuals \(+30,-30,+30\), in thousands of dollars. Calculate its mean residual and RMSE, then give one reason to be cautious about the result. (Answer: Mean residual \(=10\) thousand dollars; RMSE \(=30\) thousand dollars; only three policies contribute, so the estimate may be sensitive to the observed cases.)
- A validation set has 80 low-risk policies with a sum of squared residuals of 3,200 and 20 high-risk policies with a sum of squared residuals of 5,000. Calculate each risk group RMSE and the overall RMSE. (Answer: Low risk \(=\sqrt{40}\approx6.3249\); high risk \(=\sqrt{250}\approx15.8114\); overall \(=\sqrt{82}\approx9.0554\).)