Error Profiles Across Risk Segments

Why Segment-Level Error Profiles Matter

A candidate model can improve an overall validation metric while becoming less accurate for a particular group. This can happen when the group is small, when the overall metric is dominated by a larger group, or when the model’s errors change in opposite directions across segments. A pooled score is useful, but it does not reveal where the errors occur.

For an actuarial comparison, segments might be predefined by coverage type, policy tenure, or a risk category used in the intended business process. The point is not to search many possible slices until one looks striking. It is to compare the candidate models across groups that are relevant to the decision, using the same held-out observations and a clearly stated metric.

Definition: An error profile across risk segments is a comparison of a model’s prediction errors across prespecified groups in a validation sample. For each segment, report the metric, its denominator or sample size, and the corresponding result for each candidate model. An overall metric summarizes the full sample; it does not replace the segment profile.

A segment comparison is descriptive. It can identify where one model performs better or worse on the validation data, but by itself it does not establish why the difference occurred, prove that it will persist, or show that a model caused an outcome. Interpret the profile in the context of the data, the intended use, and the consequences of errors.

Set Up a Comparable Segment Analysis

Use a validation sample that was not used to fit or select the final model, and score each candidate on the same observations. Define segment membership independently of the candidate model’s predictions. For example, if the intended review is by policy tenure, specify the tenure bands before comparing model errors. If segments overlap, a policy may appear in more than one row, so the segment totals cannot be added to obtain the pooled result.

Specify the groups and metric.
State how the segments are defined and whether the metric is, for example, RMSE, mean absolute error, or a classification rate. Use the same metric definition for both models.

Calculate each segment result.
Report the segment’s sample size and any metric-specific denominator. A rate based on actual positive cases, for instance, must use the number of actual positives rather than the total segment size.

Compare changes and the pooled result.
Identify which groups improved or deteriorated. Then calculate the overall metric from the observations or sufficient segment totals; do not assume the overall change has the same direction as every segment change.

Assess relevance and stability.
Consider the group’s exposure, error consequences, and sample size. A large rate difference in a small segment may be unstable; a modest change in a high-consequence segment may still deserve attention.

Pooling Regression Errors Correctly

For a regression outcome, let $y_i$ be the observed value and $\hat{y}_i$ the prediction for validation observation $i$. Its squared error is $(y_i-\hat{y}_i)^2$. If the segments form a mutually exclusive and exhaustive partition of the validation observations, the pooled mean squared error is a sample-size-weighted average of segment mean squared errors.

\operatorname{MSE}_{\mathrm{pooled}} = \frac{\sum_g n_g\operatorname{MSE}_g}{\sum_g n_g}, \qquad \operatorname{RMSE}_{\mathrm{pooled}} = \sqrt{\operatorname{MSE}_{\mathrm{pooled}}}.

Here, $n_g$ is the number of validation observations in segment $g$, and $\operatorname{MSE}_g$ is the mean of the squared errors in that segment. This result assumes the same squared-error definition and outcome scale in every segment, and that each observation is counted once. It is not appropriate to pool overlapping segments by simply adding their counts.

The justification follows directly from the definition. Segment $g$ contributes $n_g\operatorname{MSE}_g$, which is the sum of its squared errors. Adding these contributions gives the total sum of squared errors across the partition. Dividing by the total number of observations gives the pooled MSE. Taking the square root then gives pooled RMSE. Because the square root is nonlinear, averaging segment RMSEs generally does not give pooled RMSE.

Worked Example: A Pooled RMSE Gain With a Segment Trade-Off

In a hypothetical validation exercise, two models predict annual claim cost, measured in thousands of dollars. The validation set has 100 routine-risk policies and 25 elevated-risk policies. Model A has segment MSEs of 4 and 16, respectively; Model B has segment MSEs of 5 and 9. Calculate the pooled RMSE for each model, identify the segment changes, and explain the common pooling mistake.

Method. Reconstruct each pooled MSE from the segment MSEs, weighting by segment sample size, and then take the square root. The units of the resulting RMSE are thousands of dollars.

Model A. The routine-risk contribution to the sum of squared errors is $100(4)=400$. The elevated-risk contribution is $25(16)=400$. Thus the total is $400+400=800$ across $100+25=125$ policies:

\operatorname{MSE}_{A,\mathrm{pooled}}=\frac{800}{125}=6.4, \qquad \operatorname{RMSE}_{A,\mathrm{pooled}}=\sqrt{6.4}\approx2.5298.

Model B. Its routine-risk contribution is $100(5)=500$, and its elevated-risk contribution is $25(9)=225$. The total is $500+225=725$:

\operatorname{MSE}_{B,\mathrm{pooled}}=\frac{725}{125}=5.8, \qquad \operatorname{RMSE}_{B,\mathrm{pooled}}=\sqrt{5.8}\approx2.4083.

Rounded to four decimal places, Model B’s pooled RMSE is lower by $2.5298-2.4083=0.1215$ thousand dollars, or about $121.50. Yet Model B’s routine-risk MSE rises from 4 to 5, while its elevated-risk MSE falls from 16 to 9. The pooled improvement therefore includes deterioration in the larger segment and improvement in the smaller segment.

Common mistake. Averaging the two segment RMSEs equally would give Model A $(\sqrt{4}+\sqrt{16})/2=(2+4)/2=3$, and Model B $(\sqrt{5}+\sqrt{9})/2\approx2.6180$. Those are not pooled RMSEs: they give the 25-policy segment the same weight as the 100-policy segment and average after taking square roots. The correction is to pool squared errors first, as above. The result checks: Model B has the lower pooled RMSE, but not a lower MSE in every segment.

Classification Rates Need the Right Denominator

For a binary outcome, an error rate must be defined with a clear denominator. For example, the false-negative rate in a segment is the number of actual positive cases predicted negative divided by the number of actual positive cases in that segment. It is not the number of false negatives divided by all observations in the segment. Other rates, such as the false-positive rate, use a different denominator.

When pooling false-negative rates across segments, weight each rate by the number of actual positives in that segment. Weighting by total segment size would answer a different question and can produce a misleading comparison when the prevalence of positives varies by segment.

Worked Example: A Lower Pooled Miss Rate but More Misses in a High-Risk Group

A hypothetical claims triage system labels cases that require specialist review as positive. In validation, there are 200 actual positive cases in a routine segment and 20 in a high-risk segment. Model A has 20 and 8 false negatives in those segments; Model B has 10 and 9. Calculate both pooled false-negative rates and compare the high-risk results.

Method. Divide false negatives by actual positives, first within each segment and then across the full validation sample. The total number of actual positives is $200+20=220$.

For Model A, the routine-segment false-negative rate is $20/200=0.10$, and the high-risk rate is $8/20=0.40$. There are $20+8=28$ false negatives overall, so the pooled rate is:

\frac{28}{220}=0.1272727\ldots\approx0.1273.

For Model B, the routine-segment rate is $10/200=0.05$, and the high-risk rate is $9/20=0.45$. There are $10+9=19$ false negatives overall, so:

\frac{19}{220}=0.0863636\ldots\approx0.0864.

The pooled false-negative rate falls from approximately 0.1273 to 0.0864, a decrease of about $0.0409$, or 4.09 percentage points. However, the high-risk segment’s rate rises from 0.40 to 0.45, a 5-percentage-point deterioration. The routine-segment rate falls by 5 percentage points. Model B makes fewer false negatives overall, but it misses a larger share of actual positives in the high-risk group.

Common mistake. Dividing Model B’s 9 high-risk false negatives by all 20 high-risk cases gives 0.45, but this is correct here only because the problem states that the segment contains 20 actual positives and provides no other cases. If the segment had additional actual negatives, its total size would not be the false-negative-rate denominator. Always identify the metric’s denominator before pooling. The pooled calculation checks because $19/220$ equals the total false negatives divided by total actual positives.

Consequences, Weighting, and Interpretation

Not all errors have the same business consequence. A small average error in a high-consequence group may matter more than a larger error in a routine group, but any weighting scheme must be explicit and defensible. One option is to report an unweighted metric for each segment alongside a separately defined consequence-weighted metric. Such a metric is not automatically the right objective: its weights should reflect the intended decision, be chosen without tuning to favor a candidate, and be explained to the reader.

For example, suppose the validation set contains 90 routine policies and 10 critical policies. The segment mean absolute errors are $100 and $500 for Model A, and $120 and $300 for Model B. The ordinary pooled MAE for Model A is $(90(100)+10(500))/100=14{,}000/100=\$140$. For Model B it is $(90(120)+10(300))/100=13{,}800/100=\$138$. Model B’s ordinary pooled MAE improves by $2 per policy, while its MAE worsens by $20 in the routine segment and improves by $200 in the critical segment.

Suppose, solely for illustration, the analysis had prespecified a consequence weight of 1 for routine policies and 5 for critical policies. A weighted mean absolute error is the sum of weighted absolute errors divided by the sum of weights. Model A’s weighted numerator is $90(1)(100)+10(5)(500)=34{,}000$, and its denominator is $90(1)+10(5)=140$, giving approximately $242.86. Model B’s numerator is $90(1)(120)+10(5)(300)=25{,}800$, giving approximately $184.29. This weighted comparison emphasizes the critical segment; it does not erase the segment-level results or prove that these particular weights are suitable.

For any reported profile, include the segment counts and state whether a metric is a rate, a per-observation average, or a weighted quantity. Small segment samples can yield volatile estimates, especially when only a few positive outcomes occur. Show the numerator and denominator for consequential rates where possible, and avoid presenting small differences as stable facts without supporting evidence.

Key takeaway: A pooled gain is not a universal gain. Show segment results, use the correct denominators and aggregation rule, and assess deterioration in light of group size and the consequences of the errors.

Common Exam Traps

Averaging segment RMSEs directly. Pool segment squared-error totals and take one square root; do not take an unweighted average of segment RMSEs.
Using the wrong denominator for a rate. A false-negative rate is conditional on actual positives, not all observations in the segment.
Reporting only the pooled result. A large segment can dominate an overall metric and mask deterioration in a smaller, consequential group.
Ignoring sample sizes. Segment rates based on few cases can be unstable; report counts alongside rates.
Changing the segments or weights after seeing results. Prespecify relevant groups and any consequence weights to reduce selective interpretation.
Overstating what the comparison shows. Validation differences describe observed predictive performance; they do not by themselves establish a cause or guarantee future performance.

Check Your Understanding

For each problem, calculate the requested segment or pooled quantity and pay close attention to the metric definition.

A regression model has segment MSEs of 9 for 40 observations and 25 for 10 observations. What is the pooled MSE and pooled RMSE? (Answer: pooled MSE $=12.2$; pooled RMSE $\approx3.493$.)
Two segments contain 80 and 20 actual positive cases. A model has 8 and 6 false negatives, respectively. What are the segment false-negative rates and pooled rate? (Answer: $0.1000$, $0.3000$, and $14/100=0.1400$.)
Model A has segment MSEs 4 and 16 for groups of 80 and 20. Model B has MSEs 5 and 9. Calculate each pooled RMSE and state which segment worsens under Model B. (Answer: Model A $\sqrt{6.4}\approx2.530$; Model B $\sqrt{5.8}\approx2.408$; the first segment worsens.)
There are 150 actual positives in Segment L and 30 in Segment H. A candidate model has 12 and 12 false negatives. Calculate the pooled false-negative rate and Segment H’s rate. (Answer: pooled $24/180=0.1333$; Segment H $12/30=0.4000$.)
A weighted MAE uses weight 1 for each of 50 routine observations and weight 4 for each of 10 severe observations. The respective segment MAEs are $60 and $240. Calculate the weighted MAE. (Answer: $(50(1)(60)+10(4)(240))/(50+40)=\$140.00$.)

What You'll Learn