Why Report Prediction Errors by Segment?
A single validation metric can summarize overall predictive performance while concealing uneven results across a portfolio. A model may have modest overall error but perform materially worse for one type of policy, region, or claim. A segment-level error table makes those differences visible by reporting the number of observations and selected numeric prediction metrics within each group.
The purpose is not to produce a long table of every possible statistic. It is to make a useful comparison: which segments have meaningfully different error, how many records support each estimate, and whether a signed average might hide errors that point in opposite directions. Segment comparisons are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
This sign convention means a positive error is an overprediction and a negative error is an underprediction. The convention could instead be defined as actual minus prediction, but it must be stated and used consistently. MAE and RMSE are nonnegative and have the same units as the outcome. Mean error also has those units, but positive and negative errors can cancel. Report the sample size \(n_g\) alongside the metrics: a result based on a handful of observations should not be presented as equally stable as one based on hundreds.
Constructing a Useful Table
Choose the segments based on a meaningful business or modeling question, define them before inspecting the results when possible, and assign each validation record to exactly one segment for a partitioned comparison. Use the same outcome definition, prediction scale, error sign, and metric formulas in every row. State the units. If records are omitted because an actual outcome or prediction is unavailable, define the included population and show the resulting group sizes; do not let the denominator change silently from metric to metric.
A compact table often includes segment, count, mean error, MAE, and RMSE. Mean error helps identify directional bias, while MAE and RMSE summarize error magnitude in different ways. RMSE gives relatively more influence to large errors because the errors are squared. These metrics answer different questions, so a gap in MAE and a gap in mean error should not be described as interchangeable findings.
Specify the outcome, prediction scale, validation records, segment definitions, and error sign.
Count the eligible records in each segment and verify that records have been assigned as intended.
Use the group’s own errors and denominator \(n_g\), not an overall denominator.
Compare magnitude and direction, consider group size, and avoid claiming that a metric difference explains its own cause.
Worked Example: Calculating Metrics for Two Segments
A hypothetical service team validates predictions of repair cost, in hundreds of dollars. The seven records below are split into two regions. Calculate mean error, MAE, and RMSE for each region and for the full set. Use \(e_i=\hat{y}_i-y_i\). The method is to calculate errors first and then aggregate within each group; this avoids mixing up prediction and actual values or using the wrong group denominator.
| Region | Actual \(y_i\) | Prediction \(\hat{y}_i\) | Error \(e_i\) | \(|e_i|\) | \(e_i^2\) |
|---|---|---|---|---|---|
| North | 10 | 8 | -2 | 2 | 4 |
| North | 12 | 13 | 1 | 1 | 1 |
| North | 9 | 12 | 3 | 3 | 9 |
| South | 15 | 11 | -4 | 4 | 16 |
| South | 8 | 8 | 0 | 0 | 0 |
| South | 11 | 13 | 2 | 2 | 4 |
| South | 14 | 16 | 2 | 2 | 4 |
North. There are \(n_{\text{North}}=3\) records. The errors sum to \(-2+1+3=2\), the absolute errors sum to \(2+1+3=6\), and the squared errors sum to \(4+1+9=14\). Therefore mean error is \(2/3=0.6667\), MAE is \(6/3=2\), and RMSE is \(\sqrt{14/3}=\sqrt{4.6667}=2.1602\).
South. There are \(n_{\text{South}}=4\) records. The errors sum to \(-4+0+2+2=0\), the absolute errors sum to \(4+0+2+2=8\), and the squared errors sum to \(16+0+4+4=24\). Mean error is \(0/4=0\), MAE is \(8/4=2\), and RMSE is \(\sqrt{24/4}=\sqrt{6}=2.4495\).
All records. There are \(3+4=7\) records. The total error is \(2+0=2\), total absolute error is \(6+8=14\), and total squared error is \(14+24=38\). Thus mean error is \(2/7=0.2857\), MAE is \(14/7=2\), and RMSE is \(\sqrt{38/7}=\sqrt{5.4286}=2.3299\).
| Group | Count | Mean error | MAE | RMSE |
|---|---|---|---|---|
| North | 3 | 0.6667 | 2.0000 | 2.1602 |
| South | 4 | 0.0000 | 2.0000 | 2.4495 |
| All records | 7 | 0.2857 | 2.0000 | 2.3299 |
The units are hundreds of dollars. Both regions have the same MAE, but South has the larger RMSE, reflecting its more influential error of \(-4\). A mean error of zero in South does not mean its predictions were exact: positive and negative errors can cancel. The calculations check out because the overall error totals equal the sums of the two regional totals.
How Segment Summaries Combine
A partitioned table also supports an overall metric check. For each segment, the total absolute error is \(n_g\operatorname{MAE}_g\), and the total squared error is \(n_g\operatorname{RMSE}_g^2\). Adding those totals across segments gives the corresponding totals for all records. Dividing by the full sample size gives the pooled metrics.
The same reasoning gives the pooled mean error as the group-size-weighted average of segment mean errors. These identities follow directly from partitioning the observations: every absolute error and every squared error belongs to one segment, so adding the segment totals counts each record exactly once. In the RMSE identity, the quantities to combine before taking the square root are the segment mean squared errors, \(\operatorname{RMSE}_g^2\), not the RMSE values themselves.
This is a useful calculation check, not a substitute for displaying the segment results. A pooled metric can be dominated by a large group and may conceal a poor result in a smaller one. Conversely, a metric from a small segment may be much less stable. Keep group counts visible and interpret gaps in light of the intended use and the available validation data.
Worked Example: Finding a High-Error Segment Without Hiding Its Size
A hypothetical property insurer reviews validation summaries for three product segments. The amounts below are totals over each segment’s records, with prediction errors measured in thousands of dollars. Compute the table metrics and pooled results. The method is to divide each total by its own segment count and take the square root only after finding mean squared error. This avoids both the wrong denominator and the mistake of treating RMSE as an additive quantity.
| Segment | \(n_g\) | \(\sum e_i\) | \(\sum |e_i|\) | \(\sum e_i^2\) |
|---|---|---|---|---|
| Standard | 100 | 20 | 400 | 2,400 |
| Coastal | 10 | -30 | 100 | 1,400 |
| Equipment | 5 | 5 | 45 | 625 |
For Standard, mean error is \(20/100=0.2\), MAE is \(400/100=4\), and RMSE is \(\sqrt{2,400/100}=\sqrt{24}=4.8990\). For Coastal, mean error is \(-30/10=-3\), MAE is \(100/10=10\), and RMSE is \(\sqrt{1,400/10}=\sqrt{140}=11.8322\). For Equipment, mean error is \(5/5=1\), MAE is \(45/5=9\), and RMSE is \(\sqrt{625/5}=\sqrt{125}=11.1803\).
Across all three segments, the count is \(100+10+5=115\). The total signed error is \(20-30+5=-5\), the total absolute error is \(400+100+45=545\), and the total squared error is \(2,400+1,400+625=4,425\). The pooled mean error is \(-5/115=-0.0435\), pooled MAE is \(545/115=4.7391\), and pooled RMSE is \(\sqrt{4,425/115}=\sqrt{38.4783}=6.2031\).
| Segment | Count | Mean error | MAE | RMSE |
|---|---|---|---|---|
| Standard | 100 | 0.2000 | 4.0000 | 4.8990 |
| Coastal | 10 | -3.0000 | 10.0000 | 11.8322 |
| Equipment | 5 | 1.0000 | 9.0000 | 11.1803 |
| All records | 115 | -0.0435 | 4.7391 | 6.2031 |
Coastal has the largest MAE and RMSE, while Equipment also has high error summaries. The near-zero pooled mean error is not evidence that the model has little error: the signed errors can offset one another, and Standard contains most of the records. Counts matter here. The results warrant investigating the smaller segments, but their counts of 10 and 5 also caution against treating their metrics as precise or conclusive on their own. A metric table identifies a pattern; it does not establish its cause.
Worked Example: Reading Dollar Errors in Context
A hypothetical claim team evaluates predictions of claim cost, recorded in thousands of dollars. The following records are divided among three product segments. Calculate each segment’s metrics and identify what can and cannot be concluded. The method is to retain the stated units and compare direction and magnitude separately; this avoids treating a low signed average as proof of low error.
| Segment | Actual | Prediction | Error |
|---|---|---|---|
| Small account | 4 | 5 | 1 |
| Small account | 6 | 5 | -1 |
| Small account | 5 | 7 | 2 |
| Commercial | 10 | 14 | 4 |
| Commercial | 8 | 6 | -2 |
| Commercial | 12 | 12 | 0 |
| Specialty | 20 | 30 | 10 |
| Specialty | 15 | 10 | -5 |
| Specialty | 10 | 15 | 5 |
For Small account, the errors are \(1,-1,2\). Their sum is \(2\), absolute-value sum is \(1+1+2=4\), and squared-value sum is \(1+1+4=6\). With \(n=3\), mean error is \(2/3=0.6667\), MAE is \(4/3=1.3333\), and RMSE is \(\sqrt{6/3}=\sqrt{2}=1.4142\).
For Commercial, the errors are \(4,-2,0\). Their sum is \(2\), absolute-value sum is \(4+2+0=6\), and squared-value sum is \(16+4+0=20\). With \(n=3\), mean error is \(2/3=0.6667\), MAE is \(6/3=2\), and RMSE is \(\sqrt{20/3}=\sqrt{6.6667}=2.5820\).
For Specialty, the errors are \(10,-5,5\). Their sum is \(10\), absolute-value sum is \(10+5+5=20\), and squared-value sum is \(100+25+25=150\). With \(n=3\), mean error is \(10/3=3.3333\), MAE is \(20/3=6.6667\), and RMSE is \(\sqrt{150/3}=\sqrt{50}=7.0711\).
The Specialty segment has substantially larger error summaries in this small validation set. Its mean error is also positive, indicating net overprediction under the chosen convention. All metrics are in thousands of dollars. Each group has only three observations, however, so these figures alone are not enough to determine whether the same pattern will persist in future data. Verify the totals: across the three groups there are nine records, total signed error \(2+2+10=14\), total absolute error \(4+6+20=30\), and total squared error \(6+20+150=176\).
Common Mistakes and Interpretation
A segment table is only as clear as its definitions. State what a segment means, how many validation records it contains, and what the reported metrics measure. Avoid conclusions stronger than the evidence supports. A difference in error can be a reason to investigate data quality, model fit, or the segment definition, but the error table itself does not determine which explanation is correct.
- Omitting the count. A large metric from a group of five is not supported by the same amount of data as a similar result from a group of 500. Report \(n_g\) next to each row.
- Using the wrong denominator. Calculate each segment metric using its own number of eligible records. Do not divide a segment total by the full validation-set count.
- Averaging RMSE values directly. RMSE is a square root of an average squared error, so pooled RMSE is based on group counts and segment RMSE squared, not a simple average of segment RMSE.
- Reading mean error as error magnitude. A mean error near zero can result from offsetting positive and negative errors. Read it alongside MAE or RMSE.
- Declaring a segment “worse” without context. Note the metric, its units, the size of the gap, and the group count. A practical judgment of materiality depends on the intended decision and the consequences of error.
- Changing definitions between rows. Keep the outcome scale, validation population, sign convention, and metric calculation consistent so the comparisons are interpretable.
Check Your Understanding
For each problem, use signed error \(e_i=\hat{y}_i-y_i\). Give calculations in the stated units.
- Segment A has errors \(2,-1,1\), and Segment B has errors \(-3,0\). Calculate each segment’s count, mean error, MAE, and RMSE. (Answer: A: \(n=3\), mean error \(0.6667\), MAE \(1.3333\), RMSE \(1.4142\); B: \(n=2\), mean error \(-1.5000\), MAE \(1.5000\), RMSE \(2.1213\).)
- Segment P has 4 records and RMSE 3; Segment Q has 6 records and RMSE 4. Find the pooled RMSE. Do not average the two RMSE values directly. (Answer: \(\sqrt{(4(3^2)+6(4^2))/10}=\sqrt{13.2}=3.6332\).)
- A validation group has errors \(0,2,-2,4\), and another has errors \(-1,-1,-1\). Calculate mean error, MAE, and RMSE for each group. Which group has the larger MAE? (Answer: First group: mean error \(1\), MAE \(2\), RMSE \(2.4495\); second group: mean error \(-1\), MAE \(1\), RMSE \(1\); the first group has the larger MAE.)
- Segment X has \(n=50\), total absolute error \(100\), and total squared error \(300\). Segment Y has \(n=5\), total absolute error \(40\), and total squared error \(500\). Calculate each segment’s MAE and RMSE, then the pooled MAE and RMSE. What should be noted about Y? (Answer: X: MAE \(2\), RMSE \(2.4495\); Y: MAE \(8\), RMSE \(10\); pooled MAE \(140/55=2.5455\), pooled RMSE \(\sqrt{800/55}=3.8138\). Y has much larger errors but only five records, so the result merits investigation and cautious interpretation.)