Why Examine Errors by Segment?
An overall performance measure summarizes a model across all evaluated observations. That summary can be useful, but it may conceal a concentrated weakness: predictions may be less accurate for one service channel, product type, claim category, or other operationally meaningful group. Segment-level diagnostics make those differences visible so that they can be investigated.
A segment comparison is a descriptive diagnostic, not an explanation of why errors occurred. If a model has a higher error rate for one channel, that finding does not establish that the channel caused the errors, that its staff caused them, or that the model is unfair. The segment may differ in case mix, data completeness, sample size, or other relevant conditions. The appropriate conclusion is limited to the observed comparison unless further analysis supports a stronger claim.
A useful segment is defined clearly enough that someone can tell which observations it includes and why the comparison matters. Segments should generally be specified before looking for the largest errors. Searching across many arbitrary groupings and reporting only the most striking difference increases the chance of highlighting a random fluctuation.
Choose a Measure That Matches the Prediction Task
For a binary classification task, a confusion matrix separates false positives from false negatives. These error types can have different operational consequences, so a single error rate may not be enough. Let \(TP\), \(FP\), \(TN\), and \(FN\) denote the numbers of true positives, false positives, true negatives, and false negatives in a segment. Then the error rate is \((FP+FN)/n\), where \(n=TP+FP+TN+FN\). The false-negative rate is \(FN/(TP+FN)\), and the false-positive rate is \(FP/(FP+TN)\).
The denominators matter. The false-negative rate is calculated among actual positives; the false-positive rate is calculated among actual negatives. Comparing raw error counts across segments of very different sizes is not a fair comparison of rates. Comparing rates without showing their denominators can also be misleading: the same rate based on a small number of cases is generally less stable than one based on many cases.
For a numeric prediction, define the residual for observation \(i\) as \(r_i=y_i-\hat{y}_i\), where \(y_i\) is the observed value and \(\hat{y}_i\) is the prediction. A positive residual means the prediction was too low under this convention. Common segment summaries include mean absolute error (MAE), root mean squared error (RMSE), and mean signed residual. MAE summarizes typical absolute error in the outcome’s units; RMSE gives relatively more influence to large errors; mean signed residual can reveal systematic underprediction or overprediction but may conceal errors that cancel.
Here \(g\) denotes a segment and \(n_g\) is the number of evaluated observations in it. These measures answer different questions. For example, a segment can have a modest mean signed residual but a large MAE if positive and negative residuals cancel. Report the measure that matches the decision or risk being examined rather than selecting whichever metric produces the most dramatic contrast.
Worked Example: Comparing Classification Errors Across Service Channels
Method. Calculate the overall error rate and the two conditional error rates separately for each channel. This avoids treating false positives and false negatives as interchangeable. The following counts are hypothetical results from a model that flags cases for additional review.
| Channel | Actual positives | TP | FN | Actual negatives | TN | FP |
|---|---|---|---|---|---|---|
| Online | 20 | 16 | 4 | 80 | 68 | 12 |
| Assisted | 10 | 6 | 4 | 40 | 36 | 4 |
Online channel. The segment size is \(20+80=100\). It has \(4+12=16\) errors, so its error rate is \(16/100=0.16\), or 16%. Its false-negative rate is \(4/20=0.20\), or 20%, and its false-positive rate is \(12/80=0.15\), or 15%.
Assisted channel. The segment size is \(10+40=50\). It has \(4+4=8\) errors, so its error rate is \(8/50=0.16\), also 16%. Its false-negative rate is \(4/10=0.40\), or 40%, and its false-positive rate is \(4/40=0.10\), or 10%.
The two channels have the same overall error rate, but the composition of their errors differs. The assisted channel has a higher false-negative rate, while the online channel has a higher false-positive rate. The counts show the size of the evidence behind each rate: both channels have four false negatives, but those four are divided by different numbers of actual positives.
Common mistake avoided. It would be incorrect to conclude that the channels perform identically because both have a 16% error rate. That rate combines two error types that may have different consequences. The calculation identifies a descriptive difference; it does not establish why the difference exists.
How Segment Summaries Relate to the Overall Metric
A pooled metric combines the underlying observations, not necessarily the segment-level metric values with equal weights. For an ordinary error rate, let \(e_g\) be the number of errors in segment \(g\) and \(n_g\) its size. The pooled error rate is \(\sum_g e_g/\sum_g n_g\). Since \(e_g=n_g(e_g/n_g)\), this can also be written as a segment-size-weighted average of the segment error rates.
This identity follows by replacing each segment’s error count \(e_g\) with its rate \(e_g/n_g\) multiplied by its denominator \(n_g\). It explains why the overall error rate is not generally the unweighted average of the segment rates.
The appropriate weights depend on the metric’s denominator. A pooled false-negative rate weights each segment by its number of actual positives, not by its total size. A pooled false-positive rate weights each segment by its number of actual negatives. This is a direct consequence of combining the relevant numerators and denominators across segments.
Worked Example: Pooling False-Negative Rates Correctly
Method. Add false negatives and actual positives across the segments before dividing. This uses the false-negative-rate denominator and avoids giving a small segment the same influence as a large segment without stating that this is the desired comparison.
Suppose a hypothetical review model is evaluated in two operational groups. Group Cedar has 100 actual positives and a false-negative rate of 10%, so its false negatives are \(100(0.10)=10\). Group Maple has 20 actual positives and a false-negative rate of 50%, so its false negatives are \(20(0.50)=10\).
The pooled false-negative rate is \((10+10)/(100+20)=20/120=1/6\), or approximately 16.67%. The unweighted average of the two segment rates is \((10\%+50\%)/2=30\%\). These are different summaries because they answer different questions.
The pooled value describes the fraction of all actual positives that were missed. The unweighted average describes the average of the two segment rates when each group receives equal weight. Neither is automatically the right headline: the choice depends on the intended interpretation. Reporting both can be useful when the equal-segment view matters as well as the observation-level pooled view.
Check. The segment counts imply 20 false negatives among 120 actual positives, so \(20/120=0.1667\) to four decimal places. The calculation is consistent with the pooled result.
Common mistake avoided. Averaging 10% and 50% and calling the result the pooled false-negative rate ignores that Cedar has five times as many actual positives as Maple. The correction is to pool the numerator and denominator, or equivalently to weight each segment rate by its actual-positive count.
Worked Example: Diagnosing Numeric Prediction Errors
Method. Calculate residuals using \(r_i=y_i-\hat y_i\), then summarize absolute error, squared error, and signed error by branch. This avoids confusing underprediction with overprediction or using signed errors alone when large errors may cancel.
A hypothetical estimator predicts repair costs, in dollars, for five completed jobs. In the North branch, observed costs are 100, 120, and 80; predictions are 90, 130, and 70. The residuals are \(100-90=10\), \(120-130=-10\), and \(80-70=10\). Thus the absolute residuals are 10, 10, and 10, while the squared residuals are 100, 100, and 100.
For North, the absolute residual total is \(10+10+10=30\), giving \(\operatorname{MAE}=30/3=10\). The squared residual total is \(100+100+100=300\), giving \(\operatorname{RMSE}=\sqrt{300/3}=\sqrt{100}=10\). The signed residual total is \(10-10+10=10\), so the mean signed residual is \(10/3\), or approximately 3.33 dollars.
In the South branch, observed costs are 50 and 200, while predictions are 70 and 170. The residuals are \(50-70=-20\) and \(200-170=30\). The absolute residuals total \(20+30=50\), so \(\operatorname{MAE}=50/2=25\). The squared residuals total \(400+900=1{,}300\), so \(\operatorname{RMSE}=\sqrt{1{,}300/2}=\sqrt{650}\), approximately 25.50 dollars. The signed residual total is \(-20+30=10\), giving a mean signed residual of \(10/2=5\) dollars.
South has higher MAE and RMSE than North, indicating larger absolute errors in this small evaluation. The mean signed residuals are both positive, but that alone does not capture the magnitude: South’s two residuals partly offset even though both absolute errors exceed North’s. Across all five jobs, the pooled MAE is \((30+50)/5=16\) dollars. The unweighted average of branch MAEs is \((10+25)/2=17.50\) dollars, a different quantity because the branches have different sample sizes.
Check. North’s residuals sum to 10, and South’s also sum to 10, so the total signed residual is 20 across five jobs and the overall mean signed residual is 4 dollars. This agrees with the size-weighted calculation \((3(10/3)+2(5))/5=4\).
Common mistake avoided. Reporting only the signed residual average could suggest that the branches have similar performance, because residuals of opposite signs cancel. MAE and RMSE add information about error magnitude, while the signed summary retains information about direction.
Interpret Differences With Appropriate Caution
A segment table is a prompt for investigation, not a causal analysis. An observed gap can reflect genuine differences in prediction quality, different outcome prevalence, different case mix, measurement practices, or random variation. Some of these explanations may coexist. The table alone does not distinguish among them.
A responsible interpretation identifies the measure and denominator, describes the observed contrast, and notes relevant limitations. For example: “In this evaluation, the false-negative rate was higher in the assisted channel than in the online channel; the estimate for the assisted channel is based on 10 actual positives. This comparison identifies an area for review but does not establish the reason for the difference.” This is more precise than stating that a channel or group “caused” poor predictions.
Consider these checks before presenting a segment comparison:
- Confirm the definition. State how group membership is assigned and whether categories overlap. If observations can appear in more than one group, do not treat the segment counts as disjoint when calculating a pooled rate.
- Show denominators. Give the segment size and, for conditional classification rates, the relevant actual-positive or actual-negative count.
- Use a metric tied to the decision. If false negatives and false positives have different consequences, show them separately rather than relying only on total error rate.
- Account for sparse data. A very high rate based on only a few eligible observations can be unstable. Report the count and avoid presenting a small subgroup estimate as conclusive.
- Avoid selective comparisons. If many segments or metrics were examined, say so. Highlighting only the largest difference can overstate how compelling the finding is.
- Keep diagnosis distinct from explanation. Use the observed pattern to motivate checks of data, procedures, and model behavior; do not claim a cause without evidence that supports it.
Operational relevance is also important. A segment distinction should help someone understand a possible service, risk, or workflow issue. Defining increasingly narrow groups merely to produce a striking rate can leave too few observations for a dependable comparison and can invite unsupported conclusions about individuals represented by that group.
Check Your Understanding
For each problem, identify the appropriate denominator and distinguish the numerical result from any causal interpretation.
- A hypothetical model has 30 actual positives in Route A, with 24 correctly flagged, and 50 actual negatives, with 45 correctly left unflagged. In Route B, it has 10 actual positives, with 7 correctly flagged, and 40 actual negatives, with 38 correctly left unflagged. Calculate each route’s false-negative rate and false-positive rate. (Answer: Route A: FNR \(6/30=20\%\), FPR \(5/50=10\%\); Route B: FNR \(3/10=30\%\), FPR \(2/40=5\%\).)
- Two hypothetical segments have 80 actual positives and FNR 15%, and 40 actual positives and FNR 30%. Find the pooled false-negative rate and the unweighted average of the segment rates. (Answer: Pooled \(=(12+12)/(80+40)=20\%\); unweighted average \(=(15\%+30\%)/2=22.5\%\).)
- A numeric model has observed values 40, 60, and 100, and predictions 50, 55, and 90. Calculate the residuals using observed minus predicted, the MAE, and the mean signed residual. (Answer: Residuals \(-10, 5, 10\); MAE \(=25/3\approx 8.33\); mean signed residual \(=5/3\approx 1.67\), in the outcome’s units.)
- A segment contains 8 actual positives, of which 6 are correctly identified. Another contains 80 actual positives, of which 64 are correctly identified. Calculate each FNR and the pooled FNR. Why should the smaller segment’s estimate be reported cautiously? (Answer: FNRs are \(2/8=25\%\) and \(16/80=20\%\); pooled FNR is \(18/88\approx20.45\%\). The smaller estimate is based on only 8 actual positives and is therefore less stable.)
- A hypothetical model has MAE of 12 dollars in one branch and 20 dollars in another. The first branch has 90 evaluated cases and the second has 10. Calculate the pooled MAE and the unweighted mean of the branch MAEs. Can the difference establish that branch assignment caused the error gap? (Answer: Pooled MAE \(=(90(12)+10(20))/100=12.80\) dollars; unweighted mean \(=(12+20)/2=16\) dollars. No; the comparison is descriptive and does not establish a cause.)