Why Report Performance by Segment?
An overall performance measure can be useful, but it may conceal important variation across groups that face different decisions or operating conditions. A model with an acceptable average error, for example, could still make substantially larger errors for one region or customer type. Segment-level reporting makes that variation visible so decision-makers can ask whether the model is suitable for the intended uses.
A segment is a defined subset of the evaluation population, such as a product type, region, or customer channel. Segments should be chosen because they are relevant to the decision, not merely because they can be found in the data. A useful report states how each segment is defined, how many observations it contains, and the performance measure used.
The denominator matters because different metrics answer different questions. Mean absolute error (MAE), for example, is the average absolute difference between a numeric prediction and its observed outcome. Recall is the proportion of actual positive cases that the model correctly identifies at a specified classification threshold. A report should show the count relevant to the metric: the number of evaluated records for MAE, but the number of actual positive cases for recall.
Comparability also depends on the evaluation setup. If one segment is evaluated over a different observation window, with a different outcome definition, or at a different classification threshold, the resulting metrics may not be directly comparable. When these conditions cannot be aligned, state the difference rather than presenting the values as though they were like-for-like.
Define the Comparison Before Interpreting It
A clear segment comparison begins with a small set of decisions. Specify which groups matter to the intended action, which performance metric answers the decision question, and what difference would be practically important enough to flag. There is no universal numerical cutoff for a material gap: the criterion depends on the consequences of the decision and should not be chosen after seeing which segments look favorable.
Use clear, reproducible categories and say whether the groups form a complete, non-overlapping partition of the evaluation records.
Use a consistent outcome, observation window, model, and threshold where applicable.
Show the segment count and the count that specifically enters the metric, such as actual positive cases for recall.
Compare results with a decision-relevant benchmark or pre-established practical criterion, and describe the evidence without assigning a cause it cannot establish.
A segment with few observations can produce a result that is sensitive to individual records. The count should therefore accompany the metric; the metric alone does not show how much data supports it. Segment comparisons are descriptive evidence about the evaluated records. By themselves, they do not establish why a difference occurred or whether changing a segment attribute would change model performance.
Worked Segment Comparisons
Worked Example: Comparing Numeric Errors Across Regions
A hypothetical service provider evaluates one model on six held-out claims, using the same outcome definition and evaluation period in two regions. Absolute prediction errors, in dollars, are 1, 2, 1, and 4 in Region North, and 5 and 7 in Region South. Calculate each region's MAE and the overall MAE. The provider's pre-established review rule is to flag a regional MAE difference of at least $3.
Method. For each region, add its absolute errors and divide by its number of evaluated claims. The segment counts are 4 for North and 2 for South.
For North, the total absolute error is \(1+2+1+4=8\) dollars. Its MAE is \(8/4=\$2\). For South, the total is \(5+7=12\) dollars, so its MAE is \(12/2=\$6\).
The regional difference is \(\$6-\$2=\$4\), which meets the stated \(\$3\) review rule. The overall MAE is calculated from all six claims: the total absolute error is \(8+12=20\) dollars, and \(20/6=\$3.333\ldots\). Rounded to the nearest cent, the overall MAE is \(\$3.33\).
Check. The segment totals add to the overall total, \(8+12=20\), and the segment counts add to the overall count, \(4+2=6\). The result supports reporting a higher MAE in South in this evaluation set. It does not show that being in South caused the higher errors.
Common mistake avoided: Averaging the two regional MAEs equally gives \((\$2+\$6)/2=\$4\), which is not the overall MAE because the regions have different counts. The overall result must reflect the four North claims and two South claims.
The second equality follows because \(\sum_{i=1}^{n_g}|e_{ig}|=n_g\operatorname{MAE}_g\) for each segment. Thus the overall MAE is the segment MAEs averaged with segment record counts as weights. This result requires the same unweighted MAE definition across a complete partition of the records. If observations use different evaluation weights, the aggregation must use the corresponding weighted totals instead.
Worked Example: Comparing Recall With the Correct Denominator
A hypothetical screening model is evaluated at the same fixed threshold for two service channels. In Channel A, 40 records have an actual positive outcome; the model identifies 32 of them as positive and misses 8. In Channel B, 10 records have an actual positive outcome; the model identifies 6 and misses 4. Calculate recall for each channel and overall. A review rule flags a difference of at least 0.15.
Method. Recall is true positives divided by actual positives, so the denominator is the number of actual positive records, not the total number of records in the channel.
Channel A recall is \(32/(32+8)=32/40=0.8000\). Channel B recall is \(6/(6+4)=6/10=0.6000\). The difference is \(0.8000-0.6000=0.2000\), which exceeds the 0.15 review rule.
For the overall recall, add the true positives and actual positives across channels: there are \(32+6=38\) true positives among \(40+10=50\) actual positive records. Overall recall is \(38/50=0.7600\).
Check. The actual-positive denominators sum to 50, and the true-positive counts sum to 38, so the overall calculation uses the same definition as the segment calculations. The result supports reporting lower recall in Channel B for this evaluation. It does not establish why the channels differ.
Common mistake avoided: Dividing Channel B's 6 true positives by all Channel B records would calculate a different quantity, not recall. A segment's total size can be useful context, but it is not the recall denominator.
Worked Example: Identifying Opposing Segment Results Behind a Near-Baseline Overall Ratio
A hypothetical portfolio model produces total predicted losses of $12,000 for 100 urban policies and $8,000 for 50 rural policies. The corresponding observed losses are $13,200 and $7,200. Calculate the observed-to-predicted ratio for each segment and overall. Explain what the comparison supports.
Method. For each group, divide its total observed loss by its total predicted loss. A ratio above 1 means observed total loss exceeded predicted total loss in that group; a ratio below 1 means the reverse.
For the urban segment, the ratio is \(13{,}200/12{,}000=1.10\). For the rural segment, it is \(7{,}200/8{,}000=0.90\). The directions differ: observed loss is 10% above predicted loss in the urban group and 10% below it in the rural group.
Across both segments, total observed loss is \(13{,}200+7{,}200=\$20{,}400\), and total predicted loss is \(12{,}000+8{,}000=\$20{,}000\). The overall ratio is \(20{,}400/20{,}000=1.02\).
Check. The overall ratio is near 1, but it does not mean both segments are near 1: the segment ratios are 1.10 and 0.90. A report should show the segment results alongside the overall result. These totals indicate differences between observed and predicted amounts in this evaluation; they do not establish that location caused either difference.
Common mistake avoided: Treating the overall ratio of 1.02 as evidence of close alignment in every segment would overlook the opposing segment-level results. The overall ratio combines totals and can conceal such differences.
What a Segment Result Supports
A sound interpretation separates the observed result from an explanation for it. “The evaluated MAE was $6 in South and $2 in North” reports a comparison. “The model performs worse in South because of the region” makes a causal claim that these summaries alone do not establish. Differences may reflect variation in the records, the outcome mix, data quality, or other factors; segment summaries do not identify which explanation applies.
State the scope precisely. For example, say “among the held-out claims evaluated during the stated period” rather than implying that the result applies to every future claim. If the segment definitions overlap, or some records cannot be assigned to a segment, say so; simple aggregation identities assume a complete, non-overlapping partition.
When a gap meets a pre-established review criterion, the report can recommend examination of the relevant segment results or operational context. It should not convert a descriptive difference into an unsupported instruction to change a model, a process, or a group's treatment. The evidence presented should support the strength of the recommendation.
This discipline is aligned with the SOA Exam PA (Predictive Analytics) syllabus: report the comparison in context, provide the counts needed to interpret it, and distinguish what the evaluation shows from what remains uncertain or unexplained.
Common Reporting Traps
- Using inconsistent conditions. Comparing different thresholds, observation windows, or outcome definitions can make a segment gap uninterpretable as a like-for-like comparison.
- Omitting the metric denominator. Report the number of evaluated records for an error average and the number of actual positive cases for recall.
- Averaging segment metrics without the right weights. For an unweighted overall MAE, weight segment MAEs by their record counts. Other metrics may require different aggregation and should not automatically be averaged the same way.
- Relying on the overall result alone. A pooled summary can conceal meaningfully different segment results, including differences in opposite directions.
- Claiming a cause from a descriptive comparison. A segment gap identifies where results differ in the evaluated data, not why they differ.
Check Your Understanding
For each problem, calculate or interpret the segment comparison using the stated metric and denominator.
- A numeric model has absolute errors of 2, 4, and 6 in Segment A, and 3 and 9 in Segment B. Calculate both segment MAEs and the overall MAE. (Answer: Segment A MAE = 4; Segment B MAE = 6; overall MAE = 4.8.)
- At one fixed threshold, Segment P has 18 true positives and 12 false negatives; Segment Q has 21 true positives and 9 false negatives. Calculate recall in each segment and the difference. (Answer: P = 0.6000; Q = 0.7000; difference = 0.1000.)
- Two disjoint segments have counts of 8 and 2, with MAEs of 5 and 15, respectively. Calculate the overall MAE and explain why the unweighted average of the two segment MAEs is wrong. (Answer: Overall MAE = 7; the unweighted average is 10 and gives equal weight to segments with different record counts.)
- A model's observed and predicted totals are $4,800 and $4,000 in Segment A, and $3,200 and $4,000 in Segment B. Calculate both observed-to-predicted ratios and the overall ratio. (Answer: A = 1.20; B = 0.80; overall = 1.00.)
- A report finds that a model's recall is lower in one channel than another using the same threshold and evaluation period. Does this alone show that the channel caused the difference? State what the result supports. (Answer: No. It supports a descriptive statement that recall differed in the evaluated records; it does not establish the cause.)