Average Performance Across Business Segments

Why an Overall Metric May Not Be Enough

A model can meet an overall performance target and still produce predictions that are consistently less useful for one part of a business. For example, a numeric prediction model may have low average error across all customers while having much larger errors for a small customer segment. Similarly, a classification model may identify most positive cases overall while missing a much larger share in a particular segment.

An overall metric answers a portfolio-wide question: how well did the model perform across the evaluated observations, under the chosen metric and weighting? It does not, by itself, answer whether the model performs acceptably for every segment. The evaluation plan should therefore specify both the overall criterion and any segment-level requirements that matter for the intended use. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

A segment should be defined in a way that is relevant to the decision being evaluated and can be identified at the time the model is used. Segment results are meaningful only when the metric has the same definition in each group. For example, comparing average absolute errors is not informative if one segment's errors are measured in dollars and another's in thousands of dollars.

Combining Segment Results for Numeric Predictions

For a numeric prediction, let \(y_i\) be the observed value and \(\hat{y}_i\) the prediction for observation \(i\). Its prediction error is \(e_i=y_i-\hat{y}_i\). The mean absolute error (MAE) is the average of \(|e_i|\), and the root mean squared error (RMSE) is the square root of the average of \(e_i^2\). These definitions apply to the same evaluation observations and use equal weight per observation. If the evaluation instead assigns observation weights, those weights must be used consistently in both the overall and segment calculations.

Suppose the evaluation population is partitioned into segments indexed by \(g\), with \(n_g\) observations in segment \(g\). Let \(\operatorname{MAE}_g\) be that segment's MAE. When each observation has equal weight, the overall MAE is the observation-count-weighted average of segment MAEs:

This identity holds because the sum of absolute errors in segment \(g\) is \(n_g\operatorname{MAE}_g\). Adding those segment sums gives the total absolute error; dividing by the total number of observations gives the overall MAE. Thus, averaging segment MAEs equally is generally wrong when segment sizes differ.

The reason is that squared errors, rather than their square roots, add across observations. The formula uses the fact that \(\operatorname{RMSE}_g^2\) is the mean squared error in segment \(g\). It assumes the same squared-error definition and equal observation weights throughout. With unequal weights, combine each segment's weighted mean squared error using its total evaluation weight, then take the square root.

Combining Classification Results Carefully

Classification metrics also have different denominators. For example, sensitivity in a segment is the number of positive cases correctly classified as positive divided by the total number of actual positive cases in that segment. Consequently, overall sensitivity is weighted by the segment's number of actual positive cases, not necessarily by its total number of observations.

This follows because \(P_g\operatorname{Sens}_g\) is the number of true positives in segment \(g\). Adding the true positives across segments and dividing by all actual positive cases gives the pooled sensitivity. The analogous weighting principle applies to other rates: identify the denominator that defines the metric, then weight by that denominator. A segment's total size is not automatically the correct weight.

Deciding What Segment Results Mean

Segment analysis is not a search for a single universally correct summary. It is a way to check whether the chosen overall success criterion is appropriate for the actual use of the predictions. A pooled metric is useful when the intended decision genuinely values performance across the combined population under its chosen weights. Segment-level requirements matter when the business has a reason to require acceptable performance in each specified group.

The evaluation plan should make that distinction explicit. A practical sequence is:

A small segment deserves attention, but its metric may be less stable because it is based on fewer observations. Report its size and interpret its result cautiously; do not silently remove it or treat a noisy estimate as definitive. Likewise, an observed gap is a reason to investigate suitability, not proof of its cause. Differences can reflect the model, the data, the evaluation sample, or a combination of factors.

Avoid creating many arbitrary segments after looking at the results and then presenting only the most favorable or unfavorable comparisons. Define important segments in advance where possible, based on intended use. If exploratory segment checks are also shown, label them as such. This keeps the overall success criterion interpretable and makes the evaluation more transparent.

Common Traps

• Averaging segments equally when their sizes differ. For observation-level MAE, weight by observation counts. For sensitivity, weight by actual positive counts.
• Averaging RMSE values directly. Combine mean squared errors first, then take the square root.
• Assuming a good overall result guarantees good subgroup results. Check any segment-level limits that matter to the intended use.
• Comparing unlike metrics. Ensure units, denominators, and rules are consistent before interpreting a gap.
• Ignoring segment sizes. Report them so readers can interpret both the contribution to the overall metric and the evidence behind each segment result.