Comparing Forecasts Across Portfolio Segments

Why Compare Forecasts by Segment?

An overall error score is useful, but it can conceal differences in how models perform across a portfolio. A forecasting model may be accurate for the largest segment and noticeably less accurate for a smaller one. A second model may reverse that pattern. The preferred model can therefore depend on both the portfolio’s segment mix and the purpose of the forecast.

This tutorial compares numeric predictions for the same target on the same evaluation data. It uses mean squared error (MSE) and root mean squared error (RMSE) to compare models overall and within segments. The key is to keep the evaluation population, error definition, and weighting rule consistent, then state clearly how segment-level trade-offs affect the decision.

Definition: For observation \(i\), let \(y_i\) be the observed numeric outcome, \(\hat{y}_{im}\) the prediction from model \(m\), and \(w_i\) a nonnegative evaluation weight. The weighted MSE for model \(m\) is \(\sum_i w_i(y_i-\hat{y}_{im})^2/\sum_i w_i\), and its RMSE is the square root of that MSE. Smaller MSE and RMSE indicate smaller squared prediction errors on the specified evaluation data and weighting scheme.

A segment is a defined group of observations, such as a product type or region. Suppose the evaluation data are divided into \(S\) nonoverlapping segments. Let \(W_s\) be the sum of the weights in segment \(s\), and let \(\operatorname{MSE}_{ms}\) be model \(m\)'s weighted MSE within that segment. The overall MSE is a weighted average of the segment MSEs, with each segment weighted by its share of total evaluation weight.

\operatorname{MSE}_{m,\mathrm{overall}} = \frac{\sum_{s=1}^{S} W_s\operatorname{MSE}_{ms}}{\sum_{s=1}^{S}W_s} = \sum_{s=1}^{S} q_s\operatorname{MSE}_{ms}, \qquad q_s=\frac{W_s}{\sum_{r=1}^{S}W_r}.

When every observation has equal weight, \(W_s\) is simply the number of evaluation observations in segment \(s\). When observations have different positive exposure weights, \(W_s\) is the sum of those exposures. The weights must match the intended evaluation target: a result weighted by policy exposure answers a different question from one that gives every segment equal influence.

Why the Overall Formula Holds

The segment formula follows directly from grouping the weighted squared errors. Within segment \(s\), its MSE is the sum of its weighted squared errors divided by \(W_s\). Multiplying that segment MSE by \(W_s\) recovers the segment’s total weighted squared error. Summing those totals across segments gives the overall weighted squared error, and dividing by total weight gives overall MSE.

W_s\operatorname{MSE}_{ms} = \sum_{i\in s}w_i(y_i-\hat{y}_{im})^2

\frac{\sum_s W_s\operatorname{MSE}_{ms}}{\sum_s W_s} = \frac{\sum_s\sum_{i\in s}w_i(y_i-\hat{y}_{im})^2}{\sum_i w_i} = \operatorname{MSE}_{m,\mathrm{overall}}.

This identity depends on the segments partitioning the evaluation observations: each observation must be counted exactly once. It also depends on using the same outcome, prediction scale, evaluation data, and weighting definition in the segment and overall calculations. RMSE is then the square root of the pooled MSE. In general, it is not the weighted average of segment RMSEs.

Worked Portfolio Comparisons

Worked Example: Different Winners in Different Segments

A hypothetical insurer evaluates two models on the same held-out policy records. The portfolio has 80 exposure units in the urban segment and 20 in the rural segment. Model P has segment MSEs of 4 in urban and 49 in rural; Model Q has segment MSEs of 9 in urban and 16 in rural. Compare the models within each segment and overall using exposure-weighted RMSE.

Method. Calculate each segment’s RMSE by taking the square root of its MSE, then calculate the overall MSE using exposure shares \(80/100=0.8\) and \(20/100=0.2\). Take the square root only after pooling the MSEs.

Urban segment. Model P has RMSE \(\sqrt{4}=2\). Model Q has RMSE \(\sqrt{9}=3\). P has lower error in this segment.

Rural segment. Model P has RMSE \(\sqrt{49}=7\). Model Q has RMSE \(\sqrt{16}=4\). Q has lower error in this segment.

Overall MSE for P. The urban contribution is \(0.8(4)=3.2\); the rural contribution is \(0.2(49)=9.8\). Thus P’s overall MSE is \(3.2+9.8=13.0\), and its overall RMSE is \(\sqrt{13.0}\approx3.606\).

Overall MSE for Q. The urban contribution is \(0.8(9)=7.2\); the rural contribution is \(0.2(16)=3.2\). Thus Q’s overall MSE is \(7.2+3.2=10.4\), and its overall RMSE is \(\sqrt{10.4}\approx3.225\).

Decision and check. Q has lower exposure-weighted overall RMSE, even though P is better in the larger urban segment. The rural errors matter because P’s rural MSE is much larger. The pooled MSEs are 13.0 and 10.4, both between their corresponding segment MSEs, as expected for weighted averages. This comparison avoids choosing a model solely from its performance in the largest segment.

Worked Example: Finding the Portfolio-Mix Threshold

A hypothetical service portfolio has two segments. Let \(p\) be the share of total exposure in Segment 1, so Segment 2 has share \(1-p\). Model X has segment MSEs 4 and 36 in Segments 1 and 2, respectively. Model Y has segment MSEs 9 and 16. Find when X has lower overall RMSE than Y.

Step 1: Write the pooled MSEs. Model X has overall MSE \(4p+36(1-p)=36-32p\). Model Y has overall MSE \(9p+16(1-p)=16-7p\).

Step 2: Compare MSEs. Because square root is strictly increasing for nonnegative numbers, X has lower RMSE exactly when its MSE is lower: \(36-32p<16-7p\). Rearranging gives \(20<25p\), or \(p>20/25=4/5=0.8\). The models tie at \(p=0.8\); for smaller shares, Y has the lower pooled RMSE.

Step 3: Check at a particular mix. If \(p=0.85\), X’s MSE is \(4(0.85)+36(0.15)=3.4+5.4=8.8\). Y’s MSE is \(9(0.85)+16(0.15)=7.65+2.4=10.05\). X is lower, consistent with \(0.85>0.8\). The corresponding RMSEs are approximately \(\sqrt{8.8}=2.966\) and \(\sqrt{10.05}=3.170\).

Decision and check. X is preferable by pooled RMSE only when Segment 1 contributes more than 80% of total exposure. At exactly 80%, both have MSE \(10.4\): X gives \(4(0.8)+36(0.2)=10.4\), and Y gives \(9(0.8)+16(0.2)=10.4\). This calculation avoids treating a model ranking as independent of portfolio composition.

Worked Example: Combining an Overall Ranking With a Segment Requirement

A hypothetical portfolio has Segment A with 40% of evaluation exposure and Segment B with 60%. Model M has segment MSEs 9 and 25; Model N has segment MSEs 16 and 16. Compare their pooled RMSEs, then apply a stated requirement that each segment’s RMSE must be no greater than 4.5.

Model M’s overall MSE. Segment A contributes \(0.4(9)=3.6\), and Segment B contributes \(0.6(25)=15.0\). Its overall MSE is \(3.6+15.0=18.6\), giving an overall RMSE of \(\sqrt{18.6}\approx4.313\).

Model N’s overall MSE. Segment A contributes \(0.4(16)=6.4\), and Segment B contributes \(0.6(16)=9.6\). Its overall MSE is \(6.4+9.6=16.0\), giving an overall RMSE of \(\sqrt{16}=4.000\). N has the lower pooled RMSE.

Apply the segment requirement. M’s segment RMSEs are \(\sqrt{9}=3\) in A and \(\sqrt{25}=5\) in B. It fails the 4.5 limit in B. N’s segment RMSE is \(\sqrt{16}=4\) in both segments, so it meets the limit in both.

Decision and check. N is the appropriate choice under the stated rule: it has lower overall RMSE and satisfies the segment cap, whereas M does not. The pooled MSEs also check as weighted averages: 18.6 lies between 9 and 25, and 16 lies between the two segment values of 16. A requirement such as the 4.5 cap must be specified as part of the decision rule; it is not implied by RMSE itself.

Choosing the Weighting Rule and Interpreting Trade-Offs

An exposure-weighted overall score represents the average squared error for a randomly selected unit of the specified exposure, provided the weights correspond to that unit. It answers, “How does the model perform across this portfolio as it is currently composed?” If the portfolio mix changes, the overall ranking can change even if every segment’s error remains the same.

An equal-weight segment summary instead gives each segment the same influence, regardless of its size. For example, averaging segment MSEs answers a different question from pooling all exposure-weighted errors. It can be useful when the decision-maker regards each segment as equally important, but it should not be presented as the portfolio’s exposure-weighted error. State whether the summary averages observations, exposures, or segments.

A defensible comparison generally reports the overall metric alongside segment-level metrics. The overall number describes aggregate performance under a specified weighting scheme; the segment results show where the aggregate advantage comes from and whether it masks a weaker outcome in a particular group. If a segment has a minimum performance requirement, evaluate that requirement directly rather than assuming a favorable overall score guarantees it.

The metrics do not determine the decision by themselves. The decision-maker must define the relevant evaluation population, the weights, any segment constraints, and the consequences of errors. RMSE penalizes larger errors more heavily because it squares them before averaging, but the comparison here does not establish that a model is suitable for every operational purpose. Both models must be scored on the same evaluation records and target definition for the ranking to be meaningful.

Common Mistakes

Averaging segment RMSEs to get pooled RMSE. In the first example, exposure-weighting the segment RMSEs for P gives \(0.8(2)+0.2(7)=3\). That is not the pooled RMSE; the correct value is \(\sqrt{0.8(4)+0.2(49)}=\sqrt{13}\approx3.606\). Pool MSEs first, then take the square root.
Assuming the overall winner is best in every segment. The first example has different segment winners. Report both the pooled result and segment results.
Using equal segment weights without saying so. An equal-weight segment summary can differ from the exposure-weighted portfolio result. Name the weighting convention and the question it answers.
Comparing metrics computed on different records. If one model is evaluated on a different set of outcomes, the difference may reflect the evaluation population rather than model performance. Compare both models on the same eligible observations.
Ignoring decision requirements. A low pooled error does not guarantee compliance with a segment-specific limit. Check explicit constraints separately.

Key takeaway: Compare forecasts using the same outcomes and a stated weighting rule. Pool segment MSEs according to their shares of evaluation weight, take the square root afterward, and use segment-level results to make trade-offs and enforce any explicit requirements.

Check Your Understanding

For each problem, identify the weighting rule before comparing the models. Report RMSEs to three decimal places when needed.

A portfolio has 75 exposure units in Segment A and 25 in Segment B. Model R has segment MSEs 4 and 25; Model S has segment MSEs 9 and 16. Which model has the lower exposure-weighted overall RMSE, and what are the two RMSEs? (Answer: R has MSE \(9.25\) and RMSE \(\sqrt{9.25}\approx3.041\); S has MSE \(10.75\) and RMSE \(\sqrt{10.75}\approx3.279\); R is lower.)
Let \(p\) be Segment 1’s exposure share. Model C has segment MSEs 1 and 25; Model D has segment MSEs 4 and 9. For what values of \(p\) does C have lower pooled RMSE? (Answer: \(p>16/19\approx0.8421\); the pooled RMSEs tie at \(p=16/19\).)
Segment A has 20% of exposure and Segment B has 80%. Model R has MSEs 4 and 16; Model S has MSEs 9 and 9. Calculate both overall RMSEs and select the lower-error model. (Answer: R has MSE \(0.2(4)+0.8(16)=13.6\), RMSE \(\sqrt{13.6}\approx3.688\); S has MSE \(9\), RMSE \(3.000\); S is lower.)
A portfolio has 90% exposure in Segment 1 and 10% in Segment 2. Model A has segment MSEs 1 and 100; Model B has segment MSEs 9 and 16. Calculate each exposure-weighted overall MSE and the equal-weight average of its two segment MSEs. (Answer: A has overall MSE \(0.9(1)+0.1(100)=10.9\) and equal-weight segment average \((1+100)/2=50.5\); B has overall MSE \(0.9(9)+0.1(16)=9.7\) and equal-weight average \((9+16)/2=12.5\). B is lower under both summaries, but the summaries have different weighting interpretations.)
Segment 1 has 90% of exposure and Segment 2 has 10%. Model U has segment MSEs 1 and 36; Model V has segment MSEs 4 and 16. Which model has lower pooled RMSE, and which satisfies a requirement that every segment RMSE be at most 5? (Answer: U has pooled MSE \(0.9(1)+0.1(36)=4.5\) and RMSE \(\sqrt{4.5}\approx2.121\); V has pooled MSE \(0.9(4)+0.1(16)=5.2\) and RMSE \(\sqrt{5.2}\approx2.280\), so U has lower pooled RMSE. U fails the segment requirement because its Segment 2 RMSE is 6; V meets it because its segment RMSEs are 2 and 4.)

What You'll Learn