Validation Performance Across Rating Segments

Why Validate Within Rating Segments?

A validation portfolio can look well calibrated overall even when predicted loss costs are materially too high for one territory and too low for another. Because opposite errors can offset, an aggregate result alone does not show whether a model performs appropriately across the rating segments in which it will be used. Segment-level validation makes those errors visible.

Calibration and predictive loss answer different questions. Calibration asks whether predicted loss costs agree with observed loss costs on average. Predictive loss measures the size of prediction errors, including errors that cancel in an average. Both should be calculated on a consistent exposure and loss basis, using validation data whose period and maturity are appropriate for the intended assessment.

Definition: For validation record $i$, let $e_i$ be its exposure, $y_i$ its observed loss cost per unit of exposure, and $\hat{y}_i$ its predicted loss cost on the same basis. For a segment $g$, its total exposure is $E_g=\sum_{i\in g}e_i$. Its exposure-weighted observed and predicted loss costs are $\bar{y}_g=\sum_{i\in g}e_i y_i/E_g$ and $\bar{\hat{y}}_g=\sum_{i\in g}e_i\hat{y}_i/E_g$. Define the calibration difference as $\bar{y}_g-\bar{\hat{y}}_g$; a positive value means observed loss cost exceeds predicted loss cost.

For a portfolio-level check, use the same formulas over all records. The exposure-weighted mean squared error (MSE) in a segment is $\sum_{i\in g}e_i(y_i-\hat{y}_i)^2/E_g$; the mean absolute error (MAE) replaces the squared residual with its absolute value. Both measures are nonnegative, and smaller values indicate smaller errors on the chosen scale. MSE gives relatively more weight to large errors; MAE is less sensitive to them.

These formulas treat each record's $y_i$ and $\hat{y}_i$ as the observed and predicted rates for that record. If records are aggregated cells, the resulting MSE is a cell-rate error measure: it does not include variation among individual policies within a cell. State the unit of exposure and rate, and do not compare scores calculated on different scales or bases.

How Aggregate Calibration Can Conceal Segment Errors

Let $d_g=\bar{y}_g-\bar{\hat{y}}_g$ be the calibration difference for segment $g$, and let $w_g=E_g/\sum_h E_h$ be its share of total exposure. The portfolio calibration difference is the exposure-weighted average of the segment differences:

\bar{y}-\bar{\hat{y}} =\frac{\sum_g E_g\bar{y}_g}{\sum_g E_g} -\frac{\sum_g E_g\bar{\hat{y}}_g}{\sum_g E_g} =\sum_g w_g d_g.

The equality follows because each segment's observed and predicted totals are its exposure multiplied by its corresponding rate. Subtracting the portfolio totals therefore adds the exposure-weighted segment differences. A positive segment difference can be offset by a negative one; the portfolio difference can be zero even when neither segment is calibrated.

Squared errors do not cancel in this way. A positive residual and a negative residual both contribute a positive squared term to MSE. This is why calibration and predictive loss should be reported separately, both overall and by material segments.

Worked Example: Opposite Errors Across Two Territories

A hypothetical insurer validates a loss-cost model in two territories. Each territory has 100 policy-years. In the North, observed and predicted loss costs are $1,200 and $1,000 per policy-year. In the South, they are $800 and $1,000 per policy-year. Calculate the segment and portfolio calibration differences and MSEs.

Method. Calculate observed minus predicted loss cost for calibration, then square each record's rate error for MSE. Since there is one rate record for each territory, each segment's MSE is the square of its territory's error.

North. Its calibration difference is $1,200 − $1,000 = $200 per policy-year. Its rate error is $200, so its MSE is $(\$200)^2=\$40{,}000$ $(\text{dollars per policy-year})^2$.

South. Its calibration difference is $800 − $1,000 = −$200 per policy-year. Its rate error is −$200, so its MSE is also $(-\$200)^2=\$40{,}000$ $(\text{dollars per policy-year})^2$.

Portfolio calibration. The exposure-weighted observed rate is $[100(\$1{,}200)+100(\$800)]/200=\$1{,}000$ per policy-year. The predicted rate is $[100(\$1{,}000)+100(\$1{,}000)]/200=\$1{,}000$. The portfolio calibration difference is therefore $0 per policy-year.

Portfolio MSE. It is $[100(\$200)^2+100(-\$200)^2]/200=\$40{,}000$ $(\text{dollars per policy-year})^2$. The zero portfolio calibration difference does not mean the predictions are error-free: each territory is miscalibrated by $200 per policy-year. The common mistake is to treat a zero aggregate difference as evidence that every segment is calibrated.

Calculating Loss Within and Across Segments

A segment calibration difference summarizes the signed average error. Its MSE and MAE summarize error magnitude. The following example shows why it is important to calculate each measure from the underlying records before combining segments.

Worked Example: Exposure-Weighted Calibration and MSE

A hypothetical validation has two segments. Segment A contains a record with 20 policy-years, observed loss cost $1,000, and predicted loss cost $800, plus a record with 80 policy-years, observed loss cost $900, and predicted loss cost $1,000. Segment B contains two records of 50 policy-years each: observed and predicted loss costs are $1,200 and $1,100 for the first, and $800 and $900 for the second. Calculate each segment's calibration difference and MSE, then calculate the portfolio calibration difference, MSE, and MAE.

Segment A observed rate. Its observed loss total is $20(\$1{,}000)+80(\$900)=\$20{,}000+\$72{,}000=\$92{,}000$. Dividing by 100 policy-years gives $920 per policy-year.

Segment A predicted rate and calibration. Its predicted total is $20(\$800)+80(\$1{,}000)=\$16{,}000+\$80{,}000=\$96{,}000$, or $960 per policy-year. The calibration difference is $920 − $960 = −$40 per policy-year.

Segment A MSE. The two errors are $200 and −$100. Therefore, its MSE is $[20(\$200)^2+80(-\$100)^2]/100=[20(\$40{,}000)+80(\$10{,}000)]/100=(\$800{,}000+\$800{,}000)/100=\$16{,}000$ $(\text{dollars per policy-year})^2$.

Segment B. Its observed total is $50(\$1{,}200)+50(\$800)=\$100{,}000$, and its predicted total is $50(\$1{,}100)+50(\$900)=\$100{,}000$. Each rate is $1,000 per policy-year, so the calibration difference is $0. The two errors are $100 and −$100, giving MSE $[50(\$100)^2+50(-\$100)^2]/100=\$10{,}000$ $(\text{dollars per policy-year})^2$.

Portfolio calibration. Total observed loss is $92,000 + $100,000 = $192,000; total predicted loss is $96,000 + $100,000 = $196,000. With 200 policy-years, the rates are $960 and $980 per policy-year, respectively, so the portfolio calibration difference is −$20 per policy-year.

Portfolio MSE and MAE. The portfolio MSE is $[100(\$16{,}000)+100(\$10{,}000)]/200=\$13{,}000$ $(\text{dollars per policy-year})^2$. For MAE, Segment A contributes $20(\$200)+80(\$100)=\$12{,}000$ in exposure-weighted absolute error, while Segment B contributes $50(\$100)+50(\$100)=\$10{,}000$. Thus MAE is $(\$12{,}000+\$10{,}000)/200=\$110$ per policy-year. As a check, the segment calibration differences combine to $[100(-\$40)+100(\$0)]/200=-\$20$, the portfolio difference calculated above. The common mistake is to use an unweighted average of record rates or to calculate MSE from the segment's average error alone.

Why Broad Segments May Still Hide Errors

A territory-level or class-level check can itself conceal errors if the model is wrong for combinations of those factors. For example, overprediction for one class and underprediction for another may offset within each territory. Cross-tabulated segments can expose that pattern, provided the combinations have enough exposure to support a useful assessment.

The following example checks territory and class separately, then checks their intersections. It shows that both sets of broad marginal summaries can be calibrated even though each combined cell has a substantial error.

Worked Example: Checking Territory-Class Intersections

A hypothetical validation contains four cells, each with 50 policy-years. The observed and predicted loss costs are: Urban, Class A, $1,500 and $1,300; Urban, Class B, $1,100 and $1,300; Rural, Class A, $900 and $1,100; Rural, Class B, $1,300 and $1,100. Calculate calibration differences for each territory and class, and calculate the cell-level MSE.

Territories. For Urban, the observed rate is $[50(\$1{,}500)+50(\$1{,}100)]/100=\$1{,}300$; the predicted rate is $[50(\$1{,}300)+50(\$1{,}300)]/100=\$1{,}300$. Its difference is $0. For Rural, the observed rate is $[50(\$900)+50(\$1{,}300)]/100=\$1{,}100$, and the predicted rate is $[50(\$1{,}100)+50(\$1{,}100)]/100=\$1{,}100$. Its difference is also $0.

Classes. For Class A, observed loss is $50(\$1{,}500)+50(\$900)=\$120{,}000$, and predicted loss is $50(\$1{,}300)+50(\$1{,}100)=\$120{,}000$. Both rates are $1,200 per policy-year. For Class B, observed loss is $50(\$1{,}100)+50(\$1{,}300)=\$120{,}000$, and predicted loss is $50(\$1{,}300)+50(\$1{,}100)=\$120{,}000$, again giving a $0 difference.

Intersections and MSE. The four cell errors are $200, −$200, −$200, and $200. Each cell has the same exposure, so cell-level MSE is $[50(\$200)^2+50(-\$200)^2+50(-\$200)^2+50(\$200)^2]/200=\$40{,}000$ $(\text{dollars per policy-year})^2$. The territory and class averages conceal the opposing errors that appear in the cross-tabulation. The common mistake is to infer good performance in every territory-class combination from good performance in each factor considered separately.

Practical Checks and Interpretation

Segment results are useful only when comparisons are defined consistently. Before interpreting them, verify that observed and predicted amounts use the same coverage, valuation period, loss development basis, and exposure unit. Confirm that each record is assigned to the segment using the rating characteristics relevant to the validation question. Exposure weights should reflect the exposure represented by the record, not an arbitrary count of rows.

A segment report can include exposure, observed loss cost, predicted loss cost, calibration difference, and one or more predictive loss measures. A calibration ratio, when predicted loss cost is positive, is observed loss cost divided by predicted loss cost; a ratio of 1 indicates calibration, a ratio above 1 indicates observed exceeds predicted, and a ratio below 1 indicates observed is below predicted. State whether a difference is observed minus predicted or the reverse, since the sign convention matters.

Very small segments can produce volatile observed rates, and a large number of segment cuts can create a misleading impression of meaningful patterns. Interpret the magnitude of an error in context, inspect the exposure supporting it, and prioritize segments that are material to rating or business use. Segment validation diagnoses where aggregate performance may be inadequate; it does not by itself establish why an error occurs or whether a rating change is appropriate.

Key takeaway: Report calibration and predictive loss separately, and calculate both at the overall level and for material rating segments. Weighted segment calibration differences must reconcile to the portfolio difference, while offsetting errors can make that portfolio difference small or zero.

Check Your Understanding

For each problem, use exposure-weighted rates and define calibration difference as observed minus predicted.

A validation has 40 policy-years in East, with observed and predicted rates of $1,500 and $1,300, and 60 policy-years in West, with rates of $1,000 and $1,100. Find each territory's calibration difference and the portfolio difference. (Answer: East $+\$200$, West $-\$100$, and portfolio $+\$20$ per policy-year.)
One segment has 30 policy-years with observed and predicted rates of $1,000 and $900, and 70 policy-years with rates of $800 and $850. Calculate the segment calibration difference and exposure-weighted MSE. (Answer: difference $-\$5$ per policy-year; MSE $\$4{,}750$ $(\text{dollars per policy-year})^2$.)
Two equally exposed segments each have 50 policy-years. Segment A has observed and predicted rates of $1,200 and $1,000; Segment B has rates of $800 and $1,000. Find the portfolio calibration difference and MSE. (Answer: difference $\$0$ per policy-year; MSE $\$40{,}000$ $(\text{dollars per policy-year})^2$.)
A segment has 25 policy-years with observed and predicted rates of $1,400 and $1,200, and 75 policy-years with rates of $1,000 and $1,200. Calculate its calibration difference and MSE. (Answer: difference $-\$100$ per policy-year; MSE $\$40{,}000$ $(\text{dollars per policy-year})^2$.)
Two equally exposed class-territory cells have errors of $+\$150$ and $-\$150$ per policy-year, where error means observed minus predicted. What are the equally weighted aggregate calibration difference and MSE? (Answer: calibration difference $\$0$ per policy-year; MSE $\$22{,}500$ $(\text{dollars per policy-year})^2$.)

What You'll Learn