Performance Variation Across Risk Segments

Why Evaluate Experience Rating by Segment?

An experience-rating plan is evaluated using outcomes across a group of risks, but a single portfolio result can conceal important differences within that group. A plan may be well calibrated overall while persistently underpredicting losses in one industry and overpredicting them in another. It may also produce stable modifications for large accounts but volatile modifications for small accounts. Those distinctions matter when assessing whether performance is consistent across the risks to which the plan is applied.

This tutorial focuses on segment-level evaluation: compare predictive performance and modification stability across groups defined in advance, such as exposure size, industry, or claim profile. The purpose is not to replace the overall evaluation, but to see whether the overall result is representative. Comparisons should use consistent definitions of the predicted and observed outcomes, and should report the amount of experience behind each segment result. A segment with little experience can show an extreme result because of random variation.

Definition: For segment $s$, let $P_s$ be the total predicted outcome for the evaluation period and $O_s$ the corresponding observed outcome, measured on the same basis. When $P_s>0$, the segment observed-to-predicted ratio is $R_s=O_s/P_s$. A ratio above 1 indicates that observed outcomes exceeded predictions; a ratio below 1 indicates that they fell below predictions. A ratio of 1 is aggregate calibration for that segment, not proof that every individual risk was predicted accurately.

For an experience-rating evaluation, the predicted quantity might be expected losses, or an outcome implied by the evaluated modifications, provided the comparison is defined consistently. The observed quantity must match it: for example, actual losses on the same coverage and evaluation basis as the predictions. A comparison between predicted losses and premium, or between quantities with different exposure periods, does not measure calibration.

Useful segment dimensions include exposure size, industry, and claim profile. Define the segments before reviewing results where practical. If definitions are selected after seeing the results, unusually favorable or unfavorable groups may be found by chance. Also state whether groups overlap: if one risk can appear in several segments, segment totals cannot simply be added to obtain a portfolio total.

Aggregate Calibration Can Hide Segment Errors

The portfolio ratio is determined by total observed outcomes divided by total predicted outcomes. It is therefore a prediction-weighted average of segment ratios, rather than an unweighted average of them. This identity explains how overprediction in one segment can offset underprediction in another.

R_{\mathrm{portfolio}} = \frac{\sum_s O_s}{\sum_s P_s} = \frac{\sum_s P_s R_s}{\sum_s P_s}, \qquad P_s>0.

The equality follows by substituting $O_s=P_sR_s$ into the numerator. Each segment's contribution to the portfolio ratio is weighted by its predicted amount. A small segment with a large ratio may have limited influence on the total; a large segment with a modest error may have a substantial influence. Consequently, an overall ratio near 1 does not establish that all segments are well calibrated.

Worked Example: Offsetting Errors Across Industries

A hypothetical insurer evaluates experience-rating predictions for two industry segments over the same period. In Segment A, predicted losses are $400,000 and observed losses are $500,000. In Segment B, predicted losses are $600,000 and observed losses are $500,000. Calculate each segment's observed-to-predicted ratio and the portfolio ratio. State what the aggregate result does and does not show.

Method. Calculate observed divided by predicted separately for each segment, then calculate the portfolio ratio from the summed amounts. This avoids treating the portfolio result as evidence that each segment is calibrated.

Segment A. The ratio is $500{,}000/400{,}000=1.25$. Observed losses exceed predicted losses by $500{,}000-400{,}000=\$100{,}000$, or 25% of the prediction.

Segment B. The ratio is $500{,}000/600{,}000=0.8333$ to four decimals. Observed losses are $600{,}000-500{,}000=\$100{,}000$ below the prediction, or about 16.67% of the prediction.

Portfolio. Total predicted losses are $400{,}000+600{,}000=\$1{,}000{,}000$. Total observed losses are $500{,}000+500{,}000=\$1{,}000{,}000$. Thus the portfolio ratio is $1{,}000{,}000/1{,}000{,}000=1.00$.

Check and interpretation. Segment A's $100,000 excess and Segment B's $100,000 shortfall cancel, producing portfolio calibration. The aggregate ratio does not show that the plan is calibrated in either segment. The common mistake is to report only the portfolio ratio of 1.00; the correction is to report the segment ratios and their predicted and observed amounts alongside it.

Calibration and Prediction Error Answer Different Questions

Calibration compares totals: do observed outcomes, in aggregate, match predictions? Prediction-error measures ask how far individual or smaller-group outcomes are from their predictions. A segment can have a ratio of 1 while individual predictions vary substantially above and below observed outcomes. Conversely, a segment's aggregate ratio can differ from 1 even if its individual errors are fairly similar in size.

For $n$ risks with predictions $P_j$ and observed outcomes $O_j$, define the signed error $e_j=O_j-P_j$. The mean absolute error is the average of $|e_j|$; the root mean squared error is the square root of the average $e_j^2$. These measures are expressed in the outcome's units and give greater influence to large errors, especially the squared-error measure. A relative-error measure, such as the average of $|O_j-P_j|/P_j$ when every $P_j>0$, puts errors on a percentage scale, but can become large for small predictions. State the chosen measure and its limitations rather than comparing unlike metrics as if they were interchangeable.

Worked Example: Good Aggregate Calibration, Different Error Profiles

A hypothetical review compares two exposure-size segments. Loss amounts and predictions are in thousands of dollars. In the small-exposure segment, the three predictions are $10,10,10$, and the corresponding observed losses are $0,10,20$. In the large-exposure segment, predictions are $100,100,100$, and observed losses are $120,90,90$. For each segment calculate the aggregate ratio, mean absolute error, root mean squared error, and mean absolute percentage error. Give dollar errors in thousands and percentages to two decimals.

Method. For each risk, calculate $e_j=O_j-P_j$. Then compute $O/P$ using segment totals, $\sum |e_j|/3$, $\sqrt{\sum e_j^2/3}$, and the average of $|e_j|/P_j$. No prediction is zero, so the percentage errors are defined.

Small-exposure segment. The errors are $0-10=-10$, $10-10=0$, and $20-10=10$. Total observed and predicted losses are both 30, so the aggregate ratio is $30/30=1.00$. The mean absolute error is $(10+0+10)/3=20/3=6.67$ thousand dollars. The root mean squared error is $\sqrt{(100+0+100)/3}=\sqrt{200/3}=8.16$ thousand dollars. The percentage errors are $10/10=100\%$, $0/10=0\%$, and $10/10=100\%$, so the mean absolute percentage error is $200\%/3=66.67\%$.

Large-exposure segment. The errors are $120-100=20$, $90-100=-10$, and $90-100=-10$. Total observed and predicted losses are both 300, giving an aggregate ratio of $300/300=1.00$. The mean absolute error is $(20+10+10)/3=13.33$ thousand dollars. The root mean squared error is $\sqrt{(400+100+100)/3}=\sqrt{200}=14.14$ thousand dollars. The percentage errors are $20/100=20\%$, $10/100=10\%$, and $10/100=10\%$, so their mean is $40\%/3=13.33\%$.

Check and interpretation. Both segments have an aggregate ratio of 1, but their errors differ. Absolute dollar errors are larger in the large-exposure segment, while relative errors are larger in the small-exposure segment. The small-segment percentage error is sensitive to its lower predictions. The common mistake is to conclude from an aggregate ratio of 1 that individual predictions are accurate; the correction is to pair calibration with an error measure suited to the decision being assessed.

Modification Stability Across Renewals

Predictive accuracy and modification stability are related evaluation questions, but they are not the same. A modification that changes little from one renewal to the next is stable; that alone does not show that it predicts outcomes well. A modification that changes substantially may reflect a real change in risk, random claim variation, changes in exposure, or features of the plan. Interpret stability alongside outcomes and the circumstances of each renewal.

One simple segment-level stability measure uses a segment's average modification $M_{s,t}$ at renewal $t$. The mean absolute change between consecutive renewals is the average of $|M_{s,t}-M_{s,t-1}|$. For comparisons across segments with different typical modification levels, this change can be divided by the segment's mean modification over the periods, provided that mean is positive. This is a descriptive measure, not a test that a given amount of change is acceptable.

C_s = \frac{\displaystyle\frac{1}{T-1} \sum_{t=2}^{T}|M_{s,t}-M_{s,t-1}|} {\displaystyle\frac{1}{T}\sum_{t=1}^{T}M_{s,t}}, \qquad \text{for } T\ge 2 \text{ and positive mean modification.}

The numerator measures the average absolute renewal-to-renewal movement; the denominator scales it by the average modification. A smaller $C_s$ indicates less relative movement over the observed sequence. The measure does not account for whether the changes were justified by new information, nor does it describe variation among individual accounts hidden within a segment average. Report the period, segment definition, and whether the statistic is based on segment averages or account-level modifications.

Worked Example: Comparing Renewal Stability by Claim Profile

A hypothetical plan review tracks average modifications over three consecutive renewals for two claim-profile segments. Segment A has modifications $0.85,1.15,1.00$; its observed-to-predicted loss ratios over those periods are $0.90,1.10,1.00$. Segment B has modifications $0.98,1.02,1.00$, with observed-to-predicted ratios $0.98,1.02,1.00$. Calculate each segment's mean absolute modification change and relative stability measure $C_s$, rounding percentages to one decimal. Then describe the limitation of the comparison.

Method. Take the absolute difference between each pair of consecutive modifications, average the two differences, and divide by the average modification across the three renewals. The ratios provide context about observed performance; they do not by themselves prove why a modification moved.

Segment A. The consecutive absolute changes are $|1.15-0.85|=0.30$ and $|1.00-1.15|=0.15$. Their mean is $(0.30+0.15)/2=0.225$. The average modification is $(0.85+1.15+1.00)/3=3.00/3=1.00$. Therefore $C_A=0.225/1.00=0.225$, or 22.5%.

Segment B. The changes are $|1.02-0.98|=0.04$ and $|1.00-1.02|=0.02$. Their mean is $(0.04+0.02)/2=0.03$. The average modification is $(0.98+1.02+1.00)/3=3.00/3=1.00$. Therefore $C_B=0.03/1.00=0.03$, or 3.0%.

Check and interpretation. Segment A's average modification changed more from renewal to renewal. Its observed-to-predicted ratios also moved from 0.90 to 1.10 and back to 1.00, while Segment B's ratios were closer to 1 in each period. These short sequences do not establish that volatility caused the outcome pattern, or that the plan's changes were unjustified. The common mistake is to label the more stable segment automatically superior; the correction is to examine stability together with predictive performance, exposure, and uncertainty.

How to Read Segment Results Responsibly

Segment comparisons are most useful when the report makes both the result and its evidential strength visible. For each segment, show its definition, number of risks or exposure units, predicted amount, observed amount, and selected accuracy and stability measures. Include uncertainty estimates where appropriate, particularly when claim counts or exposure are small. An extreme ratio in a small segment may be an unstable estimate rather than evidence of a persistent difference.

Use the same outcome definition, evaluation period, and treatment of losses across the segments being compared. If segments differ in exposure size, dollar errors naturally tend to be larger for segments with larger predicted amounts. Relative measures can help describe scale, but can be sensitive to small denominators. Report absolute and relative results when both are informative, and do not omit the underlying amounts.

Interpret segment-level patterns as diagnostics, not automatic instructions to change a plan. A persistent departure from calibration may warrant investigation into data quality, changing risk characteristics, or model fit. An unstable modification may warrant investigation into whether its movements reflect information or noise. The diagnosis requires context; a statistic alone cannot identify the cause.

This comparison does not adjust segments to a common distribution of exposure or risk characteristics. When groups contain different mixes, that composition can affect the comparison. Keep the segment results clearly labeled and do not infer that the groups are directly comparable after accounting for mix unless that adjustment has actually been performed.

Key takeaway: Evaluate experience-rating performance at both portfolio and segment levels. Report calibration and prediction error separately, measure modification stability over a stated period, and accompany each result with its exposure volume and limitations. Aggregate calibration can coexist with materially different segment outcomes.

Check Your Understanding

For each problem, use the stated segment definitions and outcome basis.

A hypothetical plan predicts $240,000 for an equipment-services segment and observes $300,000. It predicts $360,000 for a second segment and observes $300,000. Calculate both segment ratios and the portfolio ratio. (Answer: 1.2500 and 0.8333; portfolio ratio 1.0000.)
In one segment, two risks have predictions of $20,000 each and observed losses of $10,000 and $30,000. Calculate the observed-to-predicted ratio, mean absolute error, and root mean squared error. (Answer: ratio 1.00; MAE $10,000; RMSE $10,000.)
A segment's average modifications over three renewals are 0.90, 1.05, and 1.05. Calculate the mean absolute consecutive change and $C_s$, using the definition in this tutorial. (Answer: mean absolute change 0.075; average modification 1.00; $C_s=7.5\%$.)
Three segments have predictions of $100,000, $200,000, and $700,000, with observed-to-predicted ratios 0.80, 1.10, and 1.00. Find total observed losses and the portfolio ratio, and identify the segment with the largest absolute dollar discrepancy. (Answer: observed losses $1,000,000; portfolio ratio 1.0000; the first and second segments tie at $20,000 each.)
Two segments each have an observed-to-predicted ratio of 1.00. One has much larger mean absolute percentage error than the other. State whether this is contradictory and what the metrics indicate. (Answer: not contradictory; aggregate calibration is the same, but the first segment has larger average relative individual prediction errors.)

What You'll Learn