Calibration Checks for Predicted Loss Costs

Why Check Calibration by Prediction Band?

A validation model can rank risks effectively yet systematically overpredict or underpredict their losses. A grouped calibration check looks for that kind of mismatch by placing validation observations into bands based on their predictions, then comparing observed outcomes with the corresponding predicted totals. The comparison is made on data not used to fit the predictions.

The key is to compare like with like. Claim counts require an exposure-based expected count; severity requires a claim-weighted average; and pure premium requires comparing total predicted loss with total observed loss over the same exposure. A simple average of policy-level rates or predictions often answers a different question.

Definition: A prediction band is a group of validation observations formed using their predicted risk measure, such as predicted claim frequency or pure premium. For band $j$, a calibration check compares an observed aggregate with the matching predicted aggregate. An observed-to-predicted ratio is $O/E=\text{observed amount}/\text{predicted amount}$, when the predicted amount is positive. A ratio of 1 indicates agreement of the two aggregates; a ratio above 1 indicates observed experience exceeded the prediction.

Bands may be formed using fixed prediction ranges or quantiles of the predictions. Their cut points should be determined from the prediction values, not from the validation outcomes. A band can be balanced by policy count or exposure, but the choice should be stated: equal policy counts do not guarantee equal exposure, and neither grouping method changes the denominator required for a particular calibration measure.

Match the Comparison to the Predicted Quantity

Suppose validation observation $i$ has exposure $e_i$, predicted claim frequency per exposure $\hat{\lambda}_i$, predicted mean severity $\hat{m}_i$, and predicted pure premium $\hat{p}_i$. Here, $\hat{p}_i=\hat{\lambda}_i\hat{m}_i$ when pure premium is defined as expected loss per exposure and the frequency and severity predictions use consistent loss definitions. Let $N_i$ be the observed claim count and $L_i$ the observed loss.

\begin{aligned} \text{Expected claims in band }j &= \sum_{i\in j} e_i\hat{\lambda}_i,\\ \text{Predicted loss in band }j &= \sum_{i\in j} e_i\hat{p}_i,\\ \text{Observed pure premium in band }j &= \frac{\sum_{i\in j}L_i}{\sum_{i\in j}e_i}. \end{aligned}

For frequency, compare $\sum N_i$ with $\sum e_i\hat{\lambda}_i$. For severity, compare observed loss per observed claim with predicted mean severity, using expected claim counts to weight the predictions. In particular, the band-level predicted mean severity is $\sum e_i\hat{\lambda}_i\hat{m}_i \,/\, \sum e_i\hat{\lambda}_i$, provided the expected claim count is positive. For pure premium, compare observed total loss with predicted total loss, or compare the corresponding exposure-based averages.

These calculations presume that observed and predicted quantities have the same definitions and are measured on a comparable basis. For example, the loss period, claim definition, coverage, and treatment of limits must align. The expected-count formula uses a frequency rate per unit of the stated exposure; it does not apply unchanged if the model's prediction is already a claim count for the whole observation.

Why Aggregate Comparisons Work

A model prediction is an expected amount, not a promise that each individual outcome will equal its prediction. Aggregation therefore compares the observed total with the expected total for the same group, rather than asking whether each policy's realized outcome matched its forecast.

For frequency, suppose $\hat{\lambda}_i$ is the expected number of claims per unit exposure for observation $i$. By the definition of this rate, its expected count is $e_i\hat{\lambda}_i$. The expected count across a band is the sum of the individual expected counts, whether or not the counts are independent:

E\!\left[\sum_{i\in j}N_i\right] =\sum_{i\in j}E[N_i] =\sum_{i\in j}e_i\hat{\lambda}_i.

For pure premium, if $\hat{p}_i$ is expected loss per unit exposure, expected loss for observation $i$ is $e_i\hat{p}_i$. Summing gives the band's predicted loss. Dividing the total predicted loss by total exposure yields the exposure-weighted predicted pure premium. This is why an unweighted average of policy-level pure premiums is generally not the appropriate band-level comparison.

Worked Example: Checking Claim-Frequency Calibration

A hypothetical validation portfolio is divided into two bands using predicted claim frequency. Band A contains 100 earned exposure units with predicted frequency 0.08 claims per unit and 10 observed claims. Band B contains 80 exposure units with predicted frequency 0.15 and 9 observed claims. Calculate expected claims and the observed-to-predicted ratio for each band and overall.

Method. Multiply each band's exposure by its predicted frequency to obtain its expected count. Compare observed and expected counts; do not compare the raw count with the predicted rate.

For band A, expected claims are $100(0.08)=8$. Its frequency O/E is $10/8=1.25$.

For band B, expected claims are $80(0.15)=12$. Its frequency O/E is $9/12=0.75$.

Overall, observed claims are $10+9=19$, and expected claims are $8+12=20$. Thus overall O/E is $19/20=0.9500$. The band results show underprediction in A and overprediction in B, even though the combined result is close to 1.

Check: The band expected counts sum to $8+12=20$, matching the overall expected count. The mistake this method avoids is dividing observed claims by predicted frequency alone: frequency is a rate, so exposure is required to obtain an expected count.

Worked Example: Checking Claim-Weighted Severity

A hypothetical severity model is assessed in two prediction bands. Band A has 12 expected claims, predicted mean severity of $2,000, 14 observed claims, and $30,000 of observed loss. Band B has 8 expected claims, predicted mean severity of $3,000, 7 observed claims, and $18,900 of observed loss. Calculate predicted and observed aggregate severity and their O/E ratio.

Method. Weight predicted severity by expected claim counts to obtain predicted aggregate severity. Calculate observed severity as total observed loss divided by total observed claims. Do not average the two band severities equally.

Predicted loss for A is $12($2{,}000)=$24{,}000$; for B it is $8($3{,}000)=$24{,}000$. Total predicted loss is $48,000 and total expected claims are $12+8=20$. Predicted aggregate severity is therefore $$48{,}000/20=$2{,}400$.

Observed claims total $14+7=21$, and observed losses total $$30{,}000+$18{,}900=$48{,}900$. Observed aggregate severity is $$48{,}900/21=$2{,}328.57$, rounded to the nearest cent.

Severity O/E is $$2{,}328.57/$2{,}400=0.9702$, to four decimals. The data show observed aggregate severity about 2.98% below predicted severity. As a check using the unrounded ratio, $$48{,}900/$48{,}000=1.01875$ is the loss O/E, not the severity O/E: it also reflects the difference between 21 observed claims and 20 expected claims.

Common mistake avoided: The unweighted average of predicted severities, $($2{,}000+$3{,}000)/2=$2{,}500$, gives equal weight to bands with different expected claim counts. The appropriate predicted aggregate severity is $2,400.

Worked Example: Checking Pure-Premium Calibration

A hypothetical validation portfolio has two bands. Band A has 40 exposure units, predicted pure premium of $120 per unit, and $5,250 observed loss. Band B has 60 exposure units, predicted pure premium of $200 per unit, and $10,800 observed loss. Calculate predicted and observed total losses, the overall pure premiums, and loss O/E.

Method. Multiply each predicted pure premium by its exposure to obtain predicted loss. Compare that total with observed loss, then divide each total by the same total exposure.

Predicted loss in A is $40($120)=$4{,}800$. Predicted loss in B is $60($200)=$12{,}000$. Total predicted loss is $$4{,}800+$12{,}000=$16{,}800$. Total observed loss is $$5{,}250+$10{,}800=$16{,}050$, and total exposure is $40+60=100$.

Predicted pure premium is $$16{,}800/100=$168.00$ per exposure unit. Observed pure premium is $$16{,}050/100=$160.50$ per exposure unit. Loss O/E is $$16{,}050/$16{,}800=0.9554$, to four decimals.

As a band-level check, A's O/E is $$5{,}250/$4{,}800=1.09375$, and B's is $$10{,}800/$12{,}000=0.9000$. The exposure-weighted overall result is not the unweighted mean of those two ratios. The calculation avoids averaging band pure premiums without regard to exposure.

Interpret the Pattern, Not Just the Overall Ratio

An overall O/E near 1 can conceal offsetting errors, as the frequency example illustrates. Review each band's observed and predicted totals, its O/E, and the number of observations or amount of exposure supporting it. A large ratio in a small-exposure band may be volatile; a single large claim can materially affect severity or pure-premium comparisons. Grouped results are diagnostic summaries, not proof that every prediction is accurate.

Calibration also depends on the target being measured. Frequency calibration does not establish severity calibration, and severity calibration does not establish pure-premium calibration. A model can miss one component while the combined loss prediction appears close. Where the purpose is to understand the source of a mismatch, examine the relevant component alongside the aggregate loss comparison.

Key takeaway: Form bands from validation predictions, then aggregate using the correct denominator: exposure for frequency and pure premium, and claim counts for severity. Report both band-level patterns and overall totals; a good overall fit can mask meaningful differences across bands.

Common Calibration Traps

Comparing a rate with a count. A predicted frequency is not an expected claim count until it is multiplied by its matching exposure.
Using the wrong severity weights. Predicted mean severity should be weighted by expected claims, not by policy count or an unweighted average of band means.
Averaging realized policy outcomes. A policy's observed loss divided by its exposure can be zero for most policies and unusually large for a few. Summing loss and exposure before dividing gives the portfolio pure premium.
Using observed outcomes to define bands. Grouping by observed loss can make the summaries misleading. Define bands using predictions, then compare observed outcomes within them.
Treating a ratio as a full diagnosis. O/E near 1 for the full portfolio does not show that each prediction band, or each model component, is calibrated.

Check Your Understanding

Use exposure-weighted totals where appropriate. Ratios are rounded to four decimals.

A band has 50 exposure units, predicted frequency of 0.12 claims per unit, and 8 observed claims. Find expected claims and frequency O/E. (Answer: 6 expected claims; O/E = 1.3333)
Two bands have exposures and predicted frequencies of 30 and 0.04, and 70 and 0.10, respectively. Observed counts are 2 and 7. Find the overall frequency O/E. (Answer: expected claims = 8.2; O/E = 1.0976)
Band A has 5 expected claims at predicted severity $1,600 and 6 observed claims with $10,800 loss. Band B has 15 expected claims at predicted severity $2,400 and 14 observed claims with $30,800 loss. Find predicted and observed aggregate severity and severity O/E. (Answer: predicted severity = $2,200; observed severity = $2,080; O/E = 0.9455)
A band has 25 exposure units at predicted pure premium $80 and observed loss $2,700. A second band has 75 exposure units at predicted pure premium $140 and observed loss $9,750. Find predicted total loss, observed pure premium, and overall loss O/E. (Answer: predicted total loss = $12,500; observed pure premium = $124.50; O/E = 0.9960)
Two bands have exposure, predicted frequency, and predicted severity as follows: A has 40, 0.05, and $1,500; B has 60, 0.10, and $2,500. Observed counts are 3 and 5, and observed losses are $5,100 and $12,300. Find expected claims, predicted loss, observed loss, frequency O/E, severity O/E, and pure-premium loss O/E. (Answer: expected claims = 8; predicted loss = $18,000; observed loss = $17,400; frequency O/E = 1.0000; severity O/E = 0.9667; pure-premium loss O/E = 0.9667)

What You'll Learn