Why Calibration and Ranking Must Be Read Together
A ratemaking model can identify which risks are relatively higher or lower while systematically predicting too much or too little loss. The distinction matters in model selection: a model that ranks risks well may still produce a poor overall rate level, and a portfolio-level adjustment may correct the total while leaving important segment errors. Calibration and ranking therefore answer different questions and should be examined together.
For validation, compare predictions with observed outcomes on the same holdout data, using the same loss definition, exposure basis, and evaluation period for every candidate. A predicted loss total is the sum of the model's predictions over the policies or exposure units in a group. Observed loss is the corresponding realized loss total. If a model predicts a pure premium per exposure unit, multiply each prediction by its exposure before summing.
Calibration asks whether predicted levels agree with observed levels, overall or in specified segments. Ranking asks whether the model orders risks in a way that aligns with their observed outcomes. When a group contains many policies, predictions and outcomes should be aggregated using exposure consistently; a group with more exposure should not count the same as a tiny group merely because each occupies one row in a table.
Reading Group Calibration and Ranking
A practical diagnostic is to sort validation risks into groups by predicted rate or predicted loss per exposure unit, then compare observed and predicted results within each group. A useful calibration table reports exposure, observed loss, predicted loss, and the observed-to-predicted ratio. An overall ratio near 1 can conceal groups with substantial errors in opposite directions, so group ratios provide information that the portfolio total cannot.
A ranking measure can summarize whether higher predictions tend to accompany higher observed outcomes. For a small ordered set of groups with distinct observed rates, a simple concordance proportion is the number of correctly ordered group pairs divided by the total number of pairs. A pair is concordant when the group with the higher prediction also has the higher observed rate. More formal rank measures may use policy-level observations and rules for ties; the definition used should be stated. Ranking measures do not establish that predicted values are numerically accurate.
Worked Example: Similar Ranking, Different Calibration
A hypothetical insurer compares two candidate pure-premium models on the same validation portfolio. Each of four risk groups has 100 exposure units. Loss totals below are in thousands of dollars. Determine the group and portfolio calibration ratios, and compare the models' ranking.
| Group | Observed loss | Model A predicted | Model B predicted |
|---|---|---|---|
| 1 | 40 | 30 | 50 |
| 2 | 80 | 60 | 75 |
| 3 | 120 | 90 | 105 |
| 4 | 160 | 120 | 130 |
Step 1: Calculate group ratios. For Model A, the four ratios are \(40/30=1.3333\), \(80/60=1.3333\), \(120/90=1.3333\), and \(160/120=1.3333\). Each group is underpredicted by the same proportion.
For Model B, the ratios are \(40/50=0.8000\), \(80/75=1.0667\), \(120/105=1.1429\), and \(160/130=1.2308\). It overpredicts Group 1 and underpredicts each of the other groups.
Step 2: Calculate portfolio ratios. Observed loss totals \(40+80+120+160=400\) thousand dollars. Model A predicts \(30+60+90+120=300\), so its portfolio ratio is \(400/300=1.3333\). Model B predicts \(50+75+105+130=360\), so its portfolio ratio is \(400/360=1.1111\).
Step 3: Compare ranking. Both models predict Group 1 below Group 2, below Group 3, below Group 4. Observed loss per exposure unit also increases in that order because every group has equal exposure. There are \(4(3)/2=6\) group pairs, and all six are concordant for each model.
Conclusion and mistake avoided. Both models rank these groups perfectly, but their predicted levels and group calibration differ. Model B has a portfolio ratio closer to 1, though its segment errors are uneven. Choosing a model from ranking alone would miss the level error; choosing only by portfolio ratio would miss Model B's Group 1 overprediction and remaining group underprediction.
Why a Calibration Multiplier Works—and What It Cannot Fix
Suppose a candidate's predictions are to be multiplied by a single positive factor \(c\). The portfolio total after adjustment is \(c\sum_g\widehat{L}_g\). To make that total equal observed loss, solve \(c\sum_g\widehat{L}_g=\sum_g O_g\). This gives the portfolio observed-to-predicted ratio as the multiplier.
This adjustment achieves aggregate agreement on the data used to calculate it, provided the predicted total is positive. It does not guarantee agreement in each group. Nor does multiplying every prediction by the same positive constant change the order of predictions: if \(\widehat{L}_a<\widehat{L}_b\), then \(c\widehat{L}_a<c\widehat{L}_b\). Thus, a pure multiplicative level correction preserves ranking while changing the predicted level. The correction is a diagnostic or an operational adjustment only when appropriate; it does not by itself show that the underlying model is sound or that the adjustment will remain suitable on future data.
Worked Example: Correcting the Portfolio Level
A hypothetical model predicts losses of 40, 90, 170, and 300 thousand dollars in four validation groups. Observed losses are 50, 100, 200, and 350 thousand dollars. Calculate the common calibration multiplier, each adjusted group prediction, the adjusted group ratios, and the resulting portfolio ratio. Report ratios to four decimal places and adjusted losses to four decimal places in thousands of dollars.
Step 1: Find the portfolio totals. Observed loss is \(50+100+200+350=700\) thousand dollars. Predicted loss is \(40+90+170+300=600\) thousand dollars. Therefore \(c=700/600=7/6=1.166666\ldots\).
Step 2: Apply the same multiplier to every group. The adjusted predictions are \(40(7/6)=46.6667\), \(90(7/6)=105.0000\), \(170(7/6)=198.3333\), and \(300(7/6)=350.0000\), in thousands of dollars.
Step 3: Recalculate group ratios. The adjusted observed-to-predicted ratios are \(50/46.6667\approx1.0714\), \(100/105\approx0.9524\), \(200/198.3333\approx1.0084\), and \(350/350=1.0000\). The small difference in the third ratio reflects rounding; using the unrounded prediction \(595/3\), its ratio is \(600/595\approx1.0084\).
Step 4: Check the portfolio result. The adjusted predictions sum to \(46.6667+105.0000+198.3333+350.0000=700.0000\) thousand dollars, using unrounded values for the total. The adjusted portfolio ratio is therefore \(700/700=1.0000\). The adjustment preserves the groups' predicted order, but it has not eliminated the different group-level ratios.
Method and common mistake avoided. The method is a single portfolio calibration multiplier. Applying a different factor to each group would change more than the overall level and could change the model's relative rates. Conversely, claiming that this common multiplier makes every group calibrated would confuse portfolio agreement with segment agreement.
Exposure Weighting and the Information in a Ranking Summary
Group ratios should be interpreted alongside group size and the rate basis. The portfolio ratio is a ratio of summed losses, so each group's contribution depends on its predicted and observed amounts. Averaging group ratios without weights can yield a different, and often misleading, summary. Ranking summaries have another limitation: they compress an ordering into one statistic and do not show the size of prediction errors or whether a model's level is correct.
Worked Example: Unequal Exposures, a Rank Inversion, and Portfolio Calibration
A hypothetical model is evaluated in four groups ordered by its predicted loss per exposure unit. Exposure is measured in units, and rates and losses are in dollars. Calculate each group's observed rate and calibration ratio, the portfolio ratio, the unweighted average of group ratios, the concordance proportion, and the observed and predicted top-to-bottom rate ratios.
| Group | Exposure | Predicted rate | Observed loss | Predicted loss total |
|---|---|---|---|---|
| 1 | 100 | 10 | 1,200 | 1,000 |
| 2 | 200 | 20 | 3,600 | 4,000 |
| 3 | 300 | 30 | 10,800 | 9,000 |
| 4 | 400 | 40 | 13,600 | 16,000 |
Step 1: Convert observed totals to rates. Divide each observed loss by its exposure. The observed rates are \(1,200/100=12\), \(3,600/200=18\), \(10,800/300=36\), and \(13,600/400=34\) dollars per exposure unit.
Step 2: Calculate segment calibration ratios. Each group's predicted total is its predicted rate multiplied by exposure. The ratios are \(1,200/1,000=1.2000\), \(3,600/4,000=0.9000\), \(10,800/9,000=1.2000\), and \(13,600/16,000=0.8500\).
Step 3: Calculate portfolio calibration using totals. Observed loss is \(1,200+3,600+10,800+13,600=29,200\) dollars. Predicted loss is \(1,000+4,000+9,000+16,000=30,000\) dollars. The portfolio ratio is \(29,200/30,000=0.9733\) to four decimal places.
The unweighted average of the four group ratios is \((1.2000+0.9000+1.2000+0.8500)/4=1.0375\). It is not the portfolio ratio: it gives each group equal influence despite exposures of 100, 200, 300, and 400 units. The ratio of summed observed to summed predicted loss is the appropriate portfolio calculation here.
Step 4: Measure group ranking. There are six possible pairs among four groups. The predictions order them \(1<2<3<4\), while observed rates order them \(1<2<4<3\). Only the pair consisting of Groups 3 and 4 is discordant; the other five are concordant. The stated group-level concordance proportion is \(5/6=0.8333\) to four decimal places.
Step 5: Compare top and bottom rates. The predicted top-to-bottom ratio is \(40/10=4.0000\). The observed top-to-bottom ratio, comparing Group 4 with Group 1, is \(34/12=2.8333\) to four decimal places. The model separates the ends of the risk distribution, but overstates this contrast in these groups and has a small overall underprediction, as shown by its portfolio ratio below 1.
Check and interpretation. Predicted totals reconcile to \(10(100)+20(200)+30(300)+40(400)=30,000\) dollars. Observed totals reconcile to 29,200 dollars. The concordance result describes ordering only; the group ratios and totals reveal level errors that the rank summary cannot.
Using Both Diagnostics in Model Selection
A useful comparison keeps candidate models on identical validation data and reports calibration and ranking evidence side by side. For example, a compact review can include:
- Portfolio observed and predicted loss totals, their ratio, and the common evaluation basis.
- Group-level ratios for groups defined consistently, often by each model's predicted rate.
- A ranking summary with its unit of comparison and treatment of ties stated.
- The observed and predicted rates or losses in important groups, so the magnitude and direction of differences remain visible.
The groups should be interpreted carefully. If each model defines its own prediction groups, a comparison of the group tables can involve different populations in corresponding group labels. Consistent externally defined segments can help assess performance in important rating classifications; model-specific predicted groups can help show calibration across each model's score range. Neither view replaces the other.
The observed-to-predicted ratio is an aggregate diagnostic, not proof that every prediction is calibrated. A ratio near 1 may result from overprediction in some groups offsetting underprediction in others. Similarly, a high rank concordance does not establish accurate rate levels, and a good portfolio ratio does not establish good ranking. In practice, interpret these summaries with the loss definition, exposure, validation period, and uncertainty in mind. They are evidence for model selection, not a mechanical rule that determines the choice.
Common Exam Trap
Wrong calculation: Suppose four equally displayed groups have ratios \(1.20\), \(0.90\), \(1.20\), and \(0.85\). Taking their mean, \(1.0375\), and calling it the portfolio observed-to-predicted ratio is generally incorrect. It gives equal weight to groups regardless of their predicted and observed totals.
Correction: Sum observed losses and divide by summed predicted losses. In the unequal-exposure example, that is \(29,200/30,000=0.9733\), not \(1.0375\). Also avoid the opposite error: a ratio close to 1, or a strong concordance result, does not establish that every segment is calibrated. These distinctions are aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each hypothetical validation problem, calculate the requested quantities using the stated units and conventions.
- A model predicts losses of 24 and 36 thousand dollars in two groups; observed losses are 30 and 39 thousand dollars. Find each group ratio and the portfolio ratio. (Answer: Group ratios 1.2500 and 1.0833; portfolio ratio \(69/60=1.1500\).)
- Three groups, each with 100 exposure units, have observed rates of 8, 16, and 24 dollars per unit. A model predicts rates of 10, 15, and 20. Find the portfolio observed-to-predicted ratio and the group concordance proportion. (Answer: Ratio \(4,800/4,500=1.0667\); concordance \(3/3=1.0000\).)
- Two groups have exposures 100 and 300, observed losses of 1,200 and 2,700 dollars, and predicted rates of 10 and 8 dollars per exposure unit. Find the group ratios, portfolio ratio, and unweighted mean of the group ratios. (Answer: Ratios 1.2000 and 1.1250; portfolio ratio \(3,900/3,400=1.1471\); unweighted mean 1.1625.)
- A model has portfolio observed loss of 480 thousand dollars and predicted loss of 600 thousand dollars across four groups. Find the common multiplier that matches the validation portfolio total. If the predictions are multiplied by this positive factor, does their ranking change? (Answer: Multiplier \(480/600=0.8000\); no, a common positive multiplier preserves ranking.)
- Four equal-exposure groups have predicted rates ordered from low to high. Their observed rates, in that predicted order, are 11, 19, 17, and 31. Calculate group concordance and the observed-to-predicted ratio if the predicted rates are 10, 20, 30, and 40. (Answer: Five concordant pairs out of six, or 0.8333; observed total rate \(78\), predicted total rate \(100\), ratio 0.7800.)