Why a Validation Review Needs More Than One Score
A rate model can predict total losses well while misallocating them across risks. Another model can rank risks effectively but systematically overpredict in one segment and underpredict in another. A validation review therefore asks several distinct questions: How close are predictions to outcomes under an appropriate loss measure? Are predictions calibrated overall and within important segments? Do they rank risks usefully? The answers should be considered together rather than collapsed into a single unexplained winner.
For a fair comparison, both candidates must be evaluated on the same chronological test period, with consistent exposure and loss maturity. Predictions should be frozen before the outcomes from that period are examined. This preserves the test sample as a genuine out-of-sample check; repeatedly tuning models against it makes the reported performance optimistic. The earlier material on validation periods and identical validation data addresses those setup requirements.
Start With the Outcome and Its Distribution
The predictive loss must correspond to the modeled quantity and its assumptions. For claim counts, a standard choice is Poisson deviance when counts are modeled as independent Poisson observations with the stated expected counts. If a cell has exposure \(e_j\) and predicted frequency \(\hat f_j\), its expected count is \(\hat\mu_j=e_j\hat f_j\). Given observed count \(y_j\), the deviance contribution is:
When \(y_j=0\), the term \(y_j\ln(y_j/\hat\mu_j)\) is defined as zero. Smaller total deviance means better fit under this scoring rule; a score is not an absolute measure of model quality and should be interpreted with the outcome definition and validation sample in view.
The formula follows from comparing the Poisson log likelihood at the model’s expected count with the log likelihood at the saturated value, which fits that cell exactly. For a Poisson count \(Y\) with mean \(\mu\), the log likelihood, omitting constants that do not depend on \(\mu\), is \(y\ln\mu-\mu\). The saturated model uses \(\mu=y\), so twice the log-likelihood difference is:
This derivation explains the Poisson deviance formula and its nonnegative contribution: the saturated model maximizes the Poisson likelihood for that observation. It does not justify using the score when the Poisson count model is inappropriate. For example, material overdispersion may require a different model or a score suited to the intended predictive distribution.
For positive pure-premium observations, Gamma deviance is appropriate only when the assumed response distribution is Gamma, with variance proportional to the square of its mean. If observation \(j\) has exposure weight \(e_j\), observed pure premium \(y_j>0\), and prediction \(\hat\mu_j>0\), the weighted contribution is:
It is not a universal loss for all pure-premium data: in particular, a Gamma response must be positive, so this form does not directly handle zero pure premiums. Earlier material on predictive loss measures covers other choices and their assumptions.
Worked Example: Comparing Frequency Models on a Later Test Period
A hypothetical insurer froze two frequency models after fitting them on earlier years. They are now evaluated on the same later, fully matured test period. The independent cell counts are assumed Poisson with the expected counts shown below. No interest or mortality assumption applies. Which model has the lower total Poisson deviance? Round only the final totals to four decimals.
| Cell | Exposure | Observed count | Model A predicted frequency | Model A expected count | Model B predicted frequency | Model B expected count |
|---|---|---|---|---|---|---|
| 1 | 50 | 10 | 0.24 | 12 | 0.16 | 8 |
| 2 | 80 | 16 | 0.175 | 14 | 0.25 | 20 |
| 3 | 120 | 24 | 0.2166667 | 26 | 0.1833333 | 22 |
Step 1: Check the expected counts. Multiply exposure by predicted frequency in each cell. For example, Model A in cell 1 predicts \(50(0.24)=12\) claims. The remaining expected counts are \(80(0.175)=14\), \(120(0.2166667)\approx26\), and, for Model B, \(50(0.16)=8\), \(80(0.25)=20\), \(120(0.1833333)\approx22\).
Step 2: Calculate Model A’s contributions. Using the Poisson deviance formula, the cell 1 contribution is \(2[10\ln(10/12)-(10-12)] =2[10\ln(5/6)+2]\approx0.353569\). For cell 2 it is \(2[16\ln(16/14)-2]\approx0.273005\). For cell 3 it is \(2[24\ln(24/26)+2]\approx0.157950\). Thus, \(D_A\approx0.353569+0.273005+0.157950=0.784523\), or **0.7845** to four decimals.
Step 3: Calculate Model B’s contributions. For cell 1, \(2[10\ln(10/8)-2]\approx0.462871\). For cell 2, \(2[16\ln(16/20)+4]\approx0.859406\). For cell 3, \(2[24\ln(24/22)-2]\approx0.176546\). Thus, \(D_B\approx0.462871+0.859406+0.176546=1.498823\), or **1.4988** to four decimals.
Conclusion and check. Model A has lower total Poisson deviance on this test sample. The observed total is 50 claims; Model A predicts 52 and Model B predicts 50. Model B’s exact aggregate count calibration does not outweigh its larger cell-level deviations under this score. The calculation checks because every expected count equals exposure times predicted frequency, and the three contributions sum to the displayed totals.
Method and mistake avoided. This is a cell-by-cell Poisson deviance comparison on identical observations. A common mistake is to choose Model B solely because its predicted total equals the observed total; that discards how predictions differ across cells.
Use Calibration and Ranking as Complementary Checks
A calibration check compares observed outcomes with predictions. For a group of observations, a useful descriptive ratio is total observed divided by total predicted. A ratio above 1 indicates that the model underpredicted in aggregate for that group; a ratio below 1 indicates aggregate overprediction. A calibration plot places grouped observed outcomes against grouped predictions: perfect calibration lies on the line where observed equals predicted. The result can depend on how groups are formed, so the grouping rule should be stated.
Ranking asks a different question: do higher predictions tend to be assigned to risks with higher outcomes? A concordance summary can be defined by considering all pairs with different observed outcomes. A pair is concordant if the risk with the higher observed outcome also has the higher prediction, and discordant if the prediction orders the pair the other way. With no ties, the concordance proportion is the number of concordant pairs divided by the number of pairs. Ranking does not establish calibration: multiplying every prediction by the same positive constant leaves the ordering unchanged.
Worked Example: Gamma Deviance for Positive Pure Premiums
In a hypothetical validation period, three rating cells have positive observed pure premiums in dollars per policy-year. The pure-premium response is assumed independent across cells and Gamma, with variance \(\phi\mu^2/e\), where \(e\) is the cell’s exposure weight, \(\mu\) is its mean, and \(\phi\) is a common dispersion parameter. Candidate predictions are the means \(\mu\). No interest or mortality assumption applies. Compare exposure-weighted Gamma deviance; round final totals to four decimals.
| Cell | Exposure weight | Observed pure premium | Model A prediction | Model B prediction |
|---|---|---|---|---|
| 1 | 100 | $120 | $100 | $150 |
| 2 | 200 | $180 | $200 | $150 |
| 3 | 100 | $330 | $300 | $300 |
Step 1: Identify the ratios. The observed-to-predicted ratios for Model A are \(1.2, 0.9, 1.1\). For Model B they are \(0.8, 1.2, 1.1\). All observations and predictions are positive, as required by this Gamma deviance calculation.
Step 2: Calculate Model A’s weighted contributions. For ratio \(1.2\), the unweighted contribution is \(2[1.2-1-\ln(1.2)]\approx0.0353569\); multiplying by exposure 100 gives \(3.5356886\). For ratio \(0.9\), it is \(2[0.9-1-\ln(0.9)]\approx0.0107210\); multiplying by 200 gives \(2.1442063\). For ratio \(1.1\), it is \(2[1.1-1-\ln(1.1)]\approx0.0093796\); multiplying by 100 gives \(0.9379640\). Hence \(D_A\approx3.5356886+2.1442063+0.9379640=6.6178589\), or **6.6179** to four decimals.
Step 3: Calculate Model B’s weighted contributions. For ratio \(0.8\), the unweighted contribution is \(2[0.8-1-\ln(0.8)]\approx0.0462871\); multiplying by 100 gives \(4.6287103\). For ratio \(1.2\), the contribution is \(0.0353569(200)=7.0713773\). For ratio \(1.1\), it is \(0.0093796(100)=0.9379640\). Hence \(D_B\approx4.6287103+7.0713773+0.9379640=12.6380516\), or **12.6381** to four decimals.
Conclusion and check. Model A has the lower exposure-weighted Gamma deviance. The displayed contributions add to each stated total. This conclusion is conditional on the stated positive-response Gamma assumptions; it should not be transferred to a sample containing zero pure premiums without choosing a suitable model and score.
Method and mistake avoided. This example applies Gamma deviance with exposure weights. Omitting the exposure weights would treat a cell representing 200 exposure units as if it carried the same weight as a cell representing 100.
Check Segments Before Documenting a Decision
Overall performance can conceal offsetting segment errors. At a minimum, review observed-versus-predicted summaries for material rating segments, such as broad territory or vehicle groups, and check whether rankings behave sensibly within them. Report the exposures and outcome basis behind each comparison. Sparse segments can produce volatile ratios, so a striking result in a small segment should not be treated as conclusive without considering its uncertainty.
The following example uses simple descriptive summaries, not a fitted probability model. There is no distributional assumption, interest rate, or mortality assumption. The same four equal-exposure risks are scored by both candidates.
Worked Example: Combining Calibration, Ranking, and Segment Evidence
A hypothetical portfolio has four one-policy-year validation cells. Observed and predicted pure premiums are in dollars per policy-year. The two lower-risk cells form one prespecified segment and the two higher-risk cells form another. Compare overall and segment calibration, then compare ranking concordance. Do not assume a sampling distribution for these descriptive summaries.
| Cell | Segment | Observed | Model A prediction | Model B prediction |
|---|---|---|---|---|
| 1 | Lower | $80 | $100 | $60 |
| 2 | Lower | $120 | $140 | $130 |
| 3 | Higher | $240 | $200 | $310 |
| 4 | Higher | $360 | $300 | $260 |
Step 1: Check overall calibration. Total observed pure premium is \(80+120+240+360=\$800\). Model A predicts \(100+140+200+300=\$740\), so its observed-to-predicted ratio is \(800/740\approx1.0811\). Model B predicts \(60+130+310+260=\$760\), giving \(800/760\approx1.0526\). Both underpredict overall, and Model B is closer to an overall ratio of 1.
Step 2: Compare segments. In the lower segment, observed total is \(\$200\). Model A predicts \(\$240\), so observed-to-predicted is \(200/240=0.8333\); Model B predicts \(\$190\), so the ratio is \(200/190\approx1.0526\). In the higher segment, observed total is \(\$600\). Model A predicts \(\$500\), giving \(600/500=1.2000\); Model B predicts \(\$570\), giving \(600/570\approx1.0526\). Model A overpredicts the lower segment and underpredicts the higher one. Model B is closer to calibration in both.
Step 3: Compare ranking. The observed order from lowest to highest is cells 1, 2, 3, 4. Model A predicts \(100<140<200<300\), so all six pairs are concordant and the concordance proportion is \(6/6=1.0000\). Model B orders the cells 1, 2, 4, 3: five pairs are concordant and the cell 3 versus cell 4 pair is discordant. Its concordance proportion is \(5/6\approx0.8333\).
Documented decision. If the stated priority is avoiding systematic differences between these two segments, Model B is the stronger provisional choice: it has better overall and segment calibration, while its ranking is less accurate in this small sample. If ranking is the primary objective, Model A has the advantage. A defensible review records that trade-off, the prespecified priorities, the sample limitations, and any uncertainty assessment rather than declaring a universal winner from one diagnostic.
Method and mistake avoided. This is a joint descriptive review of calibration by segment and pairwise ranking. A common mistake is to infer good calibration from good ranking: Model A orders every cell correctly but has substantial opposing segment errors.
Turning Diagnostics Into a Validation Record
A concise validation record should make the decision reproducible. It should identify the chronological test period, the outcome definition and maturity, the exposure basis, the frozen candidate predictions, the score and its distributional assumptions, and the calibration and ranking summaries. It should also state material segment results, any uncertainty analysis, and the reason the selected model better meets the declared objectives.
Verify that both candidates use the same test observations, exposure basis, outcome definition, and maturity rules.
Choose a predictive loss appropriate to the outcome and distribution; show the total and enough detail to verify it.
Report observed-versus-predicted summaries and a ranking measure, explaining that they answer different questions.
Look for material pockets of underprediction, overprediction, or unstable performance, with exposure and sample size in view.
Connect the recommendation to the intended use and declared priorities; record trade-offs and limitations instead of treating one score as decisive.
A validation review is most useful when it separates evidence from judgment. The score quantifies performance under a specified loss; calibration describes systematic agreement; ranking describes ordering; segment checks reveal where portfolio averages may be misleading. Together, these diagnostics support a transparent decision, but the test sample remains evidence about performance under its conditions, not a guarantee of future results.
Check Your Understanding
Use the stated assumptions in each question; answers are given so you can check your work.
- A validation cell has 80 policy-years, 10 observed claims, and predicted frequency 0.15. Under a Poisson count assumption, find the expected count and deviance contribution. No interest or mortality assumption applies. (Answer: Expected count 12; deviance contribution 0.3536 to four decimals.)
- A positive pure-premium observation is $90, its Gamma-model prediction is $100, and its exposure weight is 50. Under the exposure-weighted Gamma deviance formula, calculate the contribution. No interest or mortality assumption applies. (Answer: 0.5361 to four decimals.)
- A segment has observed total loss cost $500 and predicted total loss cost $450. Calculate the observed-to-predicted ratio and state the direction of aggregate calibration error. (Answer: 1.1111; the model underpredicts.)
- Three risks have observed outcomes 40, 70, and 100. Model A predicts 35, 80, and 90; Model B predicts 75, 65, and 50. With no ties, calculate each model’s concordance proportion over all three pairs. (Answer: Model A \(2/3=0.6667\); Model B \(0/3=0.0000\).)
- Two models have equal total observed-to-predicted ratios, but one consistently overpredicts a low-risk segment and underpredicts a high-risk segment. Name the validation check that reveals this issue and explain why the overall ratio does not. (Answer: Segment calibration checks reveal it; offsetting segment errors can cancel in the portfolio total.)