Worked Comparison Of Competing GLM Rate Plans

Why a Rate-Plan Comparison Needs More Than One Score

Competing GLM rate plans can give different answers to the same selection question. A more complex plan may fit the training data better, while a simpler plan may have a better information criterion, a third may predict new observations more accurately, and a fourth may produce less volatile rating factors. These results are not necessarily contradictory: each measure evaluates a different property.

A useful comparison therefore begins by making the candidates genuinely comparable and stating the purpose of the decision. The examples below use hypothetical claim-frequency models and pure-premium predictions. The data and results are invented for instruction; they do not describe an actual insurer or market.

Definition: A candidate rate plan is a specified model structure, including its response, exposure treatment, distribution, link, predictors, and any constraints. A valid score comparison requires that the candidates be evaluated on a common basis. In particular, training likelihood-based measures should use the same response observations, weights or exposure treatment, response distribution, and likelihood convention. Predictive scores should use the same held-out observations and scoring rule.

For a Poisson claim-count GLM with a log link, let $Y_j$ be the claim count for observation $j$, $e_j$ its exposure, and $\mu_j$ its expected count. The model has $Y_j \sim \operatorname{Poisson}(\mu_j)$ and $\log(\mu_j)=\log(e_j)+\eta_j$, where $\eta_j$ is the predictor. The term $\log(e_j)$ is an offset with coefficient fixed at one. This setup assumes conditionally independent counts and a correctly specified Poisson mean-variance relationship; if the response distribution or observation basis differs, the comparisons below need reconsideration.

What the Main Comparison Measures Mean

The deviance measures lack of fit relative to a saturated model, which fits each observed response as closely as the model family permits. For a fitted likelihood $\ell$ and saturated-model likelihood $\ell_{\mathrm{sat}}$, the deviance is $D=2(\ell_{\mathrm{sat}}-\ell)$. Lower deviance means better in-sample fit when the response data and likelihood basis are the same. It does not, by itself, penalize the number of fitted parameters.

AIC adds a complexity penalty to the maximized likelihood. With $k$ estimated parameters and maximized log-likelihood $\ell$, $AIC=-2\ell+2k$. Lower AIC is preferred among candidates fitted to the same response on a comparable likelihood basis. AIC is a relative selection criterion, not a measure of absolute adequacy or a guarantee that a model will perform well in a particular portfolio.

D=2(\ell_{\mathrm{sat}}-\ell), \qquad AIC=-2\ell+2k

The deviance formula follows directly from the likelihood-ratio comparison: the log-likelihood advantage of the saturated model over the fitted model is $\ell_{\mathrm{sat}}-\ell$; twice that advantage is the deviance. For AIC, the penalty $2k$ adjusts the maximized fit score for the approximate optimism from estimating $k$ parameters under regularity conditions. Thus, a model can lower deviance by adding parameters but still have a higher AIC if the fit improvement is too small to offset the penalty.

Held-out predictive loss addresses a different question: how well do predictions score on observations not used to fit the model? The loss must suit the intended prediction. For example, exposure-weighted squared error can compare predicted and observed pure premiums when large dollar errors are the concern. Stability measures whether estimated factors materially change across reasonable refits, such as different periods or resamples. Neither predictive loss nor stability replaces the others; each supplies distinct evidence.

Worked Example: Ranking Training Fit and AIC

A hypothetical insurer fits three Poisson frequency plans to the same 2,400 policy-year observations, using the same response, exposure offset, and likelihood convention. Plan A has 8 estimated parameters and deviance 420; Plan B has 12 parameters and deviance 390; Plan C has 18 parameters and deviance 385. The common saturated-model log-likelihood is $-1{,}000$. Rank the plans by deviance and AIC.

Method. First recover each fitted log-likelihood using $\ell=\ell_{\mathrm{sat}}-D/2$. Then calculate AIC as $-2\ell+2k$. The common mistake this avoids is treating the lowest deviance as automatically the best overall rate plan.

Plan A: $\ell_A=-1{,}000-420/2=-1{,}210$. Therefore $AIC_A=-2(-1{,}210)+2(8)=2{,}420+16=2{,}436$.

Plan B: $\ell_B=-1{,}000-390/2=-1{,}195$. Therefore $AIC_B=-2(-1{,}195)+2(12)=2{,}390+24=2{,}414$.

Plan C: $\ell_C=-1{,}000-385/2=-1{,}192.5$. Therefore $AIC_C=-2(-1{,}192.5)+2(18)=2{,}385+36=2{,}421$.

The deviance ranking is C (385), B (390), A (420), from lowest to highest. The AIC ranking is B (2,414), C (2,421), A (2,436). B has the lowest AIC even though C has the lowest deviance. As a check, the common saturated-model term gives $AIC=2{,}000+D+2k$; this yields 2,436, 2,414, and 2,421 as calculated.

Compare Held-Out Predictive Performance on a Common Scale

A validation score is informative only if the response, exposure basis, and scoring rule match the decision objective. Here, suppose the objective is prediction of pure premium, measured in dollars per exposure unit. For held-out group $g$, let $w_g$ be its exposure, $y_g$ its observed average cost per exposure, and $\hat{y}_g$ the candidate's predicted average cost. The exposure-weighted squared error is $\sum_g w_g(y_g-\hat{y}_g)^2$. Dividing by total exposure gives a mean squared error per exposure unit. Lower values are preferred for this chosen score; the units are squared dollars, not dollars.

This score gives greater influence to groups with more exposure and penalizes larger prediction errors quadratically. It may not be appropriate if the decision instead emphasizes relative error, claim-count likelihood, or another loss function. State the scoring rule before comparing candidates, and do not select it after seeing which model wins.

Worked Example: Comparing Pure-Premium Predictions on Holdout Data

A hypothetical portfolio has three held-out risk groups, each with 100 exposure units. Their observed average costs are $100, $200, and $300 per exposure unit. Plan A predicts $105, $195, and $300; Plan B predicts $100, $210, and $290; Plan C predicts $115, $185, and $300. Calculate each plan's exposure-weighted mean squared error and identify the winner. The common mistake this avoids is comparing raw errors without accounting for different group exposures.

Method. Square each observed-minus-predicted error, multiply by the group's exposure, add the three contributions, and divide by total exposure, $100+100+100=300$.

Plan A: The errors are $-5, 5, 0$ dollars. The weighted contributions are $100(25)=2{,}500$, $100(25)=2{,}500$, and $100(0)=0$. The total is $5{,}000$, so the mean squared error is $5{,}000/300=16.6667$ dollars squared per exposure unit.

Plan B: The errors are $0, -10, 10$ dollars. The contributions are $0$, $100(100)=10{,}000$, and $100(100)=10{,}000$. The total is $20{,}000$, so the mean squared error is $20{,}000/300=66.6667$ dollars squared per exposure unit.

Plan C: The errors are $-15, 15, 0$ dollars. The contributions are $100(225)=22{,}500$, $100(225)=22{,}500$, and $0$. The total is $45{,}000$, so the mean squared error is $45{,}000/300=150.0000$ dollars squared per exposure unit.

Plan A has the lowest held-out mean squared error. Rounded to two decimals, the scores are 16.67, 66.67, and 150.00 dollars squared per exposure unit for A, B, and C, respectively. Each total is the sum of its three displayed contributions, and the denominator is the same 300 exposure units.

Check Factor Stability Before Calling a Plan Usable

A model's fitted relativities can change when the experience period or sample changes. One simple diagnostic compares each key fitted relativity with its value from a refit. For a baseline relativity $r$ and refitted value $r^*$, the absolute percentage change is $100|r^*-r|/r$. A comparison might summarize the largest such change among selected factors and refits. This is a diagnostic, not a universal pass/fail rule: acceptable movement depends on the use, volume of experience, and review standards.

Worked Example: Computing a Factor-Stability Diagnostic

Three hypothetical frequency plans are refit on several time windows. Their largest observed percentage changes among selected rating relativities are 3% for Plan A, 5% for Plan B, and 24% for Plan C. For Plan C, a selected baseline relativity is 1.40 and the corresponding refit is 1.064. Verify the calculation and interpret the result. The common mistake this avoids is describing the change as 0.336% by confusing an absolute factor difference with a percentage change.

Method. Divide the absolute difference by the baseline relativity and multiply by 100. For Plan C, $|1.064-1.40|=0.336$, and $0.336/1.40=0.24$. Therefore the relative change is $0.24(100)=24\%$, consistent with the reported maximum.

Plan A's maximum is 3%, Plan B's is 5%, and Plan C's is 24%. The diagnostic flags greater movement in C's selected factors across the tested refits. It does not prove that C is wrong, nor does it establish that A or B is stable under every possible change in data. It is evidence to consider alongside fit and predictive performance.

Reconcile Conflicting Criteria with a Stated Objective

No single ranking automatically resolves a comparison when criteria disagree. A defensible process is to confirm comparability, identify the primary objective in advance, examine secondary diagnostics, and document the trade-off. The following sequence keeps the conclusion tied to evidence rather than to whichever score happens to favor a preferred plan.

Confirm the comparison basis.
Check response definition, observations, exposure treatment, distribution, likelihood convention, and any weights. Keep the held-out set and scoring rule common across candidates.

Read fit and complexity together.
Use deviance to describe in-sample fit and AIC to balance likelihood fit with parameter count. Do not interpret a lower deviance alone as decisive evidence.

Evaluate prediction and stability.
Compare a preselected validation score suited to the prediction target, then examine whether important factors move materially across reasonable refits.

State the decision rule and trade-off.
Name the primary objective, explain why the selected plan satisfies it, and disclose where another candidate scores better. If the evidence remains close or sensitive, report that uncertainty rather than claiming a universal winner.

Apply this process to the three plans. Plan C has the smallest training deviance, but it has the highest AIC of B and C and the largest factor movement. Plan B has the lowest AIC, while Plan A has the lowest held-out squared error and the smallest stated stability change. If the hypothetical insurer had declared held-out pure-premium squared error as its primary objective before fitting, and treated the displayed stability results as acceptable, selecting A is defensible. The conclusion should explicitly acknowledge that B has the lower AIC. If likelihood-based support were the declared priority, the decision could differ and would need to address B's validation score and stability as well.

Common Exam Traps

Using the wrong AIC penalty. A common incorrect calculation is $AIC=-2\ell+k$. For Plan B above, that error gives $2{,}390+12=2{,}402$. The standard formula uses $2k$, giving $2{,}390+24=2{,}414$.
Comparing incomparable likelihoods. Deviance or AIC rankings are not made fair merely by displaying them in one table. A changed response basis, likelihood convention, or set of observations can invalidate a direct comparison.
Treating a validation winner as automatically implementable. A predictive score answers a question under a particular holdout design and loss. It does not by itself establish factor stability or suitability for every use.
Calling one criterion the universal answer. Deviance, AIC, predictive loss, and stability answer different questions. State which objective governs the choice and disclose meaningful conflicts.

Key takeaway: A rigorous rate-plan comparison reports fit, complexity, held-out prediction, and factor stability on clearly stated bases. When they disagree, choose in light of a predeclared objective and explain the trade-offs; do not disguise a multi-criteria judgment as a mechanical ranking.

Check Your Understanding

For each original comparison, identify the requested result and check your reasoning against the answer.

Two Poisson plans use the same observations and likelihood basis. Plan P has deviance 90 and 8 parameters; Plan Q has deviance 82 and 14 parameters. Compare their AIC values after removing the common saturated-model constant $2\ell_{\mathrm{sat}}$, so the comparison score is $D+2k$. (Answer: P scores $90+16=106$; Q scores $82+28=110$, so P has the lower AIC despite Q's lower deviance.)
On two held-out groups with 50 exposure units each, observed average costs are $80 and $120. Plan M predicts $90 and $120; Plan N predicts $85 and $110. Calculate each exposure-weighted mean squared error. (Answer: M has $50(100)/100=50.00$ dollars squared per exposure unit; N has $[50(25)+50(100)]/100=62.50$, so M is lower.)
A baseline rating relativity is 1.10. Across two refits it is 1.21 and 1.045. What is the largest absolute percentage change relative to baseline? (Answer: 10.00%, from $100|1.21-1.10|/1.10$; the other change is 5.00%.)
Plan Alpha has a preselected validation score of 42 and a maximum stability change of 2%; Plan Beta has score 36 and stability change of 12%. The declared rule is to minimize validation score subject to a maximum stability change of 8%. Which plan qualifies and is selected? (Answer: Alpha; Beta has the lower score but fails the 8% stability constraint.)
AIC favors one candidate, but a separately held-out score favors another. What must be stated before recommending either plan? (Answer: Confirm comparability, state the predeclared primary objective and scoring basis, and explain the trade-off and stability evidence rather than treating either ranking as universally decisive.)

What You'll Learn