Choosing Predictive Scores for Rate Models

Why the Scoring Rule Must Match the Prediction

A rate model can predict several different quantities: claim counts, average claim severity, or expected loss per unit of exposure. These predictions answer different questions, so their validation scores need not be interchangeable. A model that predicts frequency well is not necessarily the best model for severity, and a score computed on total loss is not automatically suitable for comparing claim-level predictions.

A predictive score summarizes how well predictions match outcomes on validation data. For all the scores in this tutorial, smaller is better. The score is useful only when the prediction, observed outcome, exposure treatment, and scoring assumptions are aligned. This is a practical model-selection skill aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Definition: A scoring rule assigns a numerical penalty to a prediction and the outcome that subsequently occurs. A score is proper for a target when, on average, the forecaster minimizes expected score by reporting the target quantity they are meant to predict. For example, squared-error scoring is proper for the conditional mean.

Scoring Claim Frequency

Suppose validation record $i$ has exposure $e_i$, observed claim count $N_i$, and predicted frequency $\hat{\lambda}_i$ per unit of exposure. Under a Poisson frequency model, conditional on the rating information and exposure, $N_i$ has a Poisson distribution with mean $m_i=e_i\hat{\lambda}_i$. The model assumes conditionally independent counts and equidispersion: the conditional variance of the count equals its conditional mean.

The Poisson negative log-likelihood, often called Poisson log loss, is the negative logarithm of the probability assigned to the observed count. Including the factorial term gives the full score below. The factorial term is constant across candidate models evaluated on the same outcomes, but including it makes the score the actual negative log probability.

L_i=m_i-N_i\ln(m_i)+\ln(N_i!)

The Poisson deviance contribution compares the fitted mean with the mean $N_i$ of a saturated model, which fits that observation exactly. It is zero when $m_i=N_i$, and is nonnegative. For $N_i=0$, the term $N_i\ln(N_i/m_i)$ is defined to be zero by continuity.

D_i=2\left[N_i\ln\left(\frac{N_i}{m_i}\right)-(N_i-m_i)\right]

For fixed validation outcomes, summed Poisson deviance and summed Poisson log loss rank models in the same order: their difference is a quantity that depends on the observations, not the fitted means. They are two presentations of the same likelihood comparison, not independent evidence. The offset for exposure matters: score the predicted count mean $e_i\hat{\lambda}_i$, not the frequency prediction by itself.

Worked Example: Comparing Two Frequency Models

A hypothetical validation set contains two policies. Policy A has exposure 1 and $N_A=0$; policy B has exposure 2 and $N_B=2$. Model 1 predicts frequency 1 claim per exposure for both policies. Model 2 predicts frequencies 0.5 and 1.5, respectively. Compare the models using total Poisson deviance and total Poisson log loss.

Method and predicted count means. Convert each predicted frequency to a count mean using $m_i=e_i\hat{\lambda}_i$. Model 1 has means $1(1)=1$ and $2(1)=2$. Model 2 has means $1(0.5)=0.5$ and $2(1.5)=3$.

Model 1 deviance. For policy A, $N=0,m=1$, so its contribution is $2[0-(0-1)]=2$. For policy B, $N=2,m=2$, so its contribution is $2[2\ln(1)-0]=0$. Total deviance is $2.0000$.

Model 2 deviance. Policy A contributes $2[0-(0-0.5)]=1$. Policy B contributes $2[2\ln(2/3)-(2-3)]=2[-0.810930+1]=0.378139$. The total is $1+0.378139=1.3781$, to four decimal places.

Check with Poisson log loss. For Model 1, policy A contributes $1$, and policy B contributes $2-2\ln(2)+\ln(2!)=2-\ln(2)=1.306853$. Total log loss is $2.3069$. For Model 2, policy A contributes $0.5$, and policy B contributes $3-2\ln(3)+\ln(2!)=1.495922$. Total log loss is $1.9959$. Both scores prefer Model 2 on these outcomes.

Check and trap avoided. Model 2 has the lower count-model score even though it predicts more than the observed count for policy B; the score considers both records and their exposure. Do not substitute frequency for count mean in the deviance formula. Here the Poisson deviance for policy B under Model 2 uses $m=2(1.5)=3$, not $1.5$.

Scoring Positive Claim Severity

For a severity validation record, let $Y_j>0$ be an observed claim amount and $\hat{\mu}_j>0$ the predicted mean severity. One likelihood-based option is Gamma deviance. It is appropriate when a Gamma model is used with conditional mean $\mu_j$ and variance $\phi\mu_j^2$, where the dispersion parameter $\phi$ is common across the candidate models being compared. This variance assumption allows variability to increase with the squared mean. The unscaled unit deviance is:

d_{\Gamma}(Y_j,\hat{\mu}_j) =2\left[\frac{Y_j-\hat{\mu}_j}{\hat{\mu}_j} -\ln\left(\frac{Y_j}{\hat{\mu}_j}\right)\right]

Summing these contributions gives total Gamma deviance. If the dispersion is fixed, dividing by $\phi$ does not change the ranking. Gamma deviance requires positive observed severities and positive fitted means; it is not directly defined for a zero severity. If the validation data contain only claims, that positive-claim condition is usually natural. If candidate models use different dispersion assumptions, compare their full likelihood scores rather than treating unscaled deviance alone as a complete comparison.

Squared error can also score severity predictions when the target is the conditional mean, but it penalizes errors in the original monetary units and can be strongly affected by large claims. Gamma deviance instead scores relative discrepancies in a way consistent with the stated Gamma mean-variance relationship. The right choice depends on the pricing target and the predictive distribution being assessed, not on which score happens to produce the preferred ranking.

Worked Example: Choosing a Severity Score

A hypothetical insurer validates two models on two positive claim severities, $Y_1=100$ and $Y_2=400$, in dollars. Model A predicts means 200 and 200; Model B predicts 150 and 300. Calculate each model’s total Gamma deviance and claim-level sum of squared errors. Use equal weight for the two claims.

Method and assumptions. Use Gamma deviance under a common-dispersion Gamma model, and calculate squared errors as $(Y_j-\hat{\mu}_j)^2$. The same two claims are used for both models, so a lower score indicates a better fit under the chosen score.

Model A Gamma deviance. For the first claim, the ratio $Y/\hat{\mu}=100/200=0.5$, giving $2[0.5-1-\ln(0.5)]=0.386294$. For the second claim, the ratio is $400/200=2$, giving $2[2-1-\ln(2)]=0.613706$. Total deviance is $0.386294+0.613706=1.0000$.

Model B Gamma deviance. The ratios are $100/150=2/3$ and $400/300=4/3$. The contributions are $2[2/3-1-\ln(2/3)]=0.144264$ and $2[4/3-1-\ln(4/3)]=0.091303$. Total deviance is $0.235567$, or $0.2356$ to four decimal places. Gamma deviance prefers Model B.

Squared-error check. Model A has errors $-100$ and $200$, so its sum is $100^2+200^2=50{,}000$ dollars squared. Model B has errors $-50$ and $100$, so its sum is $50^2+100^2=12{,}500$ dollars squared. Squared error also prefers Model B here.

Check and trap avoided. The agreeing rankings do not make the scores interchangeable: Gamma deviance is dimensionless, while squared error is in dollars squared and is sensitive to the claim-size scale. Do not compare raw score magnitudes across these two scoring rules as if they shared units.

Scoring Pure-Premium Predictions

Pure premium is expected loss per unit of exposure. For record $i$, write the observed aggregate loss as $S_i$, exposure as $e_i>0$, observed pure premium as $Y_i=S_i/e_i$, and predicted pure premium as $\hat{p}_i$. Squared error, $(Y_i-\hat{p}_i)^2$, directly penalizes the difference in dollars per exposure unit. If records have unequal exposure and the purpose is to give more credibility to records representing more exposure, an exposure-weighted score is:

\operatorname{SSE}_{e}=\sum_i e_i(Y_i-\hat{p}_i)^2

The weighting choice should reflect the validation objective and be used consistently for all candidates. An unweighted score gives every record equal influence; the displayed score gives greater influence to records with more exposure. Neither weighting should be applied mechanically without stating what a record represents. Squared error targets the conditional mean pure premium; it does not require a normally distributed outcome for that mean-targeting property.

Why squared error targets a mean: For a prediction $a$ of a random outcome $Y$ with finite second moment, the conditional expected squared error is minimized at the conditional mean. In fact,

E[(Y-a)^2\mid X] =\operatorname{Var}(Y\mid X)+\left(E[Y\mid X]-a\right)^2

To obtain the identity, add and subtract $\mu=E[Y\mid X]$, expand the square, and take the conditional expectation. The cross term is $2(\mu-a)E[Y-\mu\mid X]=0$, leaving the conditional variance plus $(\mu-a)^2$. Since the variance does not depend on $a$, the expression is minimized when $a=\mu$. Thus squared error is aligned with a mean target, while a likelihood-based score is aligned with the assumed outcome distribution.

Worked Example: Comparing Pure-Premium Predictions

A hypothetical validation portfolio has three records with exposures $1,2,1$ units. Their observed pure premiums are $\$80,\$160,\$240$ per exposure unit. Model A predicts $\$100,\$150,\$230$; Model B predicts $\$70,\$180,\$220$. Compare the models using exposure-weighted squared error.

Method. For each record, multiply the squared pure-premium error by its exposure, then sum. Units for the resulting score are exposure times dollars squared per exposure unit squared.

Model A. The three errors are $80-100=-20$, $160-150=10$, and $240-230=10$. The weighted contributions are $1(20^2)=400$, $2(10^2)=200$, and $1(10^2)=100$. Total score is $400+200+100=700$.

Model B. The errors are $80-70=10$, $160-180=-20$, and $240-220=20$. The weighted contributions are $1(10^2)=100$, $2(20^2)=800$, and $1(20^2)=400$. Total score is $100+800+400=1{,}300$.

Conclusion and check. Model A has the lower exposure-weighted score, 700 versus 1,300, so it performs better on this pure-premium criterion. The second record’s squared error is doubled because its exposure is 2. Omitting that weight would answer a different question and would produce totals of 600 for Model A and 900 for Model B.

How to Choose and Interpret a Score

The score should follow the modeled outcome and the quantity the pricing decision needs. A frequency model predicts counts conditional on exposure, so Poisson log loss or deviance evaluates the count distribution using the exposure-adjusted mean. A severity model predicts positive claim amounts, so Gamma deviance is a natural likelihood-based choice when the Gamma variance assumption is suitable. A pure-premium model predicts expected loss per exposure, for which squared error directly evaluates mean prediction accuracy.

Prediction target	Example score	Key condition or interpretation
Claim count	Poisson log loss or deviance	Use predicted count mean: exposure times predicted frequency; Poisson assumptions matter.
Positive claim severity	Gamma deviance	Requires positive outcomes and predictions; aligns with a Gamma mean-variance specification.
Pure premium	Squared error, optionally exposure-weighted	Targets the conditional mean; define whether records receive equal or exposure-based weight.

A lower score is evidence of better performance only for the stated score and validation sample. It is not proof that a model will be better in every future period or business segment. Compare candidates on identical validation records, use the same scoring definition and weights, and inspect the practical size of score differences. Likelihood scores from different outcome distributions, raw Gamma deviance and squared error, or totals based on different exposure weights are not directly comparable just because each is a single number.

Common Mistakes

Scoring frequency instead of count. The Poisson mean is $m=e\hat{\lambda}$. Leaving out exposure changes the score, particularly when exposures differ.
Applying the Poisson formula carelessly at zero. For $N=0$, use the limiting convention $N\ln(N/m)=0$; do not attempt to calculate $\ln(0)$.
Choosing a score after seeing which model wins. Select the score based on the prediction target and assumptions, not on the desired ranking.
Comparing score magnitudes across unlike rules. Squared error has squared monetary units; deviance is dimensionless. Their raw totals are not on a common scale.
Assuming every lower validation score establishes superior rates. A score evaluates a defined predictive target. It does not by itself settle calibration, stability, or whether the resulting rates are usable.

Key takeaway: Score the outcome the model predicts: exposure-adjusted counts with a count likelihood, positive severities with a suitable severity deviance, and pure premiums with a mean-aligned score such as squared error. State the assumptions and weighting before interpreting the ranking.

Check Your Understanding

For each problem, identify the target and apply the stated scoring rule. Scores should be rounded to four decimal places when needed.

A policy has exposure 3, observed count $N=0$, and predicted frequency $0.4$ claims per exposure unit. Under a Poisson model, calculate the deviance contribution. (Answer: 2.4000)
A policy has exposure 2, observed count $N=3$, and predicted frequency 1 claim per exposure unit. Calculate its Poisson deviance contribution. (Answer: 0.4328)
For a positive claim of $\$200$, a Gamma severity model predicts mean severity $\$100$. Calculate the unscaled Gamma deviance contribution. (Answer: 0.6137)
Two equally weighted validation records have observed pure premiums $\$50$ and $\$100$. A candidate predicts $\$60$ and $\$80$. Calculate the sum of squared errors in dollars squared. (Answer: 500)
A frequency validation record has exposure 4 and observed count 1. Compare Poisson log loss for predicted frequencies 0.25 and 0.5, including the $\ln(N!)$ term. Which prediction scores better? (Answer: Frequency 0.25 scores better; losses are 1.0000 and 1.3069, respectively.)

What You'll Learn