Scoring Frequency Predictions on Validation Data

Why Score Frequency Predictions at the Count Level?

A frequency model predicts an expected number of claims per unit of exposure, not necessarily the number of claims an individual risk will produce. Validation therefore requires two aligned quantities: the observed claim count and the model’s expected count over the same exposure period. Scoring the frequency as if it were already a count can make a sound prediction look poor—or a poor prediction look sound.

For example, a predicted frequency of 0.6 claims per policy-year implies an expected count of 0.3 for a risk with half a policy-year of exposure, but 1.2 for a risk with two policy-years. Poisson deviance measures the difference between observed and predicted counts while accounting for this exposure. It is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Definition: Let \(Y_i\) be the observed claim count for validation observation \(i\), and let \(e_i>0\) be its exposure in a stated unit, such as policy-years. If the model predicts frequency \(\hat{\lambda}_i\) claims per that same exposure unit, its predicted count is \(\hat{\mu}_i=e_i\hat{\lambda}_i\). Under the Poisson count model, \(Y_i\mid e_i,\hat{\lambda}_i\sim\operatorname{Poisson}(\hat{\mu}_i)\). The Poisson deviance contribution is \(d_i=2\left[Y_i\log(Y_i/\hat{\mu}_i)-(Y_i-\hat{\mu}_i)\right]\) when \(Y_i>0\), and \(d_i=2\hat{\mu}_i\) when \(Y_i=0\). Total deviance is \(D=\sum_i d_i\).

The prediction is a frequency; the deviance formula uses a count mean. The exposure unit must be consistent throughout. If exposure is measured in policy-years, predicted frequencies must be per policy-year. A record with zero exposure provides no count opportunity and is not scored by this formula; in ordinary validation, such a record is excluded from the scored set.

Why the Deviance Formula Has This Form

The Poisson deviance compares the fitted count mean with a saturated Poisson model, which assigns each observation its own mean equal to the observed count. A saturated model can match every observed count exactly. The difference in log likelihoods therefore measures how much fit is lost by using the model’s predicted means instead.

For a Poisson observation \(y\) with mean \(\mu\), its log likelihood, apart from the data-independent notation for the observation, is \(y\log(\mu)-\mu-\log(y!)\). At the saturated mean, the log likelihood is \(y\log(y)-y-\log(y!)\) for \(y>0\). Subtracting the fitted log likelihood from the saturated log likelihood gives:

2\{\ell_{\mathrm{sat}}-\ell_{\mathrm{fit}}\} =2\left[y\log\left(\frac{y}{\mu}\right)-(y-\mu)\right].

When \(y=0\), the term \(y\log(y/\mu)\) is defined by its limiting value, zero. Substitution leaves \(2\mu\), the stated zero-count contribution. Thus, a prediction of a positive count mean is penalized even when no claim occurs; the model assigned probability to claims that were not observed.

For a frequency GLM with a log link, the exposure offset expresses this same relationship in the model’s linear predictor:

\log(\hat{\mu}_i)=\log(e_i)+\log(\hat{\lambda}_i).

The offset \(\log(e_i)\) is not a freely estimated rating factor. It ensures the model’s expected count scales with exposure while the modeled component represents frequency per exposure unit.

Worked Examples

Worked Example: Zero Claims and Unequal Exposure

A hypothetical validation set contains two risks. Risk 1 has exposure 0.5 policy-years and one observed claim. Risk 2 has exposure 1.5 policy-years and no observed claims. Both have a predicted frequency of 1 claim per policy-year. Calculate each expected count, the total Poisson deviance, and deviance per policy-year. Use natural logarithms and round final scores to four decimal places.

Method. Convert each predicted frequency to an expected count by multiplying by exposure, then calculate the count-level deviance contributions. This prevents treating a frequency as though it were a count.

For Risk 1, \(\hat{\mu}_1=0.5(1)=0.5\). Since \(Y_1=1\), \(d_1=2[1\log(1/0.5)-(1-0.5)]\). Here \(\log(2)=0.693147\), so \(d_1=2(0.693147-0.5)=0.386294\).

For Risk 2, \(\hat{\mu}_2=1.5(1)=1.5\). Since \(Y_2=0\), use the zero-count form: \(d_2=2(1.5)=3.000000\).

Total deviance is \(D=0.386294+3.000000=3.386294\), or 3.3863. Total exposure is \(0.5+1.5=2\) policy-years, so deviance per policy-year is \(3.386294/2=1.693147\), or 1.6931.

Check. The zero-claim risk contributes 3.0000 by itself because the model expected 1.5 claims there. The total must therefore exceed 3.0000. The common mistake is to omit or miscalculate that zero-count contribution.

Worked Example: Comparing Two Frequency Models

A hypothetical risk has 2 policy-years of validation exposure and one observed claim. Model A predicts 0.25 claims per policy-year, while Model B predicts 0.75 claims per policy-year. Calculate each deviance and identify the lower-scoring model. Round final scores to four decimal places.

Method. Score both models against the same observed count and exposure. Their expected counts are not their frequencies: they are exposure multiplied by frequency.

For Model A, \(\hat{\mu}_A=2(0.25)=0.5\). Therefore, \(D_A=2[1\log(1/0.5)-(1-0.5)]\) \(=2[0.693147-0.5]=0.386294\), or 0.3863.

For Model B, \(\hat{\mu}_B=2(0.75)=1.5\). Therefore, \(D_B=2[1\log(1/1.5)-(1-1.5)]\) \(=2[-0.405465+0.5]=0.189070\), or 0.1891.

Model B has the lower deviance for this validation observation. Its expected count of 1.5 is closer to the observed count of 1 in the likelihood-based deviance sense. This single result does not establish that Model B is preferable overall; a model comparison should use the same complete validation observations and consider the aggregate score.

Check. Both expected counts are positive, and the observed count and exposure are identical for both models. The common mistake is to calculate Model B’s expected count incorrectly: \(2(0.75)=1.5\), not 0.75.

Worked Example: Aggregate Deviance and the Poisson Negative Log Score

A hypothetical portfolio has three validation records. Their exposure and observed counts are respectively \((1,0)\), \((2,2)\), and \((1,3)\), where exposure is in policy-years. A model predicts frequencies of 0.5, 0.75, and 1.5 claims per policy-year. Calculate total deviance, deviance per policy-year, and the total Poisson negative log score. Round final scores to four decimal places.

Method. First compute expected counts, then sum the individual deviance contributions. The Poisson negative log score is \(-\log P(Y_i=y_i\mid\hat{\mu}_i)=\hat{\mu}_i-y_i\log(\hat{\mu}_i)+\log(y_i!)\).

The expected counts are \(\hat{\mu}_1=1(0.5)=0.5\), \(\hat{\mu}_2=2(0.75)=1.5\), and \(\hat{\mu}_3=1(1.5)=1.5\).

For record 1, \(Y_1=0\), so \(d_1=2(0.5)=1.000000\). For record 2, \(d_2=2[2\log(2/1.5)-(2-1.5)]\) \(=2[2\log(4/3)-0.5]\) \(=2[0.575364-0.5]=0.150728\). For record 3, \(d_3=2[3\log(3/1.5)-(3-1.5)]\) \(=2[3\log(2)-1.5]\) \(=2[2.079442-1.5]=1.158883\).

Thus \(D=1.000000+0.150728+1.158883=2.309611\), or 2.3096. Total exposure is \(1+2+1=4\) policy-years. Deviance per policy-year is \(2.309611/4=0.577403\), or 0.5774.

For the negative log score, the three contributions are: \(0.5\) for record 1; \(1.5-2\log(1.5)+\log(2!)=1.382217\) for record 2; and \(1.5-3\log(1.5)+\log(3!)=2.075364\) for record 3. The total is \(0.5+1.382217+2.075364=3.957581\), or 3.9576.

Check. The total deviance is the sum of the three contributions, and deviance per policy-year divides that total by four—not by the number of records. The negative log score is a different quantity, but on these same observations it ranks candidate Poisson mean predictions in the same order as deviance.

Interpreting and Comparing Count Scores

A smaller deviance indicates a closer fit to the observed counts relative to the Poisson saturated model, for the observations scored. The absolute size of a total deviance depends on the size and exposure of the validation set, so totals are most directly comparable when the same records are scored. Dividing by total exposure gives a descriptive deviance-per-exposure measure that can help summarize sets with different exposure volumes. It is not a claim that each policy-year is an independent scored record.

The Poisson negative log score uses the probability assigned to each observed count. For observation \(i\), it is:

-\log P(Y_i=y_i\mid\hat{\mu}_i) =\hat{\mu}_i-y_i\log(\hat{\mu}_i)+\log(y_i!).

For fixed observed counts, the negative log score and Poisson deviance differ only by a term determined by the observations, not by the candidate model’s predictions. Consequently, they give the same ranking when applied to the same validation records with the same exposure and count definitions. Their reported numerical values are not interchangeable. Other simple summaries, such as squared error \((Y_i-\hat{\mu}_i)^2\), also measure count prediction error, but they do not have the same likelihood interpretation and can respond differently to large deviations.

Key takeaway: Score a frequency model by first forming \(\hat{\mu}_i=e_i\hat{\lambda}_i\), then comparing that expected count with \(Y_i\). Keep exposure units consistent, sum contributions at the observation level, and state the denominator whenever reporting a normalized score.

Common Mistakes and Conditions

Using frequency in place of expected count. The deviance formula uses \(\hat{\mu}_i=e_i\hat{\lambda}_i\). A frequency of 0.7 over 2 policy-years implies a mean count of 1.4.
Scoring zero counts as zero deviance. If \(Y_i=0\), the contribution is \(2\hat{\mu}_i\), not zero. A model that predicts claims but observes none is penalized.
Mixing exposure units. If exposure is in months, convert either exposure or frequency so their product has the intended expected-count interpretation. Do not multiply a monthly exposure by an annual frequency without conversion.
Dividing by the number of records without saying so. Total deviance, mean deviance per record, and deviance per exposure are different summaries. Specify which is reported.
Comparing scores on different validation data. Different observations or exposure definitions can change a score independently of model quality. For a direct model comparison, evaluate both candidates on identical validation records.
Over-interpreting the Poisson assumption. The likelihood interpretation assumes conditional Poisson counts, including conditional mean and variance both equal to \(\hat{\mu}_i\). If counts are materially overdispersed or dependent, Poisson deviance can still be used as a stated predictive score, but it is not by itself evidence that the Poisson distribution is an adequate model.

Check Your Understanding

For each problem, use exposure-adjusted expected counts and natural logarithms. Round numerical answers to four decimal places.

A risk has 3 policy-years of exposure, no observed claims, and predicted frequency 0.4 per policy-year. Find its expected count and deviance. (Answer: expected count 1.2; deviance 2.4000)
A risk has 0.75 policy-years of exposure, 2 observed claims, and predicted frequency 2 claims per policy-year. Calculate its expected count and deviance. (Answer: expected count 1.5; deviance 0.1507)
Two records have exposures 1 and 2 policy-years, observed counts 1 and 0, and predicted frequencies of 0.5 per policy-year for both. Find total deviance and deviance per policy-year. (Answer: total deviance 2.3863; deviance per policy-year 0.7954)
A risk has 1.5 policy-years of exposure and 2 observed claims. Model A predicts frequency 1 per policy-year; Model B predicts frequency \(4/3\) per policy-year. Calculate both deviances and identify the lower-scoring model. (Answer: Model A deviance 0.1507; Model B deviance 0.0000; Model B)
A record has 2 policy-years of exposure and no claims. Model A predicts frequency 0.5 per policy-year and Model B predicts frequency 1 per policy-year. Calculate each deviance and identify the model with the smaller score. (Answer: Model A deviance 2.0000; Model B deviance 4.0000; Model A)

What You'll Learn