Deviance Scoring for Ratemaking Predictions

Why Score Predictions With Deviance?

A validation score should measure how well predictions describe outcomes that were not used to fit the model. Deviance provides one such measure for generalized linear models (GLMs): it compares the fitted model’s likelihood with the best fit that could be achieved for those same observations. That best-fitting reference is called the saturated model, which gives each observation its own fitted mean.

A smaller deviance means the predictions are closer, in the model’s likelihood scale, to that saturated reference. Deviance is not a dollar error, and its numerical size depends on the response family, the observations, and any weights. Thus, it is most directly useful for comparing predictions on the same validation data under the same response family and scoring convention.

For an exponential-dispersion model with dispersion parameter \(\phi\), the likelihood-ratio statistic is commonly written as \(D/\phi\), where \(D\) is the unscaled deviance. This tutorial reports \(D\) unless a scaled quantity is explicitly requested. When comparing models within the same family and with the same fixed \(\phi\), scaling by \(\phi\) does not change which model has the smaller score.

Three Family-Specific Deviances

The response family determines the unit deviance. The formulas below use a fitted mean \(\hat{\mu}_i\) for each validation response \(y_i\). With case weights \(w_i\), multiply each unit-deviance term by \(w_i\). Without weights, take \(w_i=1\).

For a Poisson model, \(y_i\) is a nonnegative count and \(\hat{\mu}_i>0\) is its predicted mean count. If the model predicts a frequency rate \(\hat{\lambda}_i\) for exposure \(e_i\), the predicted count mean is \(\hat{\mu}_i=e_i\hat{\lambda}_i\). Score the observed count against that mean. Equivalently, if scoring observed frequency \(y_i/e_i\), the corresponding exposure-weighted form uses weight \(e_i\).

The Gamma formula applies to strictly positive responses, such as positive claim severities under a Gamma GLM. It does not directly score zero observations: the logarithm is undefined at \(y_i=0\), and a standard Gamma response is positive. A dataset containing zeros therefore requires a model and scoring rule appropriate to those zeros, rather than substituting zero into this formula.

For a Tweedie model with variance function proportional to \(\mu^p\), the commonly used compound Poisson-Gamma setting has \(1<p<2\). Responses are nonnegative, so zero outcomes are permitted.

For \(y_i=0\) and \(1<p<2\), the first two terms are zero, giving the unit deviance \(2\hat{\mu}_i^{\,2-p}/(2-p)\). The exponent \(p\) is part of the scoring specification: it must be stated or otherwise fixed to calculate the Tweedie deviance.

Why Deviance Measures Distance From Saturated Fit

The formulas arise from comparing log likelihoods. For a specified family, the saturated model chooses its mean for observation \(i\) to equal the observed response, \(y_i\). The fitted model instead uses \(\hat{\mu}_i\). Twice the difference between the saturated log likelihood and the fitted log likelihood, summed over observations, gives the likelihood-ratio deviance. Terms that depend only on the observed data cancel in this difference.

For example, suppose independent validation counts follow Poisson distributions with means \(\hat{\mu}_i\). Their log likelihood, omitting no parameter-dependent terms, is \(-\sum_i \hat{\mu}_i+\sum_i y_i\log(\hat{\mu}_i)-\sum_i\log(y_i!)\). The saturated Poisson log likelihood uses mean \(y_i\). Subtracting the fitted log likelihood from the saturated log likelihood cancels the factorial terms; multiplying the difference by two gives the Poisson deviance formula. For a zero count, the saturated log-likelihood contribution involving \(y_i\log(y_i)\) is interpreted by its limit as zero, which explains the stated zero convention.

This construction explains both the reference point and an important limitation: the saturated model is a benchmark on the observed validation responses, not a feasible prediction rule for future risks. A low deviance indicates that the model predicts these validation outcomes relatively well on the chosen likelihood scale; it does not establish that the model is calibrated, stable over time, or appropriate for every business use.

Calculating and Interpreting Validation Deviance

Practical Interpretation and Exam Traps

When reporting validation deviance, state the response, family, any weights, the Tweedie power where relevant, and whether the score is scaled or unscaled. These details make the score reproducible and prevent comparisons of unlike quantities. In particular, a Poisson score for counts and a Gamma score for severity are not directly comparable just because one is numerically smaller.

Deviance also depends on the validation sample composition. Scores can change when observations or weights change, even if the predictions for a subset of risks remain identical. For policy frequency, for example, scoring counts with their exposure-adjusted means reflects the amount of time each policy was observed. Applying weights inconsistently changes the implicit contribution of records.

A frequent exam error is to use a familiar-looking formula without checking the response family. The Poisson expression handles zero counts by a limit convention; the Gamma expression requires positive observations; and the Tweedie expression needs its variance power. Another error is to interpret zero deviance as proof that a model will predict future experience perfectly: it only says that the model’s fitted means match the scored responses in the data being evaluated.