Comparing Logistic GLMs for Claim Incidence

Why Compare Logistic Claim-Incidence Models?

A claim-incidence model predicts whether a policy has at least one claim during a defined observation period. Its response is binary: \(Y_i=1\) if policy \(i\) has a claim and \(Y_i=0\) otherwise. A logistic generalized linear model (GLM) assigns a predicted probability \(p_i\) to each policy. When comparing candidate models, the important question is not simply which model fits the observed data best. A model with more coefficients can usually improve its likelihood, so comparison must also consider complexity and, where the goal is prediction, how well the fitted probabilities perform on data not used to fit the models.

This tutorial focuses on comparing logistic GLMs for claim incidence using likelihood-based evidence, complexity penalties, and predicted probabilities. It is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: In a logistic GLM for independent binary outcomes, \(Y_i\) is modeled as Bernoulli with \(P(Y_i=1\mid \mathbf{x}_i)=p_i\), where \(\mathbf{x}_i\) is the predictor information for observation \(i\). The logit link specifies \(\log\left(p_i/(1-p_i)\right)=\eta_i\), with \(\eta_i\) a linear predictor. Thus \(p_i=1/(1+\exp(-\eta_i))\).

The likelihood measures how well a model assigns probability to the outcomes that actually occurred. For independent observations, a claim contributes \(p_i\) to the likelihood, while a non-claim contributes \(1-p_i\). The log-likelihood is therefore:

\ell=\sum_{i=1}^{n}\left[y_i\log(p_i)+(1-y_i)\log(1-p_i)\right].

This form assumes that the outcomes are conditionally independent given the predictors and that each outcome is Bernoulli with the model-specified probability. The logarithm is natural. A larger log-likelihood (closer to zero) indicates a better fit to the observed outcomes, but it does not by itself account for the number of estimated coefficients.

Likelihood, Deviance, and Complexity

For nested models fitted to the same response observations, the reduced model is nested in the full model: the reduced model is obtained by constraining one or more full-model coefficients. The full model cannot have a lower maximized likelihood, because it can reproduce the reduced model by setting the added coefficients to their constrained values.

Theorem: For nested models fitted by maximum likelihood to the same observations, the likelihood-ratio statistic is \(2(\ell_{\text{full}}-\ell_{\text{reduced}})\). Under the null hypothesis that the added coefficients are zero, and under standard regularity conditions, this statistic has an approximate chi-square distribution with degrees of freedom equal to the number of added coefficients. The approximation may be unreliable in small samples, with boundary parameters, or with separation or other fitting problems.

The likelihood-ratio comparison asks whether the observed fit improvement is sufficiently large relative to the added parameters. For non-nested models, this particular chi-square reference does not apply. AIC provides a different comparison:

\mathrm{AIC}=-2\ell+2k,

where \(k\) is the number of estimated model parameters, including the intercept. Smaller AIC is preferred among candidate models fitted to the same response data under comparable likelihood specifications. AIC balances fit against complexity; it is not a hypothesis test and a small difference need not establish a decisive winner.

Why the Bernoulli Log-Likelihood Has This Form

The likelihood expression follows directly from the probability model. For a Bernoulli outcome, \(P(Y_i=y_i)=p_i^{y_i}(1-p_i)^{1-y_i}\): if \(y_i=1\), this is \(p_i\), and if \(y_i=0\), it is \(1-p_i\). Conditional independence means the joint likelihood is the product of these contributions. Taking the natural logarithm turns the product into a sum:

L=\prod_{i=1}^{n}p_i^{y_i}(1-p_i)^{1-y_i} \quad\Longrightarrow\quad \ell=\log L =\sum_{i=1}^{n}\left[y_i\log(p_i)+(1-y_i)\log(1-p_i)\right].

This derivation also explains why a probability close to zero assigned to an observed claim, or close to one assigned to an observed non-claim, incurs a large negative log-likelihood contribution. Likelihood comparison therefore evaluates the probabilities assigned to the outcomes, not just whether a classification threshold gives the right label.

Worked Example: A Rating Group Improves Fit but Not AIC

A hypothetical insurer observes 12 policies over a common period. Three have at least one claim. Six policies are in each of two rating groups: Group L has one claim and five non-claims; Group H has two claims and four non-claims. Compare an intercept-only logistic model with a model having a separate incidence probability for each group. Use maximum-likelihood estimates and report log-likelihoods to six decimal places and AIC to three decimals.

Method and estimates. The intercept-only estimate is the overall claim proportion, \(3/12=0.25\). In the group model, the estimated probabilities are \(1/6\) for Group L and \(2/6=1/3\) for Group H. The group model adds one coefficient relative to the intercept-only model: the reduced model has one parameter and the full model has two.

Intercept-only likelihood. There are three claims and nine non-claims, so

\ell_{\text{reduced}} =3\log(0.25)+9\log(0.75) =-6.748021735\ldots =-6.748022.

Group-model likelihood. The two group contributions are calculated using their observed claim and non-claim counts:

\begin{aligned} \ell_{\text{full}} &=\log(1/6)+5\log(5/6)+2\log(1/3)+4\log(2/3)\\ &=-1.791759469-0.911607784-2.197224577-1.621860433\\ &=-6.522452263\ldots=-6.522452. \end{aligned}

The full model has the larger log-likelihood, as expected. The likelihood-ratio statistic is \(2[-6.522452263-(-6.748021735)]=0.451138944\), approximately \(0.451\) on one degree of freedom. The AIC values are

\begin{aligned} \mathrm{AIC}_{\text{reduced}}&=-2(-6.748021735)+2(1)=15.496043470\ldots=15.496,\\ \mathrm{AIC}_{\text{full}}&=-2(-6.522452263)+2(2)=17.044904526\ldots=17.045. \end{aligned}

Conclusion and trap avoided. The group model fits better by likelihood, but the intercept-only model has lower AIC because the likelihood improvement does not offset the added-parameter penalty. It would be a mistake to conclude that the full model is preferred merely because its log-likelihood is larger.

Use Predicted Probabilities to Assess Prediction

A likelihood-based comparison describes fit on the data used to estimate the models. If the practical objective is prediction, compare predicted probabilities on a separate validation or test set that was not used to fit the candidate models. Two common measures are log loss and the Brier score. For a test set of \(m\) binary outcomes, log loss is the average negative Bernoulli log-likelihood; the Brier score is the mean squared difference between the outcome and predicted probability:

\mathrm{LogLoss} =-\frac{1}{m}\sum_{i=1}^{m} \left[y_i\log(p_i)+(1-y_i)\log(1-p_i)\right], \qquad \mathrm{Brier} =\frac{1}{m}\sum_{i=1}^{m}(y_i-p_i)^2.

Both measures are smaller when predictions are better. Log loss penalizes assigning very small probability to an event that occurs (or very high probability to an event that does not occur) especially strongly. The Brier score penalizes squared probability errors. These scores evaluate probabilities, not only classifications at a chosen threshold.

Worked Example: Compare Two Models on Held-Out Policies

Two fitted logistic models predict claim probabilities for four held-out policies. The observed outcomes, in order, are \(1,0,1,0\). Model A predicts \(0.8000, 0.2000, 0.4000, 0.3000\); Model B predicts \(0.7000, 0.1000, 0.6000, 0.2000\). Calculate each model's average log loss and Brier score, retaining full precision until the final rounding to six decimals.

Model A log loss. The four contributions are \(-\log(0.8)\), \(-\log(1-0.2)\), \(-\log(0.4)\), and \(-\log(1-0.3)\). Thus

\begin{aligned} \text{total loss}_A &=0.223143551+0.223143551+0.916290732+0.356674944\\ &=1.719252778,\\ \mathrm{LogLoss}_A&=1.719252778/4=0.429813. \end{aligned}

Model B log loss. Its contributions are \(-\log(0.7)\), \(-\log(1-0.1)\), \(-\log(0.6)\), and \(-\log(1-0.2)\):

\begin{aligned} \text{total loss}_B &=0.356674944+0.105360516+0.510825624+0.223143551\\ &=1.196004635,\\ \mathrm{LogLoss}_B&=1.196004635/4=0.299001. \end{aligned}

Brier scores. For Model A the squared errors are \(0.04,0.04,0.36,0.09\). Their total is \(0.53\), giving \(0.53/4=0.132500\). For Model B they are \(0.09,0.01,0.16,0.04\). Their total is \(0.30\), giving \(0.30/4=0.075000\).

Conclusion and trap avoided. Model B has lower log loss and lower Brier score on these held-out policies, so it performs better on both probability measures here. Do not calculate validation scores on the training data and then describe them as independent evidence of predictive performance: the fitted model has already used those outcomes.

Interpretation and Common Comparison Traps

A model comparison is meaningful only when the comparison matches the question and the data support the calculation. For nested logistic models, the likelihood-ratio statistic evaluates the added terms under its stated assumptions. AIC compares fit and parameter count. Held-out log loss and Brier score compare predictions. These measures can favor different models without contradiction because they answer different questions.

Reversing nesting direction. If a full model adds a coefficient, the reduced model is nested in the full model. The full model has at least as large a maximized likelihood, not the other way around.
Ignoring the intercept in AIC. Count all estimated coefficients in \(k\), including the intercept. Omitting it understates the complexity penalty.
Using a likelihood-ratio test for non-nested models. The usual chi-square result requires nested models and regularity conditions. It is not a general test for any two candidate GLMs.
Confusing fit with prediction. A larger training log-likelihood is not proof of better performance on new policies. Use held-out predictions when prediction is the objective.
Comparing different response data or likelihood definitions as if equivalent. The likelihood and criterion must be calculated on comparable observations and response specifications. A change in the data set can make the numerical comparison misleading.
Reading a small criterion difference as certainty. AIC ranks models by its defined trade-off; it does not establish that the lower-AIC model is the true data-generating model.

Key takeaway: First identify whether the candidate logistic GLMs are nested and fitted to the same observations. Then use likelihood-ratio evidence only where its assumptions apply, count every estimated coefficient in AIC, and assess predicted probabilities on held-out data when prediction is the goal.

Check Your Understanding

For each original problem, calculate the requested comparison and check that the method matches the question.

A hypothetical portfolio has 10 policies, of which two have claims. An intercept-only model estimates one common incidence probability. Calculate its maximized log-likelihood to six decimals. (Answer: \(-5.004024\))
Two nested logistic models use the same outcomes. The reduced model has \(\ell=-40.2\) and \(k=2\); the full model has \(\ell=-38.7\) and \(k=3\). Calculate the likelihood-ratio statistic and each AIC, then state the AIC preference. (Answer: likelihood-ratio statistic \(3.0\) on 1 degree of freedom; reduced AIC \(84.4\), full AIC \(83.4\); AIC prefers the full model.)
For three held-out policies, outcomes are \(1,0,1\) and predicted probabilities are \(0.7000,0.2000,0.4000\). Calculate average log loss to six decimals. (Answer: \(0.498703\))
Two models predict for outcomes \(1,0\). Model A predicts \(0.6000,0.4000\), while Model B predicts \(0.8000,0.1000\). Calculate each Brier score and identify the lower one. (Answer: Model A \(0.160000\); Model B \(0.025000\); Model B is lower.)
Two non-nested models are fitted to the same binary outcomes. Model A has \(\ell=-22.0\) and \(k=4\); Model B has \(\ell=-20.5\) and \(k=6\). Calculate their AIC values and state why a nested-model likelihood-ratio test is not the appropriate comparison. (Answer: Model A AIC \(52\); Model B AIC \(53\); AIC prefers Model A, and the ordinary nested-model likelihood-ratio test does not apply because the models are non-nested.)

What You'll Learn