Why GLM Comparisons Can Disagree
Two generalized linear models can produce different rankings because comparison measures answer different questions. A likelihood criterion summarizes how well a model fits the data used to estimate it, with a penalty for the number of estimated parameters. A validation score measures predictive performance on observations not used for fitting, according to a chosen loss function. Neither ranking is automatically the right one for every purpose.
For example, a more complex GLM may improve its likelihood enough to receive a slightly better Akaike information criterion (AIC), yet predict new observations less accurately. A small validation sample, an unstable model, or a validation metric that does not match the intended use may also affect the apparent ranking. The task is not to declare one measure universally superior; it is to identify what each result says and make a choice appropriate to the modeling objective. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
What the Measures Evaluate
For a fitted model with maximized likelihood \(L\) and \(k\) estimated parameters, the AIC is
Lower AIC is preferred among candidate likelihood models fitted to the same response data under comparable likelihood definitions. The likelihood term rewards fit, while \(2k\) penalizes the number of estimated parameters. Count the parameters according to the fitted likelihood model, including an estimated dispersion parameter when the likelihood convention requires it. AIC does not require the models to be nested, but their likelihoods must be comparable.
This criterion is motivated by estimating predictive information loss, not by directly measuring prediction error on a particular held-out sample. Its penalty is an approximation under regularity conditions, including a correctly specified likelihood framework and a sufficiently informative sample relative to model complexity. AIC is not a guarantee that the model with the smallest value will win on every future sample.
Validation measures instead score predictions on data held out from fitting. For quantitative outcomes, a common choice is root mean squared error (RMSE). For binary outcomes, a probability score such as log loss assesses the predicted probabilities. The chosen metric matters: a measure focused on squared error penalizes large misses more strongly than one based on absolute error, while a classification threshold discards information about the quality of predicted probabilities.
A Short Justification for the AIC Trade-Off
The AIC expression combines two competing quantities. The maximized log-likelihood \(\log L\) is larger when the fitted model assigns higher likelihood to the observed response data, so \(-2\log L\) is smaller for better fit. Adding parameters cannot reduce the maximized likelihood when the simpler model is contained in the more complex one, but it can make the fitted model more adaptable to the sample. The term \(2k\) offsets some of the resulting optimism.
More formally, under regularity conditions, the expected difference between the fitted-data deviance and the model's expected out-of-sample deviance is approximately \(2k\). Thus, adding \(2k\) to the fitted-data quantity \(-2\log L\) gives an estimate of relative predictive information loss. This is an approximation for model comparison, not an exact correction for every sample or model. In particular, a small AIC advantage should not be interpreted as proof of meaningfully better future prediction.
Worked Examples
Worked Example: AIC Slightly Prefers One Count Model, but Validation Prefers Another
A hypothetical insurer compares two Poisson GLMs for claim counts using the same training observations. Model A has \(k=4\) estimated parameters and maximized log-likelihood \(\log L=-82.0\). Model B has \(k=7\) and \(\log L=-78.5\). On a separate validation set, the observed counts are \(0,1,2,4\). The models predict, respectively, \(0.8,1.2,2.0,3.0\) and \(0.2,2.5,0.8,4.5\). Compare AIC and validation RMSE, then state what the evidence supports.
Method. Calculate AIC from each model's training log-likelihood and parameter count. Then calculate validation RMSE from the four observed-predicted pairs. This separates fitted likelihood evidence from the specified prediction target.
For Model A:
For Model B:
Model B has the lower AIC, by \(172-171=1\). This is a small difference, so it is evidence in favor of B under AIC, not a decisive reason to disregard other evidence.
For validation RMSE, square each prediction error, average the four squared errors, and take the square root. For A, the errors \(y-\hat{y}\) are \(-0.8,-0.2,0,1.0\), so the squared errors are \(0.64,0.04,0,1.00\). Therefore:
For B, the errors are \(-0.2,-1.5,1.2,-0.5\), giving squared errors \(0.04,2.25,1.44,0.25\). Therefore:
On this validation set, A has the lower RMSE. If prediction of counts on similar cases is the main objective, these results favor A, although four observations provide limited evidence. The small AIC advantage for B does not override a clearly stated prediction target. A further check on a larger or repeated validation design would help assess whether the RMSE ranking is stable.
Common mistake. Choosing B solely because \(171<172\) treats a one-point AIC advantage as a conclusive predictive result. The correction is to report both findings: B has slightly lower AIC, while A has lower validation RMSE on this sample. Then base the decision on the intended use and the reliability of validation evidence.
Worked Example: A Logistic Model With Better AIC but Worse Validation Log Loss
A hypothetical program models whether a policyholder renews. On the same training data, logistic model C has \(k=3\) parameters and \(\log L=-51.2\), while model D has \(k=5\) and \(\log L=-48.5\). For four held-out policyholders, the outcomes are \(1,0,1,0\). Model C predicts renewal probabilities \(0.80,0.20,0.60,0.40\); model D predicts \(0.99,0.40,0.51,0.40\). Compare AIC with mean validation log loss, where a predicted probability \(p\) receives loss \(-\log(p)\) for outcome 1 and \(-\log(1-p)\) for outcome 0.
Method. Calculate the two AIC values, then apply the outcome-specific log-loss formula to each held-out case. This checks predictive probabilities rather than only whether classifications cross a threshold.
For C:
For D:
D has lower AIC by \(108.4-107=1.4\). Now compute validation losses. For C, the four losses are \(-\log(0.80)=0.2231\), \(-\log(1-0.20)=0.2231\), \(-\log(0.60)=0.5108\), and \(-\log(1-0.40)=0.5108\). Their sum is \(1.4678\), so the mean is \(1.4678/4=0.3670\), rounded to four decimals.
For D, the four losses are \(-\log(0.99)=0.0101\), \(-\log(1-0.40)=0.5108\), \(-\log(0.51)=0.6733\), and \(-\log(1-0.40)=0.5108\). Their sum is \(1.7050\), so the mean is \(1.7050/4=0.4263\), rounded to four decimals. The rounded values are checked against the unrounded logarithms.
C has the lower validation log loss, despite D's lower AIC. For a task where accurate renewal probabilities matter, C is preferable on this validation sample. D assigns probability \(0.99\) to an event that does not occur, which incurs a substantial log loss. The result does not establish that C will always predict better; four cases are too few for a stable general conclusion.
Common mistake. Treating the probability \(0.40\) as the loss contribution for an observed non-renewal is incorrect. The loss is \(-\log(1-0.40)\), because the observed outcome is zero. The score must be applied to the probability of the outcome that actually occurred.
Worked Example: A Noisy Cross-Validation Ranking Does Not Settle the Choice
Two hypothetical Poisson GLMs are compared for predicting counts. Model E has \(k=3\) and training \(\log L=-120\); model F has \(k=6\) and \(\log L=-116\). Five validation folds give RMSE values \(3.8,4.0,4.1,3.9,4.2\) for E and \(3.5,4.8,3.6,4.7,3.9\) for F. Find the AIC values and mean fold RMSEs, and explain whether the results establish a predictive winner.
Method. Compute AIC from each training fit, then average the five fold-level RMSE values. Examine the fold-to-fold variation before interpreting the small difference between averages.
For E:
For F:
F has the lower AIC by 2. The mean RMSE for E is \((3.8+4.0+4.1+3.9+4.2)/5=20.0/5=4.00\). For F, it is \((3.5+4.8+3.6+4.7+3.9)/5=20.5/5=4.10\). The cross-validation averages therefore favor E by only \(0.10\) RMSE units.
The fold results vary considerably, especially for F, whose scores range from \(3.5\) to \(4.8\). The average ranking is not enough to claim that E will reliably predict better. The folds are also not fully independent because their training sets overlap, so a simple calculation treating the five scores as independent observations would not by itself provide a definitive uncertainty test.
A defensible conclusion is that AIC modestly favors F, while the mean validation RMSE modestly favors E, with substantial fold variation. If stable prediction is important, use a carefully chosen repeated validation design or additional data, and confirm that the validation metric matches the operational objective. Until then, present the conflict and avoid an overconfident winner.
Common mistake. Selecting E simply because \(4.00<4.10\) ignores the scale of the difference and the fold variation. The correction is to report the modest average advantage and the instability, then state that the ranking is uncertain.
A Practical Resolution Process
When comparison measures disagree, work through the evidence in a fixed order. This prevents an attractive score from concealing a mismatch in data, likelihood, or purpose.
Check that the candidates use the same response observations and comparable likelihood definitions for AIC. Check that validation scores use the same held-out cases and outcome scale. A score comparison is not fair if its data or target differs without a clear reason.
Decide whether the goal is explanation of associations, estimation of event probabilities, ranking, or prediction of a quantitative outcome. A prediction metric should reflect the errors that matter for that use.
Report the actual score differences, not only which model is first. A small AIC difference or a small validation improvement may be weak evidence. For validation, examine the number of cases, fold variation, and whether a single split may have driven the result.
Validation data should represent the observations to which predictions will be applied. A random split may not assess future-time performance when conditions change over time. If the design does not represent the intended population or time period, the score may not answer the operational question.
If prediction is the primary goal and validation evidence is credible, give it substantial weight while reporting the likelihood result. If the rankings are close or unstable, say that the evidence does not clearly distinguish the candidates and recommend further validation rather than implying certainty.
Common Traps and Reporting
- Calling a lower AIC “better prediction” without qualification. AIC estimates relative predictive information loss under conditions; it is not a direct validation score for the specific prediction task.
- Comparing noncomparable quantities. AIC values from different response samples, incompatible likelihood conventions, or different response definitions do not give a meaningful ranking.
- Choosing a metric after seeing the result. Select a validation measure based on the task, not because it produces a preferred model ranking. For probability prediction, evaluate probabilities with an appropriate score rather than relying only on a thresholded classification.
- Overstating a small difference. A numerical winner is not automatically a practically important winner. State the size of the difference and limitations in the validation design.
- Reusing validation data until a preferred result appears. Repeatedly changing the model in response to the same holdout can make that holdout part of the model-development process. Use an appropriate validation strategy and reserve independent evaluation when possible.
A clear report can state: “Model F has the lower AIC by 2, while Model E has the lower mean validation RMSE by 0.10. Fold scores vary substantially, so the available results do not establish a reliable predictive advantage. Further validation is appropriate before choosing a model for deployment.” This identifies the conflict, puts it in context, and avoids a claim stronger than the evidence.
Check Your Understanding
For each problem, identify the ranking and what a careful model comparison should conclude.
- Two Gaussian GLMs use the same training data. Model G has \(k=3\) and \(\log L=-40\); model H has \(k=5\) and \(\log L=-37\). Which has lower AIC, and by how much? (Answer: \(\mathrm{AIC}_G=86\), \(\mathrm{AIC}_H=84\); H is lower by 2.)
- On six held-out quantitative outcomes, a model has squared prediction errors summing to 24. What is its RMSE? (Answer: \(\sqrt{24/6}=2\).)
- For binary outcomes \(1,0\), a model predicts probabilities \(0.75,0.25\). Calculate the mean log loss. (Answer: each loss is \(-\log(0.75)\approx0.2877\), so the mean is \(0.2877\).)
- Model J has AIC 120 and validation RMSE 5.0; model K has AIC 118 and validation RMSE 5.4 on the same validation cases. If prediction RMSE is the stated objective, which has the lower validation error, and what should be reported? (Answer: J has the lower validation RMSE; report that K has lower AIC but J performs better by RMSE on this validation set, with uncertainty considered.)
- Five-fold RMSE values for two models are J: \(2.0,2.1,2.0,2.1,2.0\), and K: \(1.6,2.5,1.7,2.4,2.0\). Find both means and describe the implication. (Answer: J mean \(2.04\); K mean \(2.04\). The means tie, but K varies more across folds, so the results do not show a mean RMSE advantage for either model.)