Comparing GLMs by Predictive Performance

Why Compare Predictions on Unseen Observations?

A GLM can describe its training data well and still predict poorly for new cases. When the purpose is prediction, the relevant question is how well each fitted model predicts outcomes it did not use for fitting. That requires evaluating candidate models on the same held-out observations and using a measure that reflects the prediction task.

A prediction is not automatically good because it is close to an observed value under every possible criterion. A model used to predict claim amounts may be judged by dollar errors, while a model used to estimate claim probabilities must be judged as a probability model. The evaluation must follow the intended use, not simply whichever statistic makes one candidate look better. This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: Let $Y$ be the response and $X$ the predictor information for a new case. For a specified loss function $L$, the predictive risk of a fitted model $f$ is $R(f)=E[L(Y,f(X))]$ for cases from the target population. A validation estimate averages the losses on held-out cases. Smaller values are preferred when the loss is defined so that lower means better.

For a GLM, the model commonly produces a predicted response mean on the response scale. A logistic GLM can produce a predicted event probability; a Poisson GLM can produce a predicted count mean. Use predictions in the form relevant to the task. For example, a logit-scale linear predictor is not itself a probability, and a log-link linear predictor is not itself a predicted count mean.

Designing a Fair Predictive Comparison

Fit each candidate using training data, then evaluate all candidates on the same held-out cases. Keep the held-out outcomes out of model fitting and any decisions that use those outcomes. If a candidate is altered after inspecting validation performance, that validation set has influenced model development; an additional untouched evaluation set is then needed for an unbiased final assessment.

The held-out cases should represent the population and time period for which predictions will be used. A random split may be appropriate when observations are exchangeable draws from a stable population. If the prediction task concerns future periods, a time-ordered holdout may better reflect that task. A low validation error is informative only to the extent that the validation design represents the intended deployment setting.

Define the prediction task.
Specify the response, prediction horizon or unit, target population, and the form of prediction that will be used.

Choose an appropriate loss.
Decide whether errors in predicted values, probabilities, or another output are consequential, and select a metric accordingly.

Score common held-out cases.
Apply each fitted model to exactly the same cases and calculate the same metric for each candidate.

Interpret the result in context.
Report the metric, its scale, and what kind of predictive error it rewards or penalizes. Do not treat it as proof of performance in every future setting.

The comparison is paired because each candidate is scored on each of the same cases. For case $i$, the difference in losses is $L(y_i,\hat{y}_{i,A})-L(y_i,\hat{y}_{i,B})$. Averaging these paired differences gives the difference between the candidates’ average validation losses. This prevents a difference in case mix from being mistaken for a difference in model performance.

Metrics for Predicted Values

For observed responses $y_i$ and predicted values $\hat{y}_i$, define the signed residual as $e_i=y_i-\hat{y}_i$. Its sign records whether the prediction is below or above the observation. Absolute error and squared error remove the sign, but penalize errors differently.

Definition: On $n$ held-out cases, the mean absolute error is $\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^n|y_i-\hat{y}_i|$. The mean squared error is $\operatorname{MSE}=\frac{1}{n}\sum_{i=1}^n(y_i-\hat{y}_i)^2$, and the root mean squared error is $\operatorname{RMSE}=\sqrt{\operatorname{MSE}}$. MAE is in the response’s units; RMSE is also in those units. MSE is in squared units.

MAE penalizes an error in direct proportion to its size. Squared error gives relatively more influence to large errors. Under squared-error loss, the conditional mean is the prediction that minimizes expected loss; under absolute-error loss, a conditional median minimizes expected loss. Thus the metric reflects what features of prediction matter. When using RMSE, take the square root after averaging the squared errors, not before.

Worked Example: Comparing Predicted Claim Amounts

In a hypothetical validation set, three claim amounts are $100, $120, and $150. Two GLMs predict amounts A: $110, $118, and $160; B: $105, $130, and $151. Calculate each model’s MAE and RMSE, then compare them. Use dollars and round final dollar results to the nearest cent.

Method and residuals. Calculate $e_i=y_i-\hat{y}_i$ for each case, then average absolute and squared errors. For A, the signed residuals are $100-110=-10$, $120-118=2$, and $150-160=-10$. For B, they are $100-105=-5$, $120-130=-10$, and $150-151=-1$.

For A, the absolute errors are $10,2,10$, so their sum is $10+2+10=22$. Therefore, $\operatorname{MAE}_A=22/3=\$7.33$. The squared errors are $100,4,100$, whose sum is $204$. Thus $\operatorname{MSE}_A=204/3=68$ dollars squared, and $\operatorname{RMSE}_A=\sqrt{68}=\$8.25$.

For B, the absolute errors are $5,10,1$, with sum $16$, giving $\operatorname{MAE}_B=16/3=\$5.33$. The squared errors are $25,100,1$, with sum $126$. Hence $\operatorname{MSE}_B=126/3=42$ dollars squared, and $\operatorname{RMSE}_B=\sqrt{42}=\$6.48$.

Conclusion and check. Model B has lower MAE and lower RMSE on these cases. The signs of the residuals are consistent with the listed predictions: for example, an overprediction of $110 when the observation is $100 gives $100-110=-10$, not $+10$. The signed residual is negative, even though its absolute error is positive.

Metrics for Predicted Probabilities

For a binary response $y_i\in\{0,1\}$, let $p_i$ be the predicted probability that $y_i=1$. Scoring probabilities directly retains information that would be lost by first converting each probability into a yes-or-no prediction using a threshold. Two common metrics are the Brier score and log loss; both are lower when predictions are better.

Definition: The Brier score is $\frac{1}{n}\sum_{i=1}^n(y_i-p_i)^2$. The log loss for binary outcomes is $-\frac{1}{n}\sum_{i=1}^n[y_i\ln(p_i)+(1-y_i)\ln(1-p_i)]$, with $0<p_i<1$. Log loss assigns a very large penalty to confidently assigning a small probability to the outcome that occurs.

For either metric, a probability must be paired with the correct observed outcome. When $y_i=1$, the log-loss contribution is $-\ln(p_i)$; when $y_i=0$, it is $-\ln(1-p_i)$. The probabilities must refer to the same event for every candidate. If a decision will eventually use a threshold, probability quality and the consequences of threshold-based decisions are distinct questions.

Worked Example: Comparing Claim-Incidence Probabilities

A hypothetical insurer evaluates two logistic GLMs on four held-out policies. The observed claim indicators are $1,0,1,0$. Model A predicts probabilities $0.8,0.2,0.6,0.4$; Model B predicts $0.7,0.1,0.9,0.1$. Calculate both models’ Brier score and log loss, rounding final scores to four decimal places.

Method. Use each observed indicator with its corresponding probability. For A, the squared probability errors are $(1-0.8)^2=0.04$, $(0-0.2)^2=0.04$, $(1-0.6)^2=0.16$, and $(0-0.4)^2=0.16$. Their sum is $0.40$, so the Brier score is $0.40/4=0.1000$.

For B, the squared errors are $(1-0.7)^2=0.09$, $(0-0.1)^2=0.01$, $(1-0.9)^2=0.01$, and $(0-0.1)^2=0.01$. Their sum is $0.12$, so the Brier score is $0.12/4=0.0300$.

For A, the four log-loss contributions are $-\ln(0.8)=0.2231$, $-\ln(1-0.2)=-\ln(0.8)=0.2231$, $-\ln(0.6)=0.5108$, and $-\ln(1-0.4)=-\ln(0.6)=0.5108$. The mean is $(0.2231+0.2231+0.5108+0.5108)/4=0.3670$, using unrounded logarithms for the final calculation.

For B, the contributions are $-\ln(0.7)=0.3567$, $-\ln(0.9)=0.1054$, $-\ln(0.9)=0.1054$, and $-\ln(0.9)=0.1054$. The mean is $0.6728/4=0.1682$. Model B has the lower score under both measures. The zero outcomes are scored using $1-p_i$, not $p_i$; this is a frequent source of incorrect log-loss calculations.

Scoring Count Predictions

For a count outcome modeled with a Poisson distribution, a GLM’s response-scale prediction is a positive mean $\hat{\mu}_i$. If the task is to predict the count mean and Poisson deviance is the chosen score, the held-out deviance for observed count $y_i$ is

D_i=2\left[y_i\ln\left(\frac{y_i}{\hat{\mu}_i}\right)-(y_i-\hat{\mu}_i)\right],

where the term $y_i\ln(y_i/\hat{\mu}_i)$ is defined as zero when $y_i=0$. Average the individual deviances to compare models on the same cases. This score is appropriate as a Poisson-based comparison when the counts and predictions have the stated interpretation; it should not be selected automatically for every prediction task.

Worked Example: Comparing Predicted Counts

In a hypothetical validation set, observed counts are $0,2,4$. Two Poisson GLMs predict means A: $1,2,3$ and B: $0.5,3,4$. Calculate mean Poisson deviance for each model and identify the lower-scoring model. Round final scores to four decimals.

Method. Apply the Poisson deviance formula separately to each case, using the zero-count convention for the first observation. For A at $y=0,\hat{\mu}=1$, $D=2[0-(0-1)]=2$. At $y=2,\hat{\mu}=2$, $D=2[2\ln(1)-0]=0$. At $y=4,\hat{\mu}=3$, $D=2[4\ln(4/3)-1]$. Since $\ln(4/3)=0.287682$, this is $2[1.150728-1]=0.301456$. The total is $2.301456$, giving mean deviance $2.301456/3=0.7672$.

For B at $y=0,\hat{\mu}=0.5$, $D=2[0-(0-0.5)]=1$. At $y=2,\hat{\mu}=3$, $D=2[2\ln(2/3)-(-1)]$. Since $\ln(2/3)=-0.405465$, this is $2[-0.810930+1]=0.378140$. At $y=4,\hat{\mu}=4$, $D=2[4\ln(1)-0]=0$. The total is $1.378140$, giving mean deviance $1.378140/3=0.4594$.

Conclusion and check. Model B has the lower mean Poisson deviance on this validation set. Its individual deviances are nonnegative, as expected for this formula; the exact-match predictions have deviance zero. The zero-count cases use the stated convention rather than attempting to evaluate $\ln(0)$.

Derivation: Why a Holdout Average Estimates Predictive Risk

Suppose the fitted model $f$ is fixed using training data, and the $n$ held-out pairs $(X_i,Y_i)$ are independent draws from the target population, independent of the training data. Assume the loss has a finite expectation. Define $L_i=L(Y_i,f(X_i))$. Conditional on the fitted model, each held-out pair has the same target-population distribution, so $E[L_i\mid f]=R(f)$. By linearity of expectation,

E\left[\frac{1}{n}\sum_{i=1}^n L_i\ \middle|\ f\right] =\frac{1}{n}\sum_{i=1}^n E[L_i\mid f] =\frac{1}{n}\sum_{i=1}^n R(f) =R(f).

Thus, under these assumptions, the average held-out loss is an unbiased estimate of the fixed model’s predictive risk. This does not guarantee that one finite validation sample will rank models correctly: its estimate has sampling variability. It also does not justify using a validation set that has affected fitting or repeated model selection as if it were untouched. If validation cases are not representative or independent in the relevant way, the calculation can still be performed, but this justification no longer applies as stated.

Common Traps and Interpretation

Scoring on training observations. Training performance measures fit to data used to estimate the model and can be optimistic as a measure of future prediction. Use held-out observations for predictive evaluation.
Comparing different cases. If candidate models are scored on different validation cases, a score difference may reflect case mix rather than model performance. Score every candidate on the same eligible cases.
Choosing a metric after seeing the winner. A metric should follow the stated prediction task. MAE and RMSE emphasize errors differently; probability scores assess probabilities rather than thresholded classifications.
Mixing prediction scales. Convert a GLM prediction to the response scale before scoring it as a response value or probability. A linear predictor on the link scale is not interchangeable with the response-scale prediction.
Overstating a small score difference. A lower validation score is evidence about performance on the evaluated cases, not a guarantee of future superiority. Consider the validation design, the metric’s meaning, and the uncertainty from finite data.

Key takeaway: Predictive comparison requires a defined task, a suitable loss, and common held-out observations. Calculate each candidate’s predictions on those same cases, interpret the metric in its own units or scale, and limit conclusions to what the validation design supports.

Check Your Understanding

For each problem, identify the comparison implied by the stated task and check the arithmetic before consulting the final answer.

A response has held-out values $50,70$. Model A predicts $55,68$, and Model B predicts $54,72$. Calculate the MAE for each and state which is lower. (Answer: Model A MAE $=\$3.50$; Model B MAE $=\$3.00$; B is lower.)
For outcomes $1,0,1$, a probability model predicts $0.8,0.3,0.6$. Calculate its Brier score to four decimals. (Answer: $(0.04+0.09+0.16)/3=0.0967$.)
Observed counts are $0,3$, and predicted Poisson means are $0.5,2$. Calculate the mean Poisson deviance to four decimals. (Answer: $[1+2\{3\ln(1.5)-1\}]/2=0.7164$.)
For outcomes $1,0$, a model predicts probabilities $0.6,0.2$. Calculate its mean log loss to four decimals. (Answer: $[-\ln(0.6)-\ln(0.8)]/2=0.3670$.)
Two GLMs are compared using MAE, but Model A is scored on 80 held-out cases and Model B on 74 different cases after six rows with missing predictors are excluded. Can the two reported MAEs alone establish which model predicts better on the same population cases? State the needed correction. (Answer: No; score both models on the same eligible held-out cases, or otherwise use a comparison design that controls for the different case sets.)

What You'll Learn