A Worked GLM Diagnostic and Model-Comparison Case

Putting GLM Diagnostics Together

A generalized linear model (GLM) analysis is not complete when a coefficient table or model-comparison p-value has been produced. For count data, a defensible assessment also asks whether individual observations have unusual residuals, whether the overall variation is compatible with the assumed Poisson model, and whether a small number of observations unduly affect the fit. These checks address different questions: a model can have a reasonable overall dispersion estimate and still contain an influential observation.

This case-based tutorial is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. It focuses on combining diagnostic and comparison results, rather than treating any one statistic as a universal pass-or-fail rule.

Model and assumptions: For independent count observations \(Y_i\), a Poisson GLM specifies \(Y_i\mid \mathbf{x}_i\sim\operatorname{Poisson}(\mu_i)\), so \(E[Y_i\mid\mathbf{x}_i]=\operatorname{Var}(Y_i\mid\mathbf{x}_i)=\mu_i\). With a log link and exposure \(e_i\), a common model is \(\log(\mu_i)=\log(e_i)+\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}\). The offset \(\log(e_i)\) has coefficient fixed at 1. The usual Poisson likelihood calculations require a suitable mean specification and conditionally independent observations; the ordinary Poisson variance assumption is also needed for standard model-based inference.

Residuals: Individual Observations

For an observed count \(y_i\) and fitted mean \(\hat{\mu}_i\), the Pearson residual measures the difference from the fitted mean in units of the model-based standard deviation. The deviance residual expresses the observation’s contribution to the likelihood-based deviance, with a sign indicating whether the observation is above or below its fitted mean.

r_{Pi}=\frac{y_i-\hat{\mu}_i}{\sqrt{\hat{\mu}_i}}, \qquad r_{Di}=\operatorname{sign}(y_i-\hat{\mu}_i) \sqrt{2\left[y_i\log\left(\frac{y_i}{\hat{\mu}_i}\right)-(y_i-\hat{\mu}_i)\right]}.

When \(y_i=0\), the term \(y_i\log(y_i/\hat{\mu}_i)\) is defined by its limiting value, 0. A large absolute residual is a prompt to investigate the observation and the model fit; it is not, by itself, proof that the observation is erroneous or that a particular model change is required. Patterns across observations can be more informative than one isolated residual.

Worked Example: Interpreting Two Claim-Count Residuals

A hypothetical insurer fits a Poisson model to independent policy groups. Consider two groups with fitted means \(\hat{\mu}_1=5\) and \(\hat{\mu}_2=3\), and observed counts \(y_1=8\) and \(y_2=0\). Calculate both residual types for each group. Use natural logarithms and round residuals to four decimal places.

Method. Substitute each observed count and fitted mean into the residual formulas, preserving the sign of \(y_i-\hat{\mu}_i\).

For group 1, \(y_1-\hat{\mu}_1=8-5=3\). Thus \(r_{P1}=3/\sqrt{5}=3/2.2361=1.3416\). For the deviance residual, \(\log(8/5)=\log(1.6)=0.4700\), so \(8\log(1.6)=3.7600\) to four decimals. The quantity inside the outer factor of 2 is \(3.7600-3=0.7600\). Therefore \(r_{D1}=+\sqrt{2(0.7600)}=\sqrt{1.5200}=1.2328\).

For group 2, \(y_2-\hat{\mu}_2=0-3=-3\). Hence \(r_{P2}=-3/\sqrt{3}=-3/1.7321=-1.7321\). Since the observed count is zero, the logarithmic term is zero. The deviance residual is \(r_{D2}=-\sqrt{2[0-(-3)]}=-\sqrt{6}=-2.4495\).

The signs agree with the direction of the discrepancies: the first group is above its fitted mean and the second is below. The second group has the larger absolute residuals. Common mistake avoided: substituting \(y=0\) into \(y\log(y/\hat{\mu})\) and trying to calculate \(\log(0)\); the whole term is taken as its limiting value, 0.

Overall Dispersion and Model Comparison

For a fitted Poisson GLM with \(n\) observations and \(p\) estimated regression coefficients, the residual degrees of freedom are usually \(n-p\). The Pearson goodness-of-fit statistic is the sum of squared Pearson residuals. Its ratio to residual degrees of freedom is a useful dispersion diagnostic: a value substantially above 1 can signal more variation than the Poisson model allows, while a value substantially below 1 can signal less. Neither the ratio nor a threshold alone identifies the cause.

X^2_P=\sum_{i=1}^{n}\frac{(y_i-\hat{\mu}_i)^2}{\hat{\mu}_i} =\sum_{i=1}^{n}r_{Pi}^2, \qquad \hat{\phi}_P=\frac{X^2_P}{n-p}.

The Poisson model fixes the conditional variance equal to the conditional mean, so its theoretical dispersion parameter is 1. The estimate \(\hat{\phi}_P\) is a diagnostic, not a repair to the Poisson likelihood. If there is meaningful overdispersion, standard errors and ordinary likelihood-ratio test calibration may be unreliable. Possible explanations include an incorrect mean structure, dependence, omitted predictors, or genuine extra-Poisson variability. These explanations have different remedies.

For nested models, the likelihood-ratio statistic is the difference between the reduced model’s deviance and the full model’s deviance. If the models are correctly nested, the regularity conditions hold, and the likelihood model is adequate, this difference is approximately chi-square with degrees of freedom equal to the number of additional coefficients in the full model.

G^2=D_{\text{reduced}}-D_{\text{full}} \ \dot{\sim}\ \chi^2_{p_{\text{full}}-p_{\text{reduced}}}.

The reason for the deviance difference is that, within the same response family, each model’s deviance is twice the log-likelihood difference between that model and the saturated model. Subtracting the full model’s deviance from the reduced model’s deviance cancels the saturated-model term and leaves twice the log-likelihood gain from adding the coefficients. Under the null hypothesis that the added coefficients are zero, the likelihood-ratio statistic has the stated approximate chi-square distribution.

Worked Example: Comparing Two Nested Poisson Models

A hypothetical portfolio study models claim counts for 25 independent risk cells using the same exposure offset. The reduced model has 3 coefficients and deviance 28.8. The full model adds two prespecified interaction terms, has 5 coefficients, and deviance 22.4. Its Pearson statistic is 19.2. Assume the Poisson likelihood and regularity conditions are suitable unless the diagnostics indicate otherwise. Compare the models at the 5% level and assess the dispersion evidence.

Method. First check the Pearson dispersion estimate. Then calculate the deviance difference and use the number of added coefficients for the chi-square reference distribution.

The full model has residual degrees of freedom \(25-5=20\). Its Pearson dispersion estimate is \(\hat{\phi}_P=19.2/20=0.96\). This does not indicate evidence of overdispersion from this statistic.

The likelihood-ratio statistic is \(G^2=28.8-22.4=6.4\). The degrees of freedom are \(5-3=2\), because the full model adds two coefficients. For a chi-square random variable with 2 degrees of freedom, the upper-tail probability is \(\exp(-G^2/2)\). Thus \(p=\exp(-6.4/2)=\exp(-3.2)=0.0408\), to four decimal places.

Since \(0.0408<0.05\), reject the null hypothesis that both added interaction coefficients are zero, conditional on the stated assumptions. The result is modest evidence in favor of the larger model, not proof that the interactions are important in every application. Common mistake avoided: using 5 degrees of freedom for the comparison. The test degrees of freedom are the difference in parameter counts, not the full model’s residual degrees of freedom.

Influence: Combining Residual Size and Leverage

A large residual identifies an observation that is poorly predicted; leverage describes how unusual its predictor values are relative to the design. An observation can have a large residual but little effect on the fitted coefficients if its leverage is low. Conversely, an unusual predictor pattern can have substantial influence even when its residual is not extreme. For this reason, residual size and leverage should be considered together.

A commonly used approximate GLM Cook’s distance is based on the Pearson residual and diagonal leverage \(h_i\):

D_i\approx \frac{r_{Pi}^{\,2}h_i}{p(1-h_i)^2}.

This is a screening approximation, not an exact universal influence measure. Its value depends on the GLM approximation and model context. A flag such as \(D_i>4/n\) can help prioritize review, but it is only a rule of thumb. Refitting after deleting an observation and examining changes in estimates, fitted means, and conclusions is a more direct sensitivity check.

Worked Example: Screening a High-Count Cell for Influence

A hypothetical insurer fits a Poisson GLM with \(p=4\) coefficients to 40 independent policy cells. The fitted mean for one cell is \(\hat{\mu}_i=6\), while the observed count is \(y_i=12\); its leverage is \(h_i=0.25\). The full-model Pearson statistic is 37.8 on 36 residual degrees of freedom. Calculate the cell’s Pearson residual and approximate Cook’s distance, compare the latter with \(4/n\), and state what to do next.

Method. Calculate the cell’s residual, use the stated approximate GLM Cook’s distance formula, and interpret both in the context of the overall dispersion estimate.

The Pearson residual is \(r_{Pi}=(12-6)/\sqrt{6}=6/2.4495=2.4495\). Its square is \(r_{Pi}^2=6\). The Cook’s distance approximation is \[ D_i\approx\frac{6(0.25)}{4(1-0.25)^2} =\frac{1.5}{4(0.75)^2} =\frac{1.5}{4(0.5625)} =\frac{1.5}{2.25} =0.6667. \] The screening reference is \(4/n=4/40=0.1000\), so \(0.6667>0.1000\). The observation merits investigation and a case-deletion refit.

The overall dispersion estimate is \(37.8/36=1.05\), which does not suggest substantial overdispersion in this summary. That does not remove the cell-level concern: overall dispersion and influence answer different questions. The values do not establish that the cell is erroneous or that deleting it is justified. Common mistake avoided: declaring the model adequate solely because the overall Pearson ratio is near 1. A near-1 ratio can coexist with a potentially influential observation.

A Defensible Diagnostic Sequence

A practical case analysis keeps the roles of the statistics separate. First examine residuals for unusual observations and systematic patterns. Next assess overall dispersion. Then investigate high-leverage or influential observations, including sensitivity to case deletion where appropriate. Finally, compare nested models only with a reference distribution justified by the fitted likelihood and diagnostics. This ordering helps avoid interpreting a small p-value from a model whose variance assumptions are visibly inadequate.

Check individual residuals.
Identify large Pearson or deviance residuals and look for patterns by fitted mean, predictor, exposure, or relevant grouping. A pattern may indicate a mean-structure issue rather than a single outlier.

Assess dispersion.
Compare the Pearson statistic with residual degrees of freedom. Treat a marked departure from 1 as evidence to investigate, not as a diagnosis of its cause.

Review influence.
Combine residual magnitude with leverage and an influence screen. Where a case is flagged, refit without it and report whether substantive conclusions change.

Compare nested models cautiously.
Use the deviance difference and the difference in parameter counts. If overdispersion or dependence makes the Poisson likelihood unsuitable, do not present the ordinary likelihood-ratio p-value as reliable without qualification.

A common exam trap is to treat a low deviance difference p-value as automatically decisive. That conclusion depends on the model assumptions, nesting, and reference distribution. If overdispersion is appreciable, simply dividing a deviance difference by an estimated dispersion factor does not automatically produce an ordinary likelihood-ratio test: quasi-likelihood models do not supply the same full likelihood. The comparison method must match the model actually fitted.

Key takeaway: Residuals diagnose observation-level discrepancies, the Pearson ratio screens overall dispersion, leverage and influence measures identify sensitivity to particular cases, and a nested-model deviance test assesses added terms under likelihood assumptions. Interpret the evidence jointly, and distinguish a diagnostic warning from a confirmed model defect.

Check Your Understanding

For each problem, identify the relevant diagnostic or comparison and avoid treating a screening statistic as a final verdict.

A Poisson GLM gives \(y=4\) and \(\hat{\mu}=2\) for one cell. Calculate its Pearson and signed deviance residuals. Round to four decimals. (Answer: \(r_P=1.4142\), \(r_D=1.2431\).)
A fitted Poisson model has Pearson statistic 27 on 24 residual degrees of freedom. Calculate the Pearson dispersion estimate and state whether it is above 1. (Answer: \(27/24=1.125\), above 1.)
A reduced model has deviance 42.7; a nested full model has deviance 33.49 and adds 3 coefficients. Calculate the likelihood-ratio statistic and approximate upper-tail p-value. (Answer: \(G^2=9.21\), 3 degrees of freedom, \(p\approx0.0266\).)
For a GLM with \(p=4\), an observation has \(r_P=2\) and \(h=0.20\). Calculate the approximate Cook’s distance and compare it with \(4/n\) when \(n=40\). (Answer: \(D_i\approx0.3125\), compared with \(4/40=0.1000\); it is flagged for review.)
A Poisson model has Pearson statistic 60 on 30 residual degrees of freedom. A nested comparison gives a deviance difference of 8 on 2 degrees of freedom. Explain why the usual likelihood-ratio conclusion needs qualification. (Answer: \(\hat{\phi}_P=60/30=2\) indicates substantial overdispersion; the ordinary Poisson likelihood-ratio reference may be unreliable, so investigate the model and use inference appropriate to the variance structure.)

What You'll Learn