Diagnosing a Misleading GLM Model Summary

Why a GLM Summary Must Be Read as a Whole

A generalized linear model summary can place coefficients, fitted values, deviance, residual degrees of freedom, and a dispersion estimate in adjacent rows. Those quantities answer different questions. A coefficient is interpreted through the model's link function; a fitted mean must agree with both the coefficients and any offset; and a deviance statistic is meaningful only in relation to the response distribution, degrees of freedom, and dispersion assumptions.

The central audit is therefore a consistency check: What response is modeled, what link is used, what quantity does the coefficient multiply, and what assumptions justify the fit statistic? A coefficient is not automatically a change in probability or a rate ratio, and a deviance value by itself does not establish that a model fits well or poorly.

Definition: A generalized linear model (GLM) specifies a response distribution from the exponential family, a mean \(\mu_i=E[Y_i]\), and a link \(g\) such that \(g(\mu_i)=\eta_i\). The linear predictor is \(\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}+o_i\), where \(\mathbf{x}_i\) contains the predictors, \(\boldsymbol{\beta}\) contains the coefficients, and \(o_i\) is a known offset when one is required. The variance is commonly written \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\), where \(V\) is the variance function and \(\phi\) is a dispersion parameter.

A summary should first be matched to its response family and link. For logistic regression, the response is binomial and the logit link is \(g(p)=\log\{p/(1-p)\}\). For a Poisson count model with a log link, \(g(\mu)=\log(\mu)\); with an exposure offset, the model commonly represents a rate. These interpretations rely on the model actually using those links and response distributions. A coefficient from a different link or family cannot be assigned the interpretations below merely because its numerical value looks familiar.

Read Coefficients Through the Link

For a logistic model, a coefficient is a change in log odds, holding the other predictors fixed. Exponentiating a coefficient gives an odds ratio for a one-unit increase in that predictor, or for a specified category contrast. It does not generally give a probability ratio or a fixed percentage-point change in probability.

For a Poisson model with a log link, exponentiating a coefficient gives a multiplicative effect on the conditional mean count when the other predictors and offset are held fixed. If the model includes the log of exposure as an offset, the same coefficient is a multiplicative effect on the modeled rate. It is not an additive change in the rate.

\begin{aligned} \text{Logistic:}\quad &\log\left(\frac{p_i}{1-p_i}\right)=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}, &\exp(\beta_j)&=\text{conditional odds ratio for a one-unit increase in }x_j;\\ \text{Poisson rate:}\quad &\log(\mu_i)=\log(e_i)+\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}, &\exp(\beta_j)&=\text{conditional multiplicative rate effect}, \end{aligned}

Here \(e_i\) is a known exposure, and \(\log(e_i)\) is an offset with coefficient fixed at one. An offset is not an estimated predictor. If exposure is omitted, measured in the wrong units, or entered as an ordinary predictor, the interpretation of the fitted means may not be the intended rate comparison.

Worked Example: Auditing a Logistic Coefficient and Fitted Probabilities

A hypothetical insurer models whether a submitted appliance claim is approved. The fitted logistic model has an intercept of \(-1.3863\) and a coefficient of \(0.6931\) for an indicator \(x\) that equals 1 for claims with a specified documentation feature and 0 otherwise. The output reports fitted probabilities near 0.200 for \(x=0\) and 0.333 for \(x=1\). Check the coefficient interpretation and the fitted values.

Method. Treat the displayed coefficients as rounded versions of \(\log(1/4)\) and \(\log(2)\), respectively. The model is a logit model, so first calculate baseline log odds, then exponentiate the contrast coefficient to obtain an odds ratio.

For \(x=0\), the fitted log odds are \(-1.3863\), approximately \(\log(1/4)\). The baseline odds are therefore \(1/4=0.25\). Converting odds to probability gives

\hat p_0=\frac{0.25}{1+0.25}=\frac{0.25}{1.25}=0.2000.

For \(x=1\), the fitted log odds are \(-1.3863+0.6931=-0.6932\), approximately \(\log(1/2)\). The odds are \(1/2=0.5\), and

\hat p_1=\frac{0.5}{1+0.5}=\frac{0.5}{1.5}=0.3333.

The odds ratio is \(\exp(0.6931)\approx2\): at the same values of other predictors, the feature doubles the modeled odds of approval. The probability increases by about \(0.3333-0.2000=0.1333\), or 13.33 percentage points. Check: both computed probabilities agree with the reported fitted probabilities to three decimal places. The common mistake is to call the coefficient a 69.31% increase in probability; it is a change in log odds, and its exponent is an odds ratio, not a probability multiplier.

Check Fitted Means Against the Model

A fitted mean is obtained by applying the inverse link to the linear predictor. For a Poisson log-link model with an exposure offset, this gives \(\hat\mu_i=e_i\exp(\mathbf{x}_i^\mathsf{T}\hat{\boldsymbol{\beta}})\). Thus, a reported fitted count should scale with exposure when the covariates are held fixed. Comparing a fitted count from one exposure amount with a rate from another without adjusting the denominator is an invalid comparison.

Worked Example: Checking a Poisson Rate Output

A hypothetical roadside-assistance program models claim counts using a Poisson distribution and a log link. Its model is \(\log(\mu_i)=\log(e_i)+\beta_0+\beta_1x_i\), where \(e_i\) is policy-years of exposure and \(x_i=1\) marks a particular vehicle category. The estimated intercept is \(-3.6889\), approximately \(\log(0.025)\), and the category coefficient is \(0.4055\), approximately \(\log(1.5)\). For 400 policy-years in that category, check a reported fitted count of 15.

Method. Use the offset as the log exposure, add the intercept and category effect to form the linear predictor, then exponentiate to recover the expected count.

The baseline rate is \(\exp(-3.6889)\approx0.025\) claims per policy-year. The category multiplier is \(\exp(0.4055)\approx1.5\), so its modeled rate is \(0.025(1.5)=0.0375\) claims per policy-year. For 400 policy-years, the fitted mean count is

\hat\mu=400(0.0375)=15.

Equivalently, the linear predictor is \(\log(400)-3.6889+0.4055\), approximately \(\log(15)\), so exponentiating gives approximately 15. Check: the output's fitted count is consistent with both the rate and the exposure. The coefficient corresponds to a 50% increase in the modeled rate, not an increase of 0.4055 claims per policy-year and not a 40.55% increase.

What Deviance and Residual Degrees of Freedom Say

Deviance measures the discrepancy between a fitted model and a saturated model, which assigns a separate fitted mean to each observation or response group. It is defined as twice the difference between the maximized log likelihood under the saturated model and the maximized log likelihood under the fitted model. Lower deviance indicates a closer fit relative to the saturated model, but the raw value alone does not say whether the remaining discrepancy is unusually large.

D=2\{\ell_{\mathrm{sat}}-\ell_{\mathrm{fit}}\}.

Residual degrees of freedom describe how many independent pieces of information remain after fitting model parameters. In a standard model with \(n\) independent observations and \(p\) estimated regression coefficients, the residual degrees of freedom are typically \(n-p\). Grouped-response structure and model parameterization affect the precise count, so the reported degrees of freedom should be checked against the data structure and number of estimated parameters.

Under suitable conditions, residual deviance can be compared approximately with a chi-square distribution having the residual degrees of freedom. The approximation is not automatic: the model must be correctly specified, observations must meet the independence assumptions, and the sample or grouping must support the approximation. For Poisson and binomial models, the conventional dispersion is fixed at one. A large deviance relative to its degrees of freedom can signal lack of fit, but it is not by itself a complete diagnosis. For grouped binomial data, for example, the arrangement and number of groups affect how informative this comparison is.

Interpretation check: A residual deviance divided by residual degrees of freedom is a useful rough diagnostic, not a universal pass/fail rule. It should be considered alongside the response family, grouping, residual patterns, and any Pearson statistic or dispersion estimate. A null deviance is measured before including the predictors; it should not be compared with residual degrees of freedom as if both described the fitted model.

Separate Dispersion Evidence from Fit Evidence

The dispersion parameter \(\phi\) scales the variance in \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\). In standard Poisson and binomial likelihood models, \(\phi\) is fixed at one. In some GLMs, such as a Gamma model, dispersion is estimated. A Pearson-based estimate commonly used as a diagnostic is the Pearson chi-square statistic divided by residual degrees of freedom. It compares observed residual variation with the variation expected under the specified variance function.

If that ratio is substantially above one in a Poisson or binomial setting, the data may show more variation than the assumed model allows. This is evidence to investigate, not proof of a particular cause. Dependence, omitted predictors, incorrect mean structure, and other misspecification can all matter. A ratio near one does not prove that every aspect of the model is correct, either. If an estimated dispersion is used to adjust standard errors, the resulting inference should be interpreted under the assumptions of the method used; it does not change the original mean-model coefficients into rate or odds ratios of a different kind.

Worked Example: Resolving Conflicting Fit and Dispersion Clues

A hypothetical Poisson regression summary for 30 independent risk groups reports residual deviance 36.0 on 24 residual degrees of freedom and Pearson chi-square 52.8. The model estimates six coefficients, including an intercept. Decide what these figures support and what they do not establish.

Method. Check the residual degrees of freedom, calculate both diagnostic ratios, and then interpret them under the Poisson assumption of dispersion one.

First, \(30-6=24\), which agrees with the reported residual degrees of freedom under the stated setup. The deviance ratio is \(36.0/24=1.50\). The Pearson ratio is \(52.8/24=2.20\). Both are above one, and the Pearson statistic indicates more residual variation relative to the Poisson variance than the deviance ratio alone suggests.

The appropriate conclusion is that the summary raises a concern about fit and possible overdispersion; the evidence should be followed up with checks of the mean structure, dependence, and residual behavior. These two ratios are not interchangeable estimates that must agree exactly, and neither alone identifies the cause. Check: the degrees of freedom match \(n-p\), while the ratios are \(1.50\) and \(2.20\). The common mistake is to declare the model adequate because the deviance ratio is “not far from one,” while ignoring the substantially larger Pearson ratio and the assumptions behind either diagnostic.

A Short Derivation: Why a Logistic Coefficient Exponentiates to an Odds Ratio

Suppose two observations have the same values of all predictors except \(x_j\), which differs by one unit. In a logistic model, the log odds for the second observation exceed those for the first by \(\beta_j\). Subtracting the two linear predictors and exponentiating gives the ratio of their odds:

\begin{aligned} \log\left(\frac{\text{odds}_2}{\text{odds}_1}\right) &=\log(\text{odds}_2)-\log(\text{odds}_1)=\beta_j,\\ \frac{\text{odds}_2}{\text{odds}_1}&=\exp(\beta_j). \end{aligned}

This establishes the odds-ratio interpretation while holding other predictors fixed. It does not establish a probability ratio: converting odds to probabilities uses \(p=\text{odds}/(1+\text{odds})\), a nonlinear transformation. Consequently, the probability difference associated with the same coefficient depends on the starting odds.

A Practical Audit Sequence

When a model summary seems to invite a quick conclusion, use the following sequence before interpreting it. Each step checks a different potential source of error.

Identify the response and link.
Confirm whether the model is logistic, Poisson, Gamma, or another GLM, and name the link. Do not transfer odds-ratio or rate-ratio language across model types.

Read the predictor coding and offset.
Determine the reference category, unit change, and exposure denominator. A coefficient's meaning depends on those definitions.

Reconstruct a fitted mean.
Combine the intercept, relevant predictor terms, and offset, then apply the inverse link. Check that the result agrees with a reported fitted probability, count, or rate.

Audit fit and dispersion separately.
Check residual degrees of freedom, interpret deviance in its context, and compare available Pearson or dispersion evidence with the model's variance assumptions.

Common Exam Traps

Calling a logistic coefficient a probability change. A logistic coefficient changes log odds; exponentiating it gives an odds ratio. Calculate probabilities from the full linear predictor.
Calling every exponentiated coefficient a rate ratio. That interpretation requires an appropriate log-link rate model, generally with a correctly specified exposure offset.
Ignoring the denominator. A fitted count depends on exposure. Compare rates only after accounting for the relevant exposure units.
Treating deviance as a standalone verdict. Interpret it with its degrees of freedom, data structure, distributional assumptions, and other diagnostics.
Equating evidence of overdispersion with a known cause. A large Pearson ratio flags a mismatch with the assumed variance; it does not by itself distinguish dependence from a deficient mean model or another source of variation.

Key takeaway: A defensible GLM interpretation connects the response family and link to the coefficient scale, verifies fitted means from the linear predictor and offset, and treats deviance and dispersion as assumption-dependent diagnostic evidence rather than isolated answers.

Check Your Understanding

For each original scenario, state the interpretation or calculation requested and check it against the final answer.

A logistic model for a hypothetical equipment inspection has intercept \(\log(1/3)\) and a binary indicator coefficient \(\log(3)\). Find the fitted probabilities for indicator values 0 and 1, and state the odds ratio. (Answer: \(p_0=0.2500\), \(p_1=0.5000\), odds ratio \(=3\).)
A Poisson log-link model uses exposure as an offset. Its baseline rate is 0.030 claims per unit of exposure, and a category coefficient is \(-0.2231\), approximately \(\log(0.8)\). For 500 exposure units in that category, find the fitted count. (Answer: category rate \(=0.024\), fitted count \(=12\).)
A Poisson GLM has residual deviance 18 on 20 residual degrees of freedom and Pearson chi-square 24. Calculate both ratios and give a cautious interpretation. (Answer: deviance ratio \(=0.90\), Pearson ratio \(=1.20\); these values do not alone establish adequate fit, but the Pearson ratio gives no strong indication of substantial overdispersion.)
A logistic summary reports an intercept of \(-2.1972\) and a coefficient of \(0.6931\) for a one-unit predictor contrast. Find the baseline and contrasted probabilities, and explain why the contrast does not double probability. (Answer: baseline \(p\approx0.1000\), contrasted \(p\approx0.1818\), odds ratio \(=2\); the odds double, not the probability.)
A grouped binomial model reports residual deviance 48 on 20 degrees of freedom and Pearson chi-square 25. Calculate both ratios and identify the main caution. (Answer: deviance ratio \(=2.40\), Pearson ratio \(=1.25\); the deviance raises a fit concern, while the differing diagnostics should be investigated in light of the grouping and model assumptions rather than treated as a definitive verdict.)

What You'll Learn