Start With the Decision, Not the Statistic
Two generalized linear models can be compared for different reasons. An analyst may want to describe an adjusted association, assess whether a model represents the observed responses adequately, or choose the model that predicts future outcomes most usefully. These are distinct objectives. A statistic that is informative for one may not answer another.
For example, a model with a small in-sample residual measure is not automatically the best model for future predictions. Nor does a precise coefficient estimate, by itself, show that a model predicts well or that a predictor causes the response. Before comparing candidates, state what decision the comparison is meant to support and what kind of error matters for that decision.
A GLM specifies a conditional response distribution and a relationship between its mean and predictors. In the standard setup, responses \(Y_i\) are independent conditional on their predictors, each response follows a chosen distribution from the model family, and the mean \(\mu_i=E[Y_i\mid x_i]\) is linked to a linear predictor:
The assumed distribution determines the variance structure as well as the form of the response. For example, a Poisson model for counts assumes \(E[Y_i\mid x_i]=\operatorname{Var}(Y_i\mid x_i)=\mu_i\); a Bernoulli model describes a binary response with a conditional probability \(\mu_i\). These assumptions are part of what is being assessed when considering model fit. They do not establish that a model is suitable for every objective.
Match Evidence to the Objective
| Objective | Main question | Evidence to consider |
|---|---|---|
| Explanation | What adjusted relationships are estimated, and how uncertain are they? | Coefficient estimates, standard errors or confidence intervals, coding and reference levels, and whether the specification answers the stated question |
| Fit | Does the assumed response distribution and mean structure represent the observed data reasonably? | Residual patterns, distribution-appropriate residual or deviance summaries, dispersion checks, and possible lack of fit |
| Prediction | How useful are predictions for new cases in the intended population? | Performance on held-out or otherwise appropriate validation data, measured with a loss aligned to the response and decision |
For explanation, the coefficient scale matters. A coefficient in a logistic GLM is a change in log odds per unit change in a predictor, holding other included predictors fixed; exponentiating it gives an odds ratio. A coefficient is not automatically a causal effect. Interpretation also depends on the coding, the included covariates, the population represented, and the assumptions behind the model.
For fit, examine whether the chosen family and mean structure are credible in context, not merely whether one summary looks favorable. Earlier material on GLM residual diagnostics and deviance-based goodness-of-fit checks provides tools for this assessment. A lack of visible residual structure is useful evidence, but a small data set may have little power to reveal a defect.
For prediction, the loss function expresses what counts as an error. Squared error penalizes large numerical errors more heavily than absolute error. For a binary outcome, probability scores assess the quality of predicted probabilities; classification accuracy instead requires a threshold and treats some errors as equivalent even when their consequences differ. The evaluation data and metric should reflect the actual prediction task.
Worked Example: Comparing Logistic Models for Explanation
Worked Example: Comparing Logistic Models for Explanation
A hypothetical analyst studies whether a policyholder renews a contract. A logistic GLM includes age band, contract type, and a service score. In the fitted model, the service-score coefficient is \(-0.20\), with standard error \(0.05\). The analyst’s question is whether the score has an adjusted association with renewal, not which model predicts renewals most accurately. Find the odds ratio and an approximate 95% confidence interval for it, then state what those results can support.
Method. For a logistic GLM, a one-unit increase in a predictor changes the fitted log odds by its coefficient when the other included predictors are held fixed. Exponentiating the coefficient gives the corresponding odds ratio. Use the specified approximate normal interval on the coefficient scale, \(\hat\beta\pm1.96\,\operatorname{SE}(\hat\beta)\), and exponentiate both endpoints.
Step 1: Calculate the odds ratio. The estimated coefficient is \(-0.20\), so
Step 2: Calculate the coefficient interval. The margin is \(1.96(0.05)=0.098\). Thus the endpoints are \(-0.20-0.098=-0.298\) and \(-0.20+0.098=-0.102\).
Step 3: Exponentiate the endpoints. The odds-ratio interval is \((\exp(-0.298),\exp(-0.102))\), approximately \((0.7423,0.9030)\). To three decimals, it is \((0.742,0.903)\).
Conclusion. Conditional on the other included predictors, a one-unit higher service score is associated with estimated renewal odds multiplied by about \(0.8187\), or lower odds. The approximate interval is below 1. This is evidence about an adjusted association under the model; it is not proof that changing the score causes renewal odds to change. This calculation answers an explanatory question, but it does not establish that this model has the best predictive performance.
Common mistake avoided: Reporting \(-0.20\) as a 20% reduction in renewal probability. It is a change in log odds; exponentiation gives an odds ratio, and neither quantity is a direct percentage-point change in probability.
Fit Evidence Depends on the Response Distribution
A fit assessment asks whether the model’s representation of the observed responses is plausible. It is not simply a contest to find the smallest raw residual. Residuals must be interpreted in light of the response family, its variance, the fitted mean, and the design. For example, a raw difference of one is not equally surprising when a fitted count mean is 1 and when it is 100.
For a Poisson model, a useful diagnostic summary is the Pearson statistic. Under the Poisson assumptions, each response has conditional variance \(\mu_i\), so the squared residual can be scaled by that variance:
This scaling explains why the statistic uses \(\hat\mu_i\) in the denominator: it compares squared deviations with the variance expected under the fitted Poisson model. It is a diagnostic summary, not a guarantee that the model is correct. Its interpretation can be limited with small samples, sparse counts, dependence, or an incorrect mean or variance structure.
Worked Example: Assessing Fit for a Poisson Count Model
A hypothetical screening program records independent counts for four equal-exposure groups. A Poisson GLM with a log link has fitted means \(0.5, 1.5, 2.5,\) and \(2.5\); the observed counts are \(0,1,2,\) and \(4\). The model estimates two parameters. Compute the Pearson statistic and its value per residual degree of freedom as a preliminary fit diagnostic.
Assumptions and method. Treat the group counts as conditionally independent Poisson responses, so each fitted variance equals its fitted mean. The approximate residual degrees of freedom are \(n-p=4-2=2\). Calculate each Pearson contribution and sum them.
For the first group, \((0-0.5)^2/0.5=0.25/0.5=0.5\). For the second, \((1-1.5)^2/1.5=0.25/1.5=0.1667\) to four decimals. For the third, \((2-2.5)^2/2.5=0.25/2.5=0.1\). For the fourth, \((4-2.5)^2/2.5=2.25/2.5=0.9\).
Adding the contributions gives \(X^2_{\mathrm{Pearson}}=0.5+0.1667+0.1+0.9=1.6667\) to four decimals. Dividing by the approximate two residual degrees of freedom gives \(1.6667/2=0.8334\) to four decimals using the rounded statistic; using unrounded contributions gives \(1.666\ldots/2=0.8333\). The ratio is therefore approximately \(0.8333\).
Conclusion. The statistic gives no indication here of a large overall Pearson discrepancy relative to the approximate degrees of freedom. It is not evidence that every model assumption holds: there are only four groups, expected counts are small, and a summary can conceal a pattern. Residual plots and subject-matter assessment remain important.
Common mistake avoided: Using \((y_i-\hat\mu_i)^2\) without scaling by \(\hat\mu_i\). That would treat count deviations as having equal variance, contrary to the Poisson assumption that the variance equals the mean.
Prediction Requires an Explicit Loss
A predictive objective concerns performance for cases beyond the data used to estimate the model. The validation design should represent the intended use: for example, a random split may be unsuitable if future observations differ systematically from the past or if records from the same policyholder appear in both sets. Compare candidate predictions on the same relevant validation cases, using the same response definition and loss.
For binary outcomes, the Brier score is the mean squared difference between the observed outcome \(y_i\), coded 0 or 1, and the predicted probability \(\hat p_i\). Smaller values indicate better average squared probability error on the evaluated cases. Its rationale follows directly from conditional expectation. If the true event probability for a case with predictors \(x\) is \(q\), then for a proposed probability \(p\):
The final term is minimized at \(p=q\), so expected squared error favors reporting the true conditional probability. This justification does not imply that a low score on a small validation sample establishes good performance for all future populations.
Worked Example: Comparing Probability Predictions
A hypothetical insurer evaluates predicted claim probabilities for five independent validation cases. The predicted probabilities are \(0.8, 0.4, 0.6, 0.3,\) and \(0.2\); the observed claim indicators are \(1,0,1,0,\) and \(0\), respectively. Calculate the Brier score and explain which objective it addresses.
Method. For each case, square the difference between the predicted probability and the observed 0-or-1 outcome, then average the five squared errors.
The squared errors, in order, are \((0.8-1)^2=0.04\), \((0.4-0)^2=0.16\), \((0.6-1)^2=0.16\), \((0.3-0)^2=0.09\), and \((0.2-0)^2=0.04\). Their sum is \(0.04+0.16+0.16+0.09+0.04=0.49\). Dividing by five gives a Brier score of \(0.49/5=0.0980\).
Conclusion. The score is \(0.0980\), and it evaluates predictive probability accuracy on these validation cases. A smaller Brier score would be preferable when comparing models on the same suitable cases. With only five cases, the estimate is highly uncertain; it does not establish future performance on its own.
Common mistake avoided: Calculating errors from a hard classification such as “claim” or “no claim” after applying a threshold. The Brier score uses the predicted probabilities themselves.
Why Objectives Can Point in Different Directions
A model comparison can produce different conclusions depending on the question. A more complex model might give a coefficient estimate that is more relevant to an explanatory question, while its predictions are not better on validation data. Alternatively, predictions may improve even though individual coefficients become less stable or less straightforward to explain. Neither outcome is contradictory: the objectives differ.
Choose evidence deliberately:
- For explanation, name the association or contrast of interest, confirm the coefficient scale and reference categories, and report uncertainty. Consider whether adjustment choices match the question. Do not describe an observational association as causal without a defensible causal design and assumptions.
- For fit, examine response-family assumptions, residual behavior, dispersion where relevant, and whether the fitted mean structure is adequate. A single numerical summary cannot certify all of these.
- For prediction, define the target cases, validation scheme, and loss before selecting a model. For continuous outcomes, RMSE or MAE may be relevant; for binary probabilities, Brier score or log loss may be appropriate. The choice depends on how errors will be used.
A frequent exam trap is to treat “best model” as a complete answer. It is incomplete until “best for what?” is stated. A claim that a model is better must identify the objective, the evidence supporting that objective, and any important limitations. The appropriate conclusion is often conditional: one model is preferable for a specified explanatory task, while another performs better under a stated predictive loss.
Check Your Understanding
For each problem, identify the objective and use evidence appropriate to that objective.
- A hypothetical logistic GLM estimates a coefficient of \(0.12\) with standard error \(0.04\) for a one-unit increase in a risk score. Use \(1.96\) standard errors to find the coefficient interval and exponentiate the estimate and endpoints. (Answer: coefficient interval \((0.0416,0.1984)\); odds ratio \(1.1275\), with interval approximately \((1.0425,1.2194)\); explanatory objective.)
- A Poisson model has observed counts \(1,2,3\) and fitted means \(1,2,2\). It estimates one parameter. Calculate the Pearson statistic and divide by approximate residual degrees of freedom. (Answer: Pearson statistic \(0.5000\); residual degrees of freedom \(2\); ratio \(0.2500\), a preliminary fit diagnostic.)
- On four binary validation cases, predicted probabilities are \(0.7,0.2,0.6,0.1\), and outcomes are \(1,0,0,0\). Calculate the Brier score. (Answer: squared errors \(0.09,0.04,0.36,0.01\); Brier score \(0.1250\), for predictive probability performance.)
- A model predicts continuous outcomes \(12,13,20\), while the observed outcomes are \(10,14,18\). Calculate RMSE and MAE, and identify which penalizes the largest error more strongly. (Answer: squared errors \(4,1,4\), so RMSE \(=\sqrt{3}\approx1.732\); absolute errors \(2,1,2\), so MAE \(=5/3\approx1.667\); RMSE penalizes larger errors more strongly.)
- An analyst wants to describe the adjusted relationship between a service score and renewal, but recommends a model solely because it has higher classification accuracy after using a 0.5 threshold. State why this evidence is mismatched and name more relevant explanatory evidence. (Answer: threshold-based accuracy assesses classification performance, not the adjusted relationship; examine the score coefficient’s scale and interpretation, its uncertainty, and whether the model specification answers the explanatory question.)