A GLM Model-Selection Workshop

From Candidate Models to a Defensible Choice

Model selection is not just the act of choosing the row with the smallest score. A defensible analysis first defines a relevant set of candidate models, checks that the fitted likelihoods are comparable, and then considers what each comparison can and cannot establish. In a clinical regression problem, those steps matter because a change in a transformation or interaction can alter both interpretation and prediction.

Consider a study of \(n=240\) patients in which the response is whether a serious event occurs within 30 days. Let \(Y_i\) be the binary outcome, \(x_i>0\) the treatment dose measured in units of 10 mg, and \(z_i\) a standardized baseline severity score. For this workshop, use logistic regression, with \(Y_i\) conditionally Bernoulli and \(\operatorname{logit}\{P(Y_i=1\mid x_i,z_i)\}=\eta_i\). The question is whether the dose relationship is adequately represented as linear on the logit scale, whether a log-dose transformation helps, and whether dose modifies the severity association.

Construct Candidates Before Ranking Them

A candidate set should correspond to plausible scientific descriptions, not to every formula that software can generate. Define \(g(x)=x\) for a linear dose effect and \(g(x)=\log(x)\) for a log-dose effect. A compact set of four models is:

The interaction specifications include both corresponding main effects. This respects the usual hierarchy principle: an interaction is not included while omitting the terms that define its component variables. The candidates also use the same patients and response. That is important: a criterion difference should reflect the models, not a changed analysis sample.

Check the Likelihood Comparison

Before comparing likelihoods, confirm that each fit uses the same response definition, observations, likelihood convention, and treatment of weights or offsets. Check that the optimizer converged, the design matrix has full column rank, and the fitted model has no separation or other warning that makes its maximum-likelihood estimates unreliable. A numerical log-likelihood is not reassuring merely because software printed it.

For nested models fitted by maximum likelihood to the same data, the likelihood-ratio statistic is twice the difference in maximized log-likelihoods. The reason is direct: if the unrestricted model has log-likelihood \(\ell_U\) and the restricted model has \(\ell_R\), the likelihood-ratio statistic is \(-2\log\{L_R/L_U\}=-2(\ell_R-\ell_U)=2(\ell_U-\ell_R)\). Since the unrestricted parameter space contains the restricted one, its maximum cannot be smaller.

The algebra establishes the statistic and its nonnegativity; it does not by itself establish the chi-squared approximation. The latter is an asymptotic result and can be unreliable with sparse data, boundary parameters, separation, or other violations of regularity. Also, a likelihood-ratio test is not a general test for comparing any two candidate models: the models must be nested for this standard parameter-difference reference distribution.

Rank Candidates with Information Criteria

AIC and BIC combine fit with a penalty for the number of estimated parameters. With maximized log-likelihood \(\ell\), parameter count \(k\), and sample size \(n\), use \(AIC=-2\ell+2k\) and \(BIC=-2\ell+k\log(n)\). Smaller values rank better under the corresponding criterion. They are comparative scores for a specified candidate set, not tests that a model is true.

For models fitted to the same observations under the same likelihood convention, a common additive constant in the reported likelihood cancels from differences in either criterion. More explicitly, if every reported log-likelihood is shifted by the same constant \(c\), then every AIC shifts by \(-2c\), and every BIC also shifts by \(-2c\). The ordering is therefore unchanged. This does not license comparing scores from different samples or incompatible response likelihoods: in those cases, the changes need not be common constants.

Use Cross-Validation with Its Uncertainty

Cross-validation assesses predictive performance on held-out observations. Here suppose the reported score is mean negative log predictive density per patient, so smaller is better. The folds and scoring rule must be the same across candidates, and the complete modeling procedure—including transformations and any data-dependent choices—must be repeated using only the training portion of each split. Otherwise information from held-out responses can leak into model construction.

When models are scored on the same folds, their comparison should use paired fold differences. Let \(S_{Ak}\) and \(S_{Bk}\) be the scores for models \(A\) and \(B\) on fold \(k\), and let \(D_k=S_{Ak}-S_{Bk}\). The estimated difference is \(\bar D=K^{-1}\sum_{k=1}^K D_k\). Its variance across the observed fold differences is based on the variance of \(D_k\), not on treating the two models’ scores as unrelated.

Putting the Evidence Together

In this example, AIC favors the linear-dose interaction model \(M_2\), BIC favors the log-dose main-effects model \(M_1\), and cross-validation gives \(M_3\) the lowest mean predictive loss but does not clearly separate it from \(M_1\). The likelihood-ratio comparisons add targeted evidence within nested pairs, but they do not settle comparisons between alternative dose transformations. These outputs answer related but distinct questions; agreement is not required for a sound analysis.

A useful report names the candidate set and the shared analysis sample, gives the criterion values or score differences, states the relevant uncertainty, and explains the selection rule. It should also describe any convergence, rank, sparse-data, or separation concerns. If scientific interpretation is central, report the fitted relationship and its uncertainty rather than presenting the selected formula as a discovered truth. If prediction is central, emphasize out-of-sample performance and its uncertainty. Model selection is a decision under limited evidence, not a substitute for checking whether the models are scientifically plausible.

Common Pitfalls

• Testing nonnested models with a likelihood-ratio reference distribution. A difference in log-likelihood is always calculable, but the standard chi-squared likelihood-ratio test described here requires nesting and regularity conditions.
• Comparing scores from different analysis samples. AIC or BIC differences can then reflect which patients were included, not only model fit and complexity.
• Ignoring interaction hierarchy. Including a product term while omitting its component main effects changes the model’s meaning and usually violates the intended hierarchical construction.
• Declaring a decisive winner from a tiny score gap. Report the size and uncertainty of paired predictive differences, as well as the ranking.
• Treating an information criterion as a hypothesis test. AIC and BIC rank candidates according to different penalties; neither supplies a p-value or establishes truth.