Stepwise Selection In GLMs

What Stepwise Selection Does

A candidate set specifies the GLMs that are eligible for comparison. Stepwise selection searches this set by making a sequence of limited moves, usually adding or deleting one term at a time. The procedure is useful when a full search is impractical or when a sequence of scientifically motivated decisions is desired. Its output, however, depends on the starting model, the available moves, and the rule used to accept a move.

Fix a finite collection of candidate models and a score \(S(M)\) to be minimized, such as AIC. For likelihood-based comparisons, the models must be fit to the same response data under comparable likelihoods; dispersion treatment and parameter counting must also be consistent. A step is accepted only if it satisfies the specified rule. In the AIC examples below, the rule is strict improvement: accept a move only when it lowers AIC. A predetermined tie-breaking rule is needed if several eligible moves have equal scores.

“Eligible” depends on the candidate set and any restrictions placed on it. For example, when hierarchy is required for interactions, an interaction cannot be added unless its parent main effects are present, and a main effect cannot be deleted while a retained interaction requires it. The search compares only moves that preserve those restrictions. A stopping rule is therefore meaningful only together with a clear specification of eligible moves and the score or threshold used to accept them.

Tracing an AIC Search

For a regular likelihood model with \(k\) estimated parameters, AIC is \(-2\ell(\hat{\theta})+2k\). When comparing two nested Gaussian linear models with an estimated residual variance, the variance parameter is counted in both models and cancels in the difference. If a model with \(q\) regression coefficients has residual sum of squares \(RSS_q\), then adding one coefficient changes AIC by

A negative change favors the larger model under AIC; a positive change favors the smaller one. The \(+2\) is the penalty for the additional coefficient. This formula is specific to the stated Gaussian setting, not a general substitute for computing AIC in every GLM.

A strict AIC rule differs from a significance-threshold rule. With a significance rule, for example, a term might be entered when its conditional test has \(p\)-value below one threshold and removed when its \(p\)-value exceeds another. Such a procedure needs explicit thresholds, a definition of the test, and a decision about what to do when several terms qualify. It does not, in general, minimize AIC at each step.

Backward and Bidirectional Searches

Backward selection starts at the opposite end: it considers the full eligible model first and asks whether deleting any term improves the score. For an AIC search, compare the refitted reduced model’s AIC with that of the current model. If multiple deletions lower AIC, choose the deletion giving the lowest resulting AIC, then refit and repeat. Stop when every eligible deletion would leave AIC unchanged or increase it.

In bidirectional selection, after a move the procedure checks both kinds of eligible move. A common version selects the best strictly score-reducing move among all eligible additions and deletions, then refits and repeats. A different implementation may use separate entry and removal thresholds. These variants need not follow the same path or stop at the same model; the algorithm should be described precisely rather than identified only by the phrase “stepwise.”

What a Stopping Point Guarantees

For a finite candidate set, a score-improving stepwise search cannot continue indefinitely if every accepted move strictly decreases the score. More specifically, if the algorithm always chooses the best improving neighboring model, its endpoint is optimal among the endpoint’s one-step neighbors. This is a local statement: it says nothing about a model that can be reached only by a sequence whose first move is unfavorable.

Proof. Each accepted step visits a model with a strictly smaller score than the model just visited. Consequently, no model can be visited twice: revisiting one would require its score to be both strictly smaller and equal to its earlier score. The candidate set contains only finitely many models, so the search cannot make infinitely many distinct visits. It must terminate. At termination, by the stated stopping condition, no eligible one-step neighbor has lower score. That is exactly local minimality with respect to the eligible moves. \(\square\)

Stepwise selection is not thereby a global search. For illustration, consider a score landscape with an initial model scoring 10, each of two single-term additions scoring 11, and the model containing both terms scoring 8. A strict forward rule stops at the initial model because neither first addition improves the score, even though the two-term model has a lower score. The numerical pattern illustrates the distinction between a one-step stopping point and a global minimum; it is not a claim that every such score pattern arises in every GLM or candidate set.

Instability and Inferential Caution

A stepwise path is conditional on the observed data. Small data changes can alter which candidate looks best at an early step; that choice changes the later set of available moves and can change the final model. Near-ties in the selection score are a practical warning: the selected term may be determined by a small difference in fit rather than a strong distinction between candidate models.

This dependence on the search matters for inference. Ordinary standard errors, confidence intervals, and tests computed after selection generally describe uncertainty as if the selected model had been fixed in advance. They do not automatically account for the fact that the same data were used to choose that model. A small conditional \(p\)-value after searching many alternatives is therefore not, by itself, a reliable measure of evidence that accounts for the search. Prediction assessment should likewise respect the selection process: when evaluating predictive performance, the selection step belongs inside the training procedure, not in information available from held-out observations.

Stepwise procedures can still be useful as transparent, constrained search heuristics, especially when the candidate terms and scientific restrictions are specified before inspecting the results. They should not be mistaken for a guarantee of scientific truth, a guarantee of global score optimality, or a correction for uncertainty created by searching.

Common Mistakes

• Treating the endpoint as globally best. A stopping rule based on one-term moves establishes at most a local optimum among eligible neighbors.
• Leaving the rule implicit. “Stepwise” alone does not say whether terms are chosen by AIC, likelihood-ratio tests, or thresholds, nor how ties are handled.
• Comparing scores from incompatible fits. AIC comparisons require comparable likelihoods and consistent parameter counting; a change in response data or likelihood basis can invalidate the comparison.
• Reading selected-model standard errors as if selection never occurred. The usual post-fit calculations do not automatically include uncertainty from the search.