A Worked Stepwise Analysis

Why Work Through a Stepwise Path?

Stepwise selection is a search procedure: it builds or reduces a regression model through a sequence of comparisons between nested models. The final model depends not only on the data, but also on the search direction, the rule for adding or deleting predictors, and the thresholds used. Tracing the calculations makes those choices visible. It also helps prevent a consequential mistake: confusing a p-value that is small enough to retain a predictor with one that is large enough to remove it.

The example below uses the same hypothetical data for forward selection, backward elimination, and bidirectional selection. The aim is not to claim that any one selected model is objectively correct. It is to show exactly how each procedure follows its stated rule, and how modest threshold changes can produce a different answer.

Model, Tests, and Selection Rules

Suppose data were recorded for \(n=30\) hypothetical service centers. The response \(Y\) is each center’s mean follow-up delay, measured in days. Four candidate quantitative predictors are \(A\) (staffing hours), \(B\) (referral volume), \(C\) (a measure of urban location), and \(D\) (outreach spending). Each candidate model includes an intercept. These variables are used only to illustrate the calculations; the example does not describe a real study.

For a full-rank model with \(k\) predictors, the residual degrees of freedom are \(n-k-1\). Its residual sum of squares, abbreviated RSS, is the sum of squared residuals and therefore has units of days squared here. To test whether adding one predictor improves a reduced model, compare that model with the larger, nested model using a partial F test. The hypotheses are that the additional predictor’s coefficient is zero, conditional on the predictors already in the model, versus that it is not zero.

We use an entry threshold of 0.10 and a removal threshold of 0.10. Thus, a candidate may enter when its p-value is at most 0.10, and a predictor is removed only when its deletion p-value is greater than 0.10. These are illustrative decision rules, not universal recommendations. For the usual exact partial F tests, assume a correctly specified linear mean model for the comparisons, independent errors with mean zero and constant variance, and normally distributed errors. The same observations must be used in every compared model. These assumptions provide the standard reference distribution for the tests; the selection arithmetic itself does not establish that the assumptions hold.

A Small Set of Regression Summaries

The following hypothetical fits provide the RSS values needed to trace the paths. The null model contains only an intercept. The other rows give selected candidate models; all fit summaries use the same 30 centers. Since adding predictors cannot increase RSS, the listed values decrease along each nested path.

Additional fits used to check deletion decisions have RSS \(37.0\) for \(BC\), \(27.0\) for \(ABD\), \(31.0\) for \(ACD\), and \(35.0\) for \(BCD\). Each is a hypothetical regression summary, not an estimate from an actual service-center study. Assuming each compared model is full rank and each added predictor increases the design-matrix rank by one, these comparisons have numerator degrees of freedom 1.

Why the Partial F Calculation Works

For a reduced model with RSS \(RSS_R\) and a larger model with RSS \(RSS_F\), adding \(q\) predictors reduces RSS by \(RSS_R-RSS_F\). The partial F statistic divides the reduction per added predictor by the larger model’s residual mean square. For a single predictor, \(q=1\).

Here \(df_F\) is the residual degrees of freedom in the larger model. For a prespecified comparison, under the null hypothesis and the linear-model assumptions stated above, and provided the larger model has the required rank, this statistic has an \(F\) distribution with \(q\) and \(df_F\) degrees of freedom. In a stepwise path, comparisons are selected adaptively, so their p-values are nominal per-comparison values and do not by themselves provide selection-adjusted inference or overall error control. For one added predictor, a large value indicates a reduction in RSS that is large relative to the residual mean square. The p-value is the upper-tail probability from that reference distribution.

For example, the test for adding \(B\) to model \(A\) compares RSS 38.0 with RSS 29.0. Substitution gives:

The reference distribution is \(F_{1,27}\), giving \(p\approx0.007426\). This is a test of \(B\) conditional on \(A\), not a test of \(B\) in isolation.

Worked Example: Forward Selection

Worked Example: Backward Elimination

Worked Example: Bidirectional Selection

Threshold Sensitivity and Interpretation

The agreement at a 0.10 threshold does not mean the selected model is insensitive to the rule. The addition of \(C\) to \(AB\) has \(p\approx0.0517\). If the entry threshold were tightened to 0.05, forward selection would stop at \(AB\), because \(0.0517\) is above 0.05. Bidirectional selection using 0.05 for entry would likewise not add \(C\). In backward elimination with a 0.05 removal threshold, \(C\) would be removed from \(ABC\), because its deletion p-value, 0.0517, exceeds 0.05. In each case, the resulting model would be \(AB\).

This comparison shows why a threshold is part of the procedure rather than a property discovered from the data. A p-value just above or below a cutoff does not create a sharp scientific distinction. The cutoff determines a mechanical decision; it does not prove that a predictor has no relationship with the response when excluded, or that a retained predictor is important in every practical sense.

The regression coefficients in a selected model describe fitted associations conditional on the other included predictors, in their stated units. For example, a coefficient for staffing hours would express the fitted change in mean follow-up delay, in days, associated with a one-hour difference in staffing hours while holding the other included predictors fixed. It would not by itself establish a causal effect. Moreover, because these predictors were chosen by repeatedly searching the same data, ordinary coefficient p-values and confidence intervals do not automatically account for that search. Selection results should be reported as results of a specified procedure, with the thresholds and search path made clear.

Common Mistakes

• Reversing the deletion rule. Wrong: “The deletion p-value is 0.0517, so remove \(C\) at a 0.10 removal threshold.” Correction: removal requires a p-value greater than 0.10, and 0.0517 is smaller, so retain \(C\).
• Using a marginal test for a conditional decision. The decision to add \(C\) to \(AB\) uses the partial test comparing \(AB\) with \(ABC\), not the test of \(C\) alone against the null model.
• Stopping after an addition without rechecking. In bidirectional selection, addition and deletion checks are made as specified by the procedure; a change to the model can change the conditional tests.
• Treating the chosen model as certain. The selected set can change with the thresholds or procedure. Report the rule and the observed path rather than implying that selection has identified a uniquely true model.
• Reading association as causation. A selected regression coefficient is an adjusted association under the model, not proof that changing the predictor would cause the response to change.