Stepwise Selection in a Retail Sales Analysis

Why Stepwise Selection Needs Careful Interpretation

A retail analyst may have several plausible predictors of store sales: customer traffic, promotion intensity, seasonal demand, local income, and competitor activity. Including every available variable can make a model difficult to interpret, while choosing predictors by intuition alone may overlook useful combinations. Stepwise selection offers a systematic search through a specified set of candidate models. It is a search procedure, however, not a guarantee that the final model is the uniquely correct explanation of sales.

This tutorial follows a hypothetical analysis of weekly sales across 24 comparable stores. The outcome is sales in thousands of dollars per week. The candidate predictors are a traffic index, promotion intensity, and a seasonal-demand index. The examples are constructed for instruction and do not describe real stores.

The Model and the Search Rule

For a chosen subset of \(q\) predictors, write the ordinary linear regression model as

Here \(Y_i\) is sales for store \(i\), the \(x\)'s are the selected predictors, and the \(\beta\)'s are regression coefficients. For the usual Gaussian linear-model likelihood calculations, assume the errors are independent, have mean zero and common variance \(\sigma^2\), and are normally distributed: \(\varepsilon_i\sim N(0,\sigma^2)\). These assumptions concern the model errors conditional on the predictors. The sales observations and predictors must also be measured comparably, and the candidate model set and selection rule should be specified before inspecting the selection results. These conditions make the stated model-comparison calculations meaningful; they do not ensure that the candidate set contains a good model.

A stepwise procedure repeatedly compares models that differ by adding or removing candidate predictors. In forward selection, the search starts with an intercept-only model and considers adding predictors. In backward selection, it starts with a specified larger model and considers removals. A bidirectional procedure may allow both additions and removals at each stage. The procedure stops according to its stated rule—for example, when no permitted change improves the chosen criterion. Different starting models, candidate sets, or stopping rules can lead to different results.

One possible rule is to compare models with an AIC-style score. Let \(RSS\) be the residual sum of squares for a model fitted to the same \(n\) observations, and let \(k\) be its number of fitted slope coefficients. An intercept is included in every model here, so its contribution is common when models are compared. We use the simplified score

Smaller \(AIC^*\) is preferred by this criterion. It rewards lower residual error but penalizes additional slopes. It is a model-comparison score, not a probability that a model is true, and its values are meaningful here only for models fitted to the same response observations under the same outcome scale and likelihood setup.

Why the Score Balances Fit and Complexity

Under the normal-error model, the maximized likelihood estimates the error variance by \(RSS/n\). Substituting this estimate into the Gaussian log likelihood gives a negative twice-log-likelihood of the form \(n\ln(RSS/n)\) plus terms common to models fitted to the same data. AIC adds twice the number of estimated parameters to this quantity. The simplified score above omits the common terms and uses the number of slope coefficients for \(k\); the intercept and error-variance contributions are common across these comparisons.

Consequently, if a proposed step changes a model from \(k_1\) to \(k_2\) slopes and from \(RSS_1\) to \(RSS_2\), the score change is

A negative change favors the new model by this criterion; a positive change favors the old one. This comparison makes the trade-off explicit: adding a predictor is worthwhile only when the reduction in residual error is large enough to offset the complexity penalty.

Resampling to Examine Selection Stability

A selected predictor can depend on which observations happen to be in the data. One way to examine that dependence is to resample the observational units, rerun the entire selection procedure on each resample, and summarize how often each predictor or model is selected. For an ordinary bootstrap illustration, each resample has the original sample size and is drawn with replacement from the observed stores. This procedure is most defensible when stores can reasonably be treated as independent sampling units; if the data have meaningful dependence or a different sampling design, the resampling scheme must reflect it.

Selection frequency is descriptive. It reports how often the specified procedure selects a predictor in the resamples. It is not a posterior probability that the predictor matters, and it does not repair a misspecified model or account for every uncertainty in the analysis.

Search Multiplicity and Near-Ties

Searching many candidate predictors creates more opportunities for a variable to look useful by chance. A simple calculation illustrates the issue. Suppose eight candidate predictors have no genuine association with the outcome after accounting for the model, and suppose—for illustration—that eight null-hypothesis p-values are independent and each is uniformly distributed between 0 and 1. The probability that none is below 0.05 is \(0.95^8\). Therefore the probability that at least one is below 0.05 is

So even in this simplified setting, screening eight null predictors creates about a 33.66% chance of finding at least one p-value below 0.05. Real predictors are often correlated, so the independence calculation need not apply exactly. The general caution remains: ordinary p-values and confidence intervals calculated after data-driven selection do not automatically retain their usual interpretation. A small p-value from the winning step should not be presented as though that predictor had been the only one considered in advance.

A related caution is that two models can have very similar scores. In the example, the traffic-and-season model has \(RSS=149\), while the promotion-and-season model has \(RSS=145\). Both have two slopes, and

The promotion-and-season model has the lower score, but the difference is small. This ranking says only which of these two models scores better under the stated criterion and data. It is not a test establishing that one model is true, nor evidence that the other model is impossible. The selected model is not uniquely established simply because a search returns one winner.

How to Report a Stepwise Analysis

A clear report describes the outcome, candidate predictors, observation set, selection direction, criterion, and stopping rule. It should distinguish the search result from subsequent interpretation. For example: “In the hypothetical sample of 24 stores, forward selection among traffic, promotion, and seasonal demand using the stated AIC-style score stopped with promotion and seasonal demand.” If resampling was done, report the resampling unit and scheme, the number of resamples, and the selection frequencies. Do not imply that selection frequencies are significance levels or causal probabilities.

The model also inherits the limitations of its data and assumptions. The linear conditional-mean form may not represent the relationship well; errors may be dependent or have unequal variance; important predictors may be absent; and predictor measurements may be noisy. Stepwise selection addresses none of those issues by itself. The earlier material on collinearity and influential observations is also relevant: correlated predictors can compete for selection, and a small number of unusual observations can affect fitted models and the selected path.

Common Mistakes

• Treating the selected model as the only correct one. The procedure selects one endpoint according to a particular search path and score. Similar-scoring alternatives may also be reasonable.
• Reading selection frequency as probability of truth. A frequency such as 80% means the procedure selected that predictor in 80% of the specified resamples—not that there is an 80% probability its coefficient is nonzero.
• Ignoring how many candidates were searched. Selection after screening many predictors can make conventional post-selection p-values and intervals overconfident if treated as though the model were prespecified.
• Calling a selected association causal. A selected predictor may be associated with sales, but an observational regression search alone does not show that changing the predictor would change sales.
• Comparing scores from different observations. The score comparison shown here assumes candidate models use the same response observations and compatible likelihood calculations.