Comparing Stability Across Selection Procedures

Why Compare Selection Procedures?

A variable-selection result can depend on more than the observed data. It can also depend on which search procedure is used and which observations happen to be in the analysis sample. With correlated predictors, for example, two variables may carry similar information, so a procedure may choose one in some samples and the other in others. A useful stability comparison therefore asks two related questions: How consistently does each procedure select each predictor? And how similar are the complete selected models?

These questions are not interchangeable. A procedure may repeatedly select one predictor while varying the rest of its model. Alternatively, two procedures may have similar predictor inclusion frequencies but select different combinations on particular resamples. Inclusion frequencies summarize predictors one at a time; model overlap summarizes the selected sets together.

A Shared Resampling Plan

For a fair comparison, all procedures should be applied to the same resampled data sets. Otherwise, a difference in results could be due to the procedures, the samples, or both. Before starting, specify the response, the candidate predictors, any forced or excluded predictors, the resampling plan, and the rule that determines the final selected model. Use the same choices wherever they apply to the three procedures.

In the hypothetical illustration below, five candidate predictors are labelled A, B, C, D, and E. A plan draws ten subsamples without replacement from one hypothetical data set. For each subsample, best-subset, forward, and backward selection are run on that same subsample, using the same candidate set and prespecified rule for choosing a final model. The table lists the predictors in each selected model. It is an illustrative set of hypothetical outcomes, not evidence from a real data set.

A model label such as ABC means that the selected model contains A, B, and C. The same row number always refers to the same resampled data set across all three procedures. This pairing matters: it lets us compare the procedures’ results under the same data perturbation.

Predictor Inclusion Frequencies

An inclusion frequency describes the behavior of a specified procedure under a specified resampling plan. It is not the probability that a predictor is truly relevant, and it is not a test of whether its regression coefficient is nonzero. Changing the resampling fraction, model-selection rule, candidate set, or procedure can change the frequency.

The complete counts make it easier to compare patterns without relying on a few selected models. The numerator in each entry is the number of resamples in which the predictor was selected; the denominator is ten for every procedure.

These are descriptive summaries of the ten outputs. For instance, A has a high inclusion frequency for all three procedures in this illustration, while C is selected much less often by forward selection than by best-subset selection. Such differences suggest where selection behavior is sensitive to the procedure. They do not by themselves explain why the procedures differ or establish which model is scientifically preferable.

Comparing Selected-Model Overlap

The numerator counts shared predictors; the denominator counts distinct predictors appearing in at least one of the models. The ratio is at most 1 because the intersection is contained in the union, so \(|S_1\cap S_2|\leq |S_1\cup S_2|\). It equals 1 precisely when the sets are identical: if the intersection and union have the same size, there can be no predictor in one set but not the other. It equals 0 when the intersection is empty. This measure compares predictor membership, not coefficient values or predictive accuracy.

To summarize overlap across the full resampling plan, calculate a Jaccard index for each paired set of outputs, then average those ten indices. The mean gives each resample equal weight. Also report the exact agreement rate when useful: it is the proportion of paired outputs whose selected predictor sets are identical. Exact agreement is stricter than a high Jaccard index, since even one additional predictor makes two sets different.

How to Read the Comparison

Inclusion frequencies and overlap answer different questions, so a useful report presents them together. Frequencies identify predictors whose selection is sensitive to the procedure or resampling. Pairwise overlap indicates whether the procedures tend to return similar combinations of predictors on the same resamples. Exact agreement adds a simple summary of how often the entire sets match.

The average Jaccard index can hide variation across resamples. For example, a moderate mean might arise from mostly similar models with a few large disagreements, or from consistently partial overlap. Looking at individual paired results, or reporting a range or distribution of the indices, helps reveal that distinction. With only ten resamples, as in this teaching illustration, the summaries are coarse; a real analysis should choose a resampling plan with enough repetitions for its descriptive purpose and report that plan clearly.

A resampling frequency is conditional on the observed data and the chosen resampling scheme. It does not account for every source of uncertainty, and it cannot turn a selected model into a confirmed scientific explanation. A predictor selected frequently may be a useful candidate under the stated procedure, but selection frequency alone is not a measure of causal effect, coefficient precision, or out-of-sample prediction quality.

Common Mistake

Wrong reasoning: “A appears in 8 of the 10 forward models, so its inclusion frequency is 0.80.” This count overlooks that forward models 1–8 and 10 include A; only model 9, BDE, omits it.

Correction: Check every row of the relevant procedure’s column. A appears in 9 of 10 forward models, so its inclusion frequency is \(9/10=0.90\). More generally, count one selection for each resample containing the predictor, and use the number of resamples—not the total number of predictor appearances—as the denominator. Also avoid treating high inclusion frequency as proof that a predictor is truly important.