Why Assess Selection Stability?
A penalized regression fit can select a set of predictors by shrinking some estimated coefficients to zero. But that set may depend on which observations happened to be used for fitting. If a slightly different sample produces a different set, reporting only one selected model can make the result look more definitive than it is.
Selection stability describes how consistently a specified fitting procedure selects predictors when the data are perturbed according to a defined resampling plan. It helps distinguish predictors that repeatedly enter the fitted model from those that appear only occasionally. It does not establish that a selected predictor is a cause of the outcome, nor does it by itself establish that the model will predict well on new data.
For a lasso fit, a numerical predictor is commonly called selected when its fitted coefficient is nonzero. That definition depends on the coefficient scale and numerical tolerance used by the software, so the rule should be documented. For a factor represented by several indicator coefficients, report whether selection is counted per indicator or at the original factor level. A factor-level rule might count the factor as selected if at least one of its indicators is nonzero; that is a reporting choice, not a universal definition.
The frequency is conditional on the entire procedure used to produce the fits. That procedure includes the resampling scheme, preprocessing, model specification, and how the penalty parameter is chosen. Keeping the penalty fixed answers a different question from retuning it in each resample.
Define the Resampling and Selection Procedure
A stability analysis is interpretable only when the resampling units and fitting steps are clear. For example, resamples might be repeated training subsets or the training portions of repeated folds. Each fit must use only its designated training observations for preprocessing and fitting. If the modeling procedure includes tuning, decide whether tuning is repeated within each resample or whether every fit uses a penalty selected beforehand.
Decide whether the question concerns selection at a fixed penalty, or selection produced by a complete procedure that also chooses the penalty.
Use a stated resampling plan and keep the outcome definition, predictor definitions, coding rules, and selection rule consistent. Apply learned preprocessing within each training resample.
For each fit, save which predictors meet the stated nonzero-coefficient or group-level rule. Also record any fit for which a predictor could not be estimated or was not eligible.
Calculate selection frequencies and, when useful, compare whole selected sets. Describe the resampling and tuning conditions; do not turn a frequency into a claim of causality.
The denominator in \(\hat{\pi}_j\) should include only fits in which predictor \(j\) was eligible and the selection status could be determined. If a predictor is absent from a resample by design, or cannot be represented under that resample’s coding, treating it automatically as “not selected” can conflate non-selection with non-availability. State how such cases are handled.
Worked Examples of Selection Frequencies
Worked Example: Comparing Frequencies at a Fixed Penalty
A hypothetical insurer models a binary renewal outcome using a penalized regression. To isolate resampling variability at one specified penalty, the analyst fixes the penalty and fits the same lasso specification on eight eligible resamples. Each fit uses the same predictor definitions and the same rule: a predictor is selected if its fitted coefficient is nonzero. The recorded selection counts are 7 for vehicle age, 5 for annual mileage, 2 for a roadside-assistance indicator, and 0 for a regional indicator. Calculate each frequency and identify what the results support.
Method. Use the empirical selection frequency, with \(B=8\) eligible fits for each predictor.
Vehicle age is selected in 7 fits: $$ \hat{\pi}_{\text{age}}=\frac{7}{8}=0.875. $$ Annual mileage is selected in 5: $$ \hat{\pi}_{\text{mileage}}=\frac{5}{8}=0.625. $$ The roadside-assistance indicator is selected in 2: $$ \hat{\pi}_{\text{assistance}}=\frac{2}{8}=0.250. $$ The regional indicator is selected in none: $$ \hat{\pi}_{\text{region}}=\frac{0}{8}=0. $$
At this fixed penalty and under these resamples, vehicle age was selected most often, while the regional indicator was never selected. This is evidence about selection reproducibility under the specified procedure, not proof that vehicle age causes renewal or that the regional indicator has no predictive value under every model.
Check. The counts sum to 14 predictor selections across the eight fits. The sum of the four frequencies is \(0.875+0.625+0.250+0=1.750\), and \(14/8=1.750\), so the summaries are consistent. The common mistake would be to interpret a frequency of 0.875 as an 87.5% probability that vehicle age is causally important; it is only the observed fraction of these specified fits selecting it.
Worked Example: Measuring Overlap Between Selected Sets
A hypothetical health-benefit model is fitted on two resamples. The first selects the set \(S_1=\{\text{age},\text{prior visits},\text{plan type},\text{region}\}\). The second selects \(S_2=\{\text{age},\text{prior visits},\text{deductible}\}\). Calculate their Jaccard similarity and describe what it says about agreement.
Method. For two nonempty selected sets, the Jaccard similarity is the size of their intersection divided by the size of their union: $$ J(S_1,S_2)=\frac{|S_1\cap S_2|}{|S_1\cup S_2|}. $$ The intersection contains age and prior visits, so \(|S_1\cap S_2|=2\). The union contains age, prior visits, plan type, region, and deductible, so \(|S_1\cup S_2|=5\).
Therefore, $$ J(S_1,S_2)=\frac{2}{5}=0.4. $$
The sets share two predictors and contain five distinct predictors in total, giving a Jaccard similarity of 0.4. It is a compact measure of set overlap, not a measure of prediction error. It also does not say whether the shared predictors have similar coefficient values.
Check. The intersection is a subset of the union, and \(2/5\) lies between 0 and 1 as required. A common mistake is to divide by the size of one selected set: \(2/4=0.5\) would not be the Jaccard similarity because its denominator must be the union, not \(S_1\) alone.
Worked Example: Correlated Predictors Can Substitute
In a hypothetical portfolio model, two highly related measures—number of recent claims and total recent claim amount—are examined across ten eligible lasso fits at the same penalty. The first measure is selected in six fits, the second in five, and both are selected together in one fit. Calculate each individual frequency, the frequency that at least one is selected, and the frequency that neither is selected.
Method. Use the stated counts, taking care not to add the individual frequencies as if the two selection events were disjoint.
The individual frequencies are $$ \hat{\pi}_{\text{claim count}}=\frac{6}{10}=0.6,\qquad \hat{\pi}_{\text{claim amount}}=\frac{5}{10}=0.5. $$
The number of fits selecting at least one measure is the count for the first plus the count for the second minus the count selecting both: $$ 6+5-1=10. $$ Thus, the frequency selecting at least one is \(10/10=1.0\), and the frequency selecting neither is \((10-10)/10=0\).
The two individual frequencies sum to 1.1 because the single fit selecting both is counted twice in that sum. The union frequency, however, is 1.0. Here the pair, considered together as a predictive signal, appears in every fit, although neither individual variable is selected in every fit. This pattern can occur when correlated predictors can substitute for one another; it does not identify one measure as uniquely important.
Check. Inclusion-exclusion gives \(6+5-1=10\), exactly the number of eligible fits. A common mistake is to claim that 11 of 10 fits selected at least one predictor by adding 6 and 5 without subtracting the overlap. The correction is to count fits in the union, not selections with duplicate fits counted twice.
Why the Frequency Formula Is Appropriate
The selection frequency is an empirical proportion. To see its interpretation, imagine repeatedly applying the same resampling-and-fitting procedure to data generated under a fixed data-generating process. For predictor \(j\), define \(I_{jb}\) to be 1 when repetition \(b\) selects it and 0 otherwise. If each repetition follows the same procedure, the expected value of an indicator equals the probability of selection under that procedure: \(E[I_{jb}]=\pi_j\). Therefore, the average indicator has expected value
This justifies the empirical proportion as a natural summary of the selection probability for that procedure. The argument does not require the indicators to be independent for the expectation identity. However, the resampled fits—especially overlapping fold training sets—are generally dependent. Thus, treating the observed frequency as if it came from \(B\) independent Bernoulli trials can give misleading uncertainty calculations. In this tutorial, the frequency is a descriptive stability summary; it is not automatically a confidence interval or a formal significance test.
Fixed Penalty or Retuned Procedure?
There are two useful but different stability questions:
| Procedure across resamples | Question answered | Important qualification |
|---|---|---|
| Use the same penalty value in every fit | How much does the selected set vary at this specified penalty? | It conditions on that penalty and does not include variation from choosing it. |
| Choose the penalty again within each resample | How much does selection vary when the fitting procedure includes its own penalty choice? | The tuning step must use only the observations available within that resample. |
The first approach isolates variation in the fitted active set at a chosen penalty. The second assesses the behavior of a broader procedure that includes tuning. Neither is universally the right answer: the appropriate choice depends on whether the question is about a fixed fitted specification or the repeatability of an end-to-end modeling procedure. State the choice explicitly. If preprocessing is learned from data, estimate it within each resample rather than using information from observations outside that resample.
Interpreting Stability Carefully
A high selection frequency means that a predictor repeatedly met the stated selection rule under the specified perturbations. It does not mean the predictor is uniquely valuable: a correlated alternative might have served a similar role. A low frequency can reflect weak signal, competition among related predictors, the selected penalty, limited data, or changes in the composition of resamples. The frequency alone does not separate these explanations.
Avoid presenting an arbitrary cutoff as a universal standard. If a report labels predictors above some threshold as “stable,” state that threshold and treat it as a descriptive convention. Include the number and type of resamples, whether the penalty was fixed or retuned, the selection rule, and how factor indicators or unavailable predictors were handled. Where correlated predictors matter, report their individual frequencies alongside a clearly defined group-level summary, if one is appropriate.
Most importantly, selection stability is not causal importance. A predictive model can repeatedly select a variable because it is associated with the outcome, because it proxies for another feature, or because the modeling procedure favors it. Causal conclusions require a separate design and assumptions. Selection frequency also is not a substitute for assessing predictive performance on data not used to fit or tune the model.
Check Your Understanding
For each problem, use the stated resampling and selection rules. Treat frequencies as descriptive summaries.
- A lasso model is fitted on 12 eligible resamples at a fixed penalty. A predictor is nonzero in 9 fits. What is its selection frequency? (Answer: \(9/12=0.750\).)
- Two selected sets are \(A=\{p,q,r\}\) and \(B=\{q,r,s,t\}\). Find their Jaccard similarity. (Answer: \(2/5=0.4\).)
- Across 20 fits, predictor \(u\) is selected 8 times, predictor \(v\) 11 times, and both are selected 4 times. How often is at least one selected, and what is that frequency? (Answer: \(8+11-4=15\) fits; \(15/20=0.750\).)
- A factor has three indicator coefficients. In 10 fits, at least one indicator is nonzero in 7 fits, while a particular indicator is nonzero in 3. Give both frequencies under (a) a factor-level rule and (b) an indicator-level rule. (Answer: factor-level \(7/10=0.700\); that indicator \(3/10=0.300\).)
- Two correlated predictors have individual selection frequencies 0.55 and 0.60 across the same 20 fits. They are selected together in 5 fits. How many fits select at least one, and what is the frequency? (Answer: 11 and 12 selections minus 5 joint selections gives 18 fits; \(18/20=0.900\).)