Why Fold-to-Fold Selection Stability Matters
Cross-validation is often used to compare predictive performance, but the fitted models within its folds can also reveal how dependent a selected model is on the particular training observations. Two folds may have nearly the same validation loss while selecting different predictors, or while assigning opposite signs to the same coefficient. That matters when a model will be explained or used to describe relationships, even if its immediate predictive performance appears similar.
The purpose of this comparison is not to declare one fold’s coefficients correct and another’s wrong. A fold-specific fit is based on a different training subset, so some variation is expected. Instead, compare the fitted selections and signs in a consistent way, then interpret those patterns alongside the validation losses. Fold-to-fold stability is a diagnostic description, not a formal test of whether a predictor matters in a population.
For interpretable comparisons, use the same response, candidate predictor set, elastic-net tuning setting, and fold assignments. In particular, compare fits at the same \(\alpha\) and \(\lambda\), where \(\alpha\) controls the lasso-versus-ridge mixture and \(\lambda\) controls penalty strength. Each fold’s preprocessing, such as centering and scaling, must be estimated using that fold’s training data only. If each fold instead chooses its own tuning setting, changes in selection can reflect both fold variation and changes in tuning; keep those sources distinct when describing the results.
The elastic-net fit uses a penalized objective. For a Gaussian response with squared-error loss, one common convention is:
Here \(n\) is the training-fold size, \(\beta_0\) is an unpenalized intercept, and the \(\beta_j\) are predictor coefficients. Scaling conventions for the objective can differ across software, so comparisons should use the same implementation and convention. The stability ideas below apply to any fitted elastic-net coefficients, provided the fits being compared are genuinely comparable.
Summarizing Selection, Signs, and Overlap
For each predictor, record how often its coefficient is nonzero. It is useful to report sign counts using all \(K\) folds as the denominator: positive frequency, negative frequency, and zero frequency then sum to one. Also report sign consistency among the folds in which the predictor was selected. These are distinct summaries. A predictor can be selected often but have an unstable sign, or be selected rarely while having the same sign whenever selected.
Here \(I(\cdot)\) equals one when its condition is true and zero otherwise. The positive-sign frequency over all folds is defined by replacing the condition with \(\hat{\beta}_j^{(k)}>0\); the negative-sign and zero frequencies are defined similarly. If a sign-consistency summary is desired only among selected folds, divide the largest of the positive and negative counts by the number of selected folds, and state that denominator explicitly.
To compare entire selected sets, the Jaccard overlap for folds \(a\) and \(b\) is the size of their intersection divided by the size of their union. It equals one for identical nonempty sets and zero for disjoint sets. If both sets are empty, the ratio is undefined; report that case separately rather than forcing a value. A mean pairwise Jaccard overlap summarizes the fold pairs, but can hide which predictors are changing, so it should accompany predictor-level frequencies.
where \(S_k\) is the selected predictor set in fold \(k\). These summaries do not replace the fold validation losses. They describe a different feature of the analysis: how consistently the fitting procedure produces the same representation.
Worked Example: Selection Frequencies With Similar Fold Losses
A hypothetical insurer fits an elastic-net model for a claim-cost response at one fixed \((\alpha,\lambda)\) setting. Four matched folds produce the following coefficients and validation losses. Predictors have been standardized within each training fold. Calculate selection and sign frequencies, then describe the pattern.
| Fold | \(\hat{\beta}_A\) | \(\hat{\beta}_B\) | \(\hat{\beta}_C\) | \(\hat{\beta}_D\) | Validation loss |
|---|---|---|---|---|---|
| 1 | 0.42 | 0.03 | 0 | 0 | 1.02 |
| 2 | 0.38 | 0 | -0.31 | 0 | 1.04 |
| 3 | 0.45 | 0.02 | 0 | 0 | 0.99 |
| 4 | 0.40 | 0 | -0.28 | 0 | 1.03 |
Step 1: Count selected folds. Predictor A is nonzero in all four folds, so its selection frequency is \(4/4=1\). Predictor B is nonzero in folds 1 and 3, giving \(2/4=0.5\). Predictor C is nonzero in folds 2 and 4, also giving \(2/4=0.5\). Predictor D is never selected, giving \(0/4=0\).
Step 2: Record signs. A is positive in all four folds, so its positive-sign frequency is \(4/4=1\). B is positive in two folds, negative in none, and zero in two: its corresponding frequencies are \(2/4=0.5\), \(0/4=0\), and \(2/4=0.5\). C is negative in two folds and zero in two, so its negative-sign frequency is \(0.5\). D is zero in all four folds.
Step 3: Check average loss. The mean validation loss is \((1.02+1.04+0.99+1.03)/4=4.08/4=1.02\). The losses range from 0.99 to 1.04. The folds therefore have broadly similar loss values in this example, while B and C appear in different folds.
Conclusion and mistake avoided. A has stable inclusion and sign; B and C have only 0.5 inclusion frequencies and do not appear together here. Similar losses do not establish stable selection. Nor do these four folds show that B or C is irrelevant: they show that selection varies across these training subsets at the specified setting.
Why Similar Predictions Can Have Different Selections
When predictors carry overlapping information, a model can achieve similar predictions using different members of a correlated group. This is one reason selection instability need not appear as a large change in validation loss. The following argument shows how close predictors can produce close fitted contributions even when a coefficient is attached to one rather than the other.
Suppose one fit uses a coefficient \(b\) on predictor values \(x_1\), while another uses the same coefficient on \(x_2\). Their contribution difference for an observation is \(b x_1-b x_2=b(x_1-x_2)\).
If \(x_2=x_1+e\), where \(e\) is a small difference, then \(b(x_1-x_2)=-be\). When \(e\) is small over the observations, the two contributions can be close.
Close fitted contributions can lead to similar predictions and losses, although the selected variable differs. This explains a possible pattern; it does not guarantee that correlated predictors will always be exchanged, because the penalty, data, and other predictors also affect the fitted coefficients.
Worked Example: A Coefficient Selected With Opposite Signs
At a fixed elastic-net tuning setting, a hypothetical model is fitted on five training folds. Predictor \(M\) and predictor \(N\) are both selected in every fold. Their coefficients and validation losses are shown below. Determine their selection frequencies and sign stability, and explain why a coefficient average alone is inadequate.
| Fold | \(\hat{\beta}_M\) | \(\hat{\beta}_N\) | Validation loss |
|---|---|---|---|
| 1 | 0.20 | 0.40 | 0.810 |
| 2 | -0.17 | 0.39 | 0.812 |
| 3 | 0.19 | 0.42 | 0.808 |
| 4 | -0.15 | 0.38 | 0.811 |
| 5 | 0.18 | 0.41 | 0.809 |
Step 1: Calculate selection frequencies. Both M and N have nonzero coefficients in all five folds. Each selection frequency is \(5/5=1\).
Step 2: Calculate the sign summaries. M is positive in folds 1, 3, and 5, and negative in folds 2 and 4. Its positive and negative frequencies over all folds are \(3/5=0.6\) and \(2/5=0.4\). Among its five selected fits, the most common sign occurs in \(3/5=0.6\) of them. N is positive in all five folds, so its positive-sign frequency and selected-fit sign consistency are both \(5/5=1\).
Step 3: Calculate the mean loss. The losses sum to \(0.810+0.812+0.808+0.811+0.809=4.050\). Dividing by five gives a mean of \(0.810\); the range is \(0.812-0.808=0.004\). M's sign changes even though the losses are close.
Step 4: Check the coefficient average. M's coefficient average is \((0.20-0.17+0.19-0.15+0.18)/5=0.25/5=0.05\). This average is positive but conceals the two negative fold fits and the limited sign consistency. N's coefficient average is \((0.40+0.39+0.42+0.38+0.41)/5=2.00/5=0.40\). Reporting only average coefficients would therefore hide a meaningful difference in sign stability.
Conclusion and mistake avoided. M is selected consistently but its sign is not; N is selected with a stable positive sign. Do not infer stable direction from a nonzero selection frequency or from an average coefficient. The result describes fold-specific fits, not a formal conclusion about causal direction.
Using Set Overlap to Locate Instability
Predictor-level frequencies identify which variables vary, while set overlap indicates how similar the full selections are. A modest mean overlap can arise because folds repeatedly exchange one member of a correlated group while retaining other predictors. Pairwise overlap should therefore be read together with a table of predictor frequencies and signs, not treated as a stand-alone score or a universal cutoff.
Worked Example: Pairwise Jaccard Overlap for Substitute Predictors
Four folds of a hypothetical severity model produce the following selected sets at the same elastic-net tuning setting: \(S_1=\{P,R\}\), \(S_2=\{Q,R\}\), \(S_3=\{P,R\}\), and \(S_4=\{Q,R\}\). Their validation losses are 0.50, 0.51, 0.49, and 0.50. Calculate the predictor frequencies and the mean pairwise Jaccard overlap.
Step 1: Count inclusion. P is selected in folds 1 and 3, so its frequency is \(2/4=0.5\). Q is selected in folds 2 and 4, also \(2/4=0.5\). R is selected in all four, so its frequency is \(4/4=1\). Each time R is selected, its fitted coefficient is negative; thus its negative-sign frequency is \(4/4=1\).
Step 2: Compare the six fold pairs. There are \(4(4-1)/2=6\) pairs. The identical pairs (1, 3) and (2, 4) each have Jaccard overlap 1. Each of the other four pairs compares \(\{P,R\}\) with \(\{Q,R\}\): the intersection is \(\{R\}\), of size 1, and the union is \(\{P,Q,R\}\), of size 3. Each such overlap is \(1/3\).
Step 3: Average the overlaps. The mean is \((1+1+4(1/3))/6=(2+4/3)/6=(10/3)/6=5/9\), approximately 0.5556. The mean validation loss is \((0.50+0.51+0.49+0.50)/4=2.00/4=0.50\).
Conclusion and mistake avoided. R is selected consistently with a stable negative sign, while P and Q alternate. The mean overlap of \(5/9\) reflects that exchange, despite losses centered at 0.50. A mean overlap alone would not tell us that R is stable; the predictor-level counts supply that detail.
Common Mistakes and Interpretation Limits
Fold-to-fold summaries can be useful, but only when their denominators and scope are clear. They are descriptive summaries of the fitted procedure under a particular validation plan and tuning choice. They are not independent observations of a predictor’s true importance, and they should not be turned into a significance test without a suitable inferential framework.
- Using only selected folds as the inclusion denominator. In the five-fold example, M is selected in all five folds, so its selection frequency is \(5/5=1\), not \(5/3\). The value \(3/5=0.6\) is its positive-sign frequency, not its inclusion frequency.
- Confusing selection stability with sign stability. A predictor can be nonzero in every fold and still switch signs, as M did.
- Calling similar losses proof of equivalent models. Similar validation losses do not establish that two models have the same selected variables, coefficients, or behavior in other data.
- Comparing fits at incompatible settings. If \(\alpha\), \(\lambda\), candidate predictors, or preprocessing differ across folds, selection changes cannot be attributed to fold variation alone.
- Reading fold frequencies as formal probabilities. A frequency such as \(3/5\) describes these five fits. It is not automatically a probability that the predictor is truly relevant, nor a confidence level.
Check Your Understanding
For each problem, distinguish selection frequency from sign frequency and interpret overlap or loss summaries only as requested.
- A predictor has coefficients \(0.30, 0, 0.20, 0\) across four folds. Find its selection frequency, positive-sign frequency over all folds, and positive-sign consistency among selected fits. (Answer: selection frequency \(2/4=0.5\); positive-sign frequency \(2/4=0.5\); selected-fit positive-sign consistency \(2/2=1\).)
- Across five folds, predictor \(T\) has coefficients \(0.30,-0.20,0,0.10,-0.10\). Calculate its selection frequency and its positive, negative, and zero frequencies over all folds. (Answer: selection \(4/5=0.8\); positive \(2/5=0.4\); negative \(2/5=0.4\); zero \(1/5=0.2\).)
- Four selected sets are \(S_1=\{A,B\}\), \(S_2=\{A\}\), \(S_3=\{B,C\}\), and \(S_4=\{A,C\}\). Find A, B, and C’s selection frequencies and the mean of the six pairwise Jaccard overlaps. (Answer: A \(3/4=0.75\), B \(2/4=0.5\), C \(2/4=0.5\); pairwise overlaps are \(1/2,1/3,1/3,0,1/2,1/3\), with mean \(1/3\).)
- Two elastic-net settings have fold losses 0.40, 0.42, 0.38, 0.40 and 0.39, 0.41, 0.39, 0.41, respectively. At the first setting, U is selected in all folds and V in folds 1 and 3. At the second, U is selected in all folds and W in folds 1, 2, and 4. Calculate both mean losses and the inclusion frequencies of V and W. (Answer: both mean losses are \(1.60/4=0.40\); V is selected in \(2/4=0.5\) folds and W in \(3/4=0.75\) folds. Similar mean losses do not make the selected sets identical.)
- Two standardized predictors are strongly correlated. Across comparable folds, one is selected in some fits and the other in the remaining fits, while validation losses are close. Give one reasonable interpretation and one conclusion that is not justified by this pattern. (Answer: the predictors may provide overlapping information, allowing different selections with similar predictions; it is not justified to conclude that either predictor is unimportant or that the fold frequency is its probability of being truly relevant.)