Why Predictor Screens Need a Stability Check
A screening method may identify predictors that appear strongly associated with an outcome in one data set. But a practical question remains: would it identify much the same predictors if the observed data had been slightly different? A screen that repeatedly selects the same features may offer a more dependable shortlist than one whose choices change substantially across samples.
Stability is not a new measure of association. It describes how consistently a chosen screening procedure selects or ranks predictors when the data are resampled or divided into different partitions. It is aligned with the SOA Exam PA (Predictive Analytics) syllabus. The earlier material on univariate screening and multiple testing explains how individual screening statistics are obtained; here, the focus is on how repeatable the resulting selections are.
A bootstrap sample is formed by sampling observations with replacement from the observed data, usually keeping the original sample size. A data partition is a subset of observations formed under a specified splitting scheme. For either approach, apply the same screening rule to every replicate. For example, if the rule retains the top \(k\) predictors, retain exactly the top \(k\) in each replicate, using a consistent tie-breaking rule.
The choice of resampling design matters. Bootstrap samples represent repeated samples from the empirical distribution, while partitions show how selections differ among subsets of the available data. The resulting stability summaries are descriptive of that design; they are not automatically measures of performance on new data.
Inclusion Frequencies
Suppose \(B\) repeated screens are run and \(S_b\) is the set of predictors selected in replicate \(b\). The inclusion frequency of predictor \(j\) is the proportion of replicates in which it is selected:
Here, \(I(j\in S_b)\) equals 1 if predictor \(j\) is selected in replicate \(b\), and 0 otherwise. The frequency is between 0 and 1. A value of 1 means the predictor was selected in every replicate; it does not mean that the predictor is certainly important in the population.
Worked Example: Counting Top-Two Selections
A hypothetical insurer screens four candidate variables for a numeric outcome and retains the top two in each of four bootstrap samples. The selected sets are \(\{M,N\}\), \(\{M,P\}\), \(\{M,N\}\), and \(\{Q,N\}\). Find each predictor’s inclusion frequency and identify the most consistently selected predictors.
Method. Count how many of the four sets contain each predictor, then divide each count by \(B=4\).
Predictor \(M\). \(M\) appears in the first, second, and third sets. Its count is 3, so \(\widehat{\pi}_M=3/4=0.75\).
Predictor \(N\). \(N\) appears in the first, third, and fourth sets. Thus \(\widehat{\pi}_N=3/4=0.75\).
Predictor \(P\). \(P\) appears only in the second set, giving \(\widehat{\pi}_P=1/4=0.25\).
Predictor \(Q\). \(Q\) appears only in the fourth set, giving \(\widehat{\pi}_Q=1/4=0.25\).
The check is that the four frequencies sum to \(0.75+0.75+0.25+0.25=2\). Each of the four screens selected two predictors, so the total number of selections is \(4(2)=8\), and the frequencies must sum to \(8/4=2\). \(M\) and \(N\) are tied as the most frequently selected. The common mistake is to count \(M\) in only two sets; inspecting all four sets confirms it appears three times.
Ranks and Overlap Tell Different Parts of the Story
Inclusion frequency depends on a cutoff. If the screen retains the top two, a predictor ranked third is treated the same as one ranked last: neither is selected. When a complete ranking is available in every replicate, average ranks can show more detail. Rank 1 denotes the highest-ranked predictor, rank 2 the next highest, and so on. The mean rank for predictor \(j\) is the average of its replicate ranks.
Lower mean ranks indicate better average positions, but a mean alone can conceal variation. It is useful to report the full set of ranks or their spread as well. If a screen produces only a selected list rather than a complete ranking, do not quietly assign unselected predictors arbitrary ranks. Report inclusion frequencies separately, or state an explicit and consistent rank convention.
Another summary compares selected sets directly. For two sets \(S_a\) and \(S_b\), their Jaccard similarity is the size of their intersection divided by the size of their union. It equals 1 for identical sets and 0 for sets with no common predictors.
Worked Example: Comparing Ranks and Top-Two Sets
A hypothetical claims study ranks four candidate predictors in three data partitions. The rank orders are: partition 1, \(A,B,C,D\); partition 2, \(B,A,D,C\); and partition 3, \(A,C,B,D\). Calculate the top-two inclusion frequencies and mean ranks. Then find the mean pairwise Jaccard similarity of the top-two sets.
Top-two sets. The selected sets are \(S_1=\{A,B\}\), \(S_2=\{A,B\}\), and \(S_3=\{A,C\}\). Predictor \(A\) is selected 3 times, so its frequency is \(3/3=1\). Predictor \(B\) is selected 2 times, so its frequency is \(2/3\). Predictor \(C\) is selected once, so its frequency is \(1/3\). Predictor \(D\) is never selected, so its frequency is 0. As a check, the frequencies sum to \(1+2/3+1/3+0=2\), matching the two selections per partition.
Mean ranks. For \(A\), the ranks are \(1,2,1\), so \(\bar r_A=(1+2+1)/3=4/3\). For \(B\), the ranks are \(2,1,3\), so \(\bar r_B=6/3=2\). For \(C\), the ranks are \(3,4,2\), so \(\bar r_C=9/3=3\). For \(D\), the ranks are \(4,3,4\), so \(\bar r_D=11/3\). These averages show that \(A\) ranks highest on average, though \(B\) sometimes ranks first.
Pairwise Jaccard similarities. \(S_1\) and \(S_2\) are identical, so their similarity is \(2/2=1\). For \(S_1\) and \(S_3\), the intersection is \(\{A\}\) and the union is \(\{A,B,C\}\), giving \(1/3\). The same calculation gives \(J(S_2,S_3)=1/3\). The mean over the three pairs is \((1+1/3+1/3)/3=5/9\), approximately \(0.556\). The calculation is consistent with two identical selections and one selection that differs by one feature.
The common mistake is to call \(B\) stable merely because it is selected in two of three partitions. Its rank varies from 1 to 3, and the Jaccard summary also shows that the selected sets are not identical. Frequency, rank, and set overlap describe related but distinct aspects of repeatability.
Why Inclusion Frequency Has This Form
The inclusion-frequency formula is the average of a 0–1 indicator. To see why it represents the observed selection rate, define \(I_b=I(j\in S_b)\) for one predictor. Across the \(B\) replicates, the total number of times it is selected is \(\sum_{b=1}^{B} I_b\). Dividing that count by the number of replicates gives the proportion selected, which is exactly \(\widehat{\pi}_j\).
There is also a probability interpretation, with an important qualification. If replicate screens are generated under a specified resampling procedure, let \(\pi_j\) denote the probability that this procedure selects predictor \(j\), conditional on the observed data and the fixed screening rule. For each replicate, \(E[I_b]=\pi_j\). Therefore:
Thus the average indicator targets the procedure’s selection probability under that resampling scheme. This does not establish that the predictor has a causal effect, is statistically significant, or will improve prediction. It describes selection consistency under specified choices and available data.
Worked Example: Stable Groups, Unstable Individual Choices
A hypothetical portfolio analysis contains three closely related measurements, \(X_1\), \(X_2\), and \(X_3\), along with a distinct feature \(Z\). Each of four bootstrap screens retains the top three predictors. The selected sets are \(\{X_1,X_2,Z\}\), \(\{X_1,X_3,Z\}\), \(\{X_2,X_3,Z\}\), and \(\{X_1,X_2,Z\}\). Calculate individual inclusion frequencies, then assess the group of three related measurements.
Individual frequencies. \(X_1\) appears in sets 1, 2, and 4, so its frequency is \(3/4=0.75\). \(X_2\) appears in sets 1, 3, and 4, giving \(3/4=0.75\). \(X_3\) appears in sets 2 and 3, giving \(2/4=0.50\). \(Z\) appears in all four sets, giving \(4/4=1.00\). The frequencies sum to \(0.75+0.75+0.50+1.00=3\), which matches three selections in each of four screens.
Group frequency. In every set, at least one of \(X_1,X_2,X_3\) appears. If the question is whether this group contributes a selected representative, the group is represented in \(4/4=1.00\) of the screens. Yet no single member has frequency 1.00. This pattern is consistent with interchangeable features: the screen reliably chooses from the group, but the particular member changes.
The appropriate interpretation is not that all three measurements have equal or confirmed predictive value. Their changing individual selections are a warning that the data or screening rule may not distinguish them reliably. The group-level calculation is a useful diagnostic, not proof that the group has independent information or should automatically be retained in a final model.
Interpreting Instability and Avoiding Traps
Low stability can arise for several reasons. A limited sample may contain too little information to separate weak signals from random variation. Strongly correlated or otherwise similar predictors may substitute for one another in a marginal screen. A hard cutoff can also make near-tied predictors switch places, so a small change in the data changes which side of the cutoff they fall on. These patterns call for cautious interpretation, not automatic rejection of every unstable predictor.
A stable screen is not necessarily a good screen. A consistently selected feature may be consistently irrelevant to the eventual modeling objective, and a predictor with modest marginal association may still matter in a model for other reasons. Stability does not replace assessment of predictive performance, subject-matter judgment, or the limitations of the screening method. It also does not correct problems created by an unsuitable candidate set or selection rule.
For a fair comparison, hold the screening method, candidate set, outcome definition, and selection cutoff fixed across replicates. If they change, observed differences mix data-driven instability with changes in the procedure. When rankings contain ties, specify the tie rule; otherwise, counts near the cutoff can depend on arbitrary ordering. Report the number and type of replicates and the summaries used, rather than presenting a frequency without its context.
State the candidate predictors, ranking rule, and cutoff such as top \(k\).
Decide whether repeated bootstrap samples or data partitions address the stability question, and state how they are formed.
Apply the fixed screen in every replicate and record selections or complete ranks.
Report inclusion frequencies and, when useful, rank summaries or set overlap. Investigate changing choices, especially among related predictors, without treating stability as proof of predictive value.
A frequent arithmetic trap is to report an inclusion frequency without carefully counting every set. For the four sets in the first worked example, a mistaken count of two for \(M\) would give \(2/4=0.50\). The correction is to check membership set by set: \(M\) occurs in the first, second, and third sets, so the correct calculation is \(3/4=0.75\). As a general check, if every replicate retains exactly \(k\) predictors, the sum of all candidate predictors’ inclusion frequencies must equal \(k\).
Check Your Understanding
For each problem, use the stated resampling design and selection rule. Treat frequencies as proportions of the listed replicates.
- A screen retains the top two predictors in five bootstrap samples: \(\{A,B\}\), \(\{A,C\}\), \(\{A,B\}\), \(\{B,D\}\), and \(\{A,D\}\). Find the inclusion frequency of each predictor. (Answer: \(A=0.80,\ B=0.60,\ C=0.20,\ D=0.40\).)
- Four partitions produce top-two sets \(\{U,V\}\), \(\{U,W\}\), \(\{V,W\}\), and \(\{U,V\}\). Find the inclusion frequencies of \(U,V,W\), and check their sum against the number selected per partition. (Answer: \(U=0.75,\ V=0.75,\ W=0.50\); the sum is \(2\).)
- Three partitions rank predictors as \(P,Q,R,S\); \(Q,P,S,R\); and \(P,R,Q,S\). Find the top-two inclusion frequencies and mean rank of each predictor. (Answer: Frequencies \(P=1,\ Q=1,\ R=0,\ S=0\); mean ranks \(P=4/3,\ Q=2,\ R=3,\ S=11/3\).)
- Four top-two sets are \(\{A,B\}\), \(\{A,C\}\), \(\{B,C\}\), and \(\{B,D\}\). Calculate the mean Jaccard similarity over all six distinct pairs of sets. (Answer: \(5/18\approx0.2778\).)
- Across 20 repeated screens, predictor \(G\) is selected 16 times, while at least one member of a related group \(\{G,H,J\}\) is selected in all 20 screens. Give the individual frequency for \(G\), the group representation frequency, and a cautious interpretation. (Answer: \(G=16/20=0.80\); group representation \(=20/20=1.00\); the group is consistently represented, but the screen does not consistently choose \(G\) or establish that any member is causally or predictively valuable.)