Calibrating Stability Thresholds for Feature Selection

Why Calibrate a Stability Threshold?

Stability selection turns repeated feature selection on subsamples into a feature-level frequency, then retains features whose frequencies exceed a chosen cutoff. The cutoff is consequential: a lower value may retain a broad set of candidates, while a higher value may leave only features selected consistently. Neither result is automatically preferable. Calibration means examining how the selected set changes across reasonable cutoffs and relating that change to the intended level of conservatism and, where its assumptions are credible, a false-selection bound.

The frequency threshold is not the penalty parameter used by the base selector. The penalty and its tuning rule determine which features are selected in each subsample; the frequency threshold is applied afterward to the resulting frequencies. Changing the threshold alone does not refit the base models or change those frequencies.

Definition: Suppose $p$ candidate features are considered and a base selection procedure is run on each of $B$ subsamples. Let $I_{bj}=1$ if feature $j$ is selected in subsample $b$, and $I_{bj}=0$ otherwise. The empirical selection frequency of feature $j$ is $$ \hat{\pi}_j=\frac{1}{B}\sum_{b=1}^{B}I_{bj}. $$ For a threshold $\pi_{\mathrm{thr}}$, define the thresholded set as $\hat{S}_{\pi_{\mathrm{thr}}}=\{j:\hat{\pi}_j\geq\pi_{\mathrm{thr}}\}$. This tutorial uses the inclusive rule “at least the threshold.”

The empirical frequency summarizes how often the specified base procedure selected a feature across the specified subsamples. It is not a probability that the feature is truly associated with the outcome. A frequency can depend on the subsampling design, the base procedure and its tuning rule, the data, and the number of subsamples. A high frequency is evidence of repeatability under that setup, not proof of biological relevance or a causal effect.

What Changes When the Threshold Changes?

For fixed frequencies, increasing the threshold cannot add a feature: if $\pi_2>\pi_1$, then $\hat{S}_{\pi_2}\subseteq\hat{S}_{\pi_1}$. The selected-set size is therefore nonincreasing in the threshold. The changes occur at the observed feature frequencies. Between two consecutive distinct frequencies, the selected set is unchanged, so testing many cutoffs in that interval adds no new set.

A useful sensitivity analysis reports more than a single preferred cutoff. It shows a small grid of thresholds, the number of features retained at each, and which features enter or leave. Features well above the cutoffs in the grid are more robust to this particular choice; features near a cutoff are threshold-sensitive. This is a descriptive stability assessment, not an additional inferential guarantee.

Worked Example: How a Threshold Grid Changes the Feature Set

In a hypothetical genomic analysis, a fixed base selector is run on $B=100$ subsamples. Suppose eight features have the following selection counts. Frequencies are counts divided by 100.

Feature	Selections	Frequency
A	96	0.96
B	91	0.91
C	83	0.83
D	76	0.76
E	68	0.68
F	52	0.52
G	31	0.31
H	12	0.12

At $\pi_{\mathrm{thr}}=0.60$, the selected set is $\{A,B,C,D,E\}$, with five features: each of their frequencies is at least 0.60, and the next frequency is 0.52. At 0.75, the set is $\{A,B,C,D\}$, with four features. At 0.90, it is $\{A,B\}$, with two features. The sets are nested as expected. Feature E leaves between the 0.60 and 0.75 cutoffs, while C and D leave before the 0.90 cutoff. This shows which conclusions depend on the chosen threshold; it does not establish which features are true signals.

Using the False-Selection Bound

The stability-selection false-selection bound can provide a quantitative constraint, but it applies under assumptions and is not a universal guarantee. In its standard form, let $q$ be the expected number of features selected by the base procedure on a subsample, let $p$ be the number of candidate features, and let $V$ be the number of selected noise features in the thresholded set. For $\pi_{\mathrm{thr}}>1/2$, the bound is

\mathbb{E}[V]\leq \frac{q^2}{(2\pi_{\mathrm{thr}}-1)p}.

This result relies on the stability-selection assumptions, including exchangeability of noise features and a base selection procedure that is not worse than random guessing in selecting noise features. The definition of $q$, the subsampling scheme, and the threshold must match the setup to which the bound is applied. In particular, $q$ is an expected base-selection count, not the number retained after thresholding. The bound controls an expected count of false selections, not the probability that a particular selected feature is false.

Solving the bound for a target $v_0>0$ gives a minimum cutoff sufficient for the bound to be no greater than that target:

\frac{q^2}{(2\pi_{\mathrm{thr}}-1)p}\leq v_0 \quad\Longrightarrow\quad \pi_{\mathrm{thr}}\geq\frac{1}{2}\left(1+\frac{q^2}{p v_0}\right).

The inequality is useful only if its right-hand side is at most 1; otherwise this bound cannot meet the target for any allowable frequency threshold. Even when it can, a cutoff just satisfying the inequality is not automatically the best scientific choice. The assumptions, observed sensitivity curve, and purpose of the analysis still matter.

Worked Example: Calibrating to a Target Bound

Suppose a hypothetical analysis considers $p=12{,}000$ features and the base procedure selects an average of $q=30$ features per subsample. The analyst wants the bound on the expected number of false selections to be no greater than $v_0=2$. Assume for this calculation that the stability-selection assumptions are credible.

Substituting into the solved inequality gives $$ \pi_{\mathrm{thr}}\geq\frac{1}{2}\left(1+\frac{30^2}{12{,}000\times2}\right) =\frac{1}{2}(1+0.0375)=0.51875. $$ Thus a threshold of 0.51875 or higher makes the stated bound no greater than 2. For comparison, at a threshold of 0.60 the bound is $$ \frac{30^2}{(2\times0.60-1)\times12{,}000} =\frac{900}{2{,}400}=0.375 $$ expected false selections. At 0.90 it is $900/(0.8\times12{,}000)=0.09375$. These are upper bounds under the assumptions, not estimates of the actual false-selection count.

The calculation does not say that 0.51875 is a scientifically optimal cutoff. It says only that this cutoff meets the specified bound in this setup. The analyst should still inspect how many features the empirical threshold retains and whether that set changes materially across nearby cutoffs.

The Base Selection Count Also Matters

A threshold-sensitivity analysis should record the base procedure’s selection count alongside the frequency cutoff. Increasing $q$ can change the frequencies themselves, because more features may be selected within each subsample. It also increases the false-selection bound through $q^2$. Consequently, one cannot generally tune $q$ and the frequency threshold independently while assuming the observed threshold curve stays fixed.

Worked Example: Comparing Base Selection Counts

Consider a hypothetical analysis with $p=10{,}000$ features and a target bound of $v_0=0.5$. First suppose the base procedure has expected selection count $q=20$. The required threshold is $$ \frac{1}{2}\left(1+\frac{20^2}{10{,}000\times0.5}\right) =\frac{1}{2}(1+0.08)=0.54. $$ If the base procedure instead selects an expected $q=40$ features per subsample, the calculation becomes $$ \frac{1}{2}\left(1+\frac{40^2}{10{,}000\times0.5}\right) =\frac{1}{2}(1+0.32)=0.66. $$ Under the bound, the second setup requires a higher cutoff to meet the same target. This does not mean that a higher threshold will undo every consequence of increasing $q$: the frequencies and the resulting selected features may also have changed. The two setups should therefore be analyzed separately, with their own frequency tables and threshold curves.

A Practical Sensitivity Analysis

A disciplined analysis makes the comparison reproducible and interpretable. The base selector, its tuning rule, the subsampling scheme, and the candidate feature set should be held fixed while comparing thresholds. If any of these change, the resulting frequencies describe a different analysis.

Compute and inspect frequencies.
For each feature, report its selection count out of $B$ and its empirical frequency. Confirm that the selection indicator and denominator are defined consistently across subsamples.

Choose informative cutoffs.
Include cutoffs that bracket the region where the selected-set size changes, and include the value suggested by a target bound when that bound is applicable. Avoid reporting a grid so dense that it obscures the meaningful changes.

Summarize set changes.
For each cutoff, report the selected-set size and identify features entering or leaving relative to adjacent cutoffs. Pay particular attention to features close to the cutoff.

State the calibration rationale.
Explain whether the final cutoff was chosen to meet a bound, to retain a set of a manageable size, or for another stated purpose. Do not present the threshold as data-independent if it was chosen after inspecting the results.

Diagnostics, Alternatives, and Limitations

Several checks help distinguish a meaningful sensitivity analysis from a mechanical threshold table. First, verify the nesting property: as the threshold rises, the set size must not increase. A violation indicates an implementation or reporting error. Second, inspect the frequency values around the proposed cutoff. If many features lie very near it, small changes in the cutoff can alter the selected set substantially. Third, report the number of subsamples and the selection counts. With a finite $B$, frequencies change in increments of $1/B$; a cutoff finer than that resolution cannot distinguish additional empirical values.

If subsample results were independent Bernoulli observations for a fixed feature, a rough standard error for its empirical frequency would be $\sqrt{\hat{\pi}_j(1-\hat{\pi}_j)/B}$. This can flag frequencies whose apparent distance from the cutoff is small relative to finite-resampling variation. It is only a rough diagnostic: dependence induced by the resampling design means this expression need not be a valid standard error for the actual analysis. It does not replace the threshold-sensitivity table.

The main alternatives answer different questions. A single penalized fit selects features for one fitted data set and tuning choice; it does not show how selection recurs across subsamples. Cross-validation selects a tuning choice for predictive performance, not a stability threshold or a false-selection guarantee. A bound-based cutoff offers a transparent constraint when its assumptions and inputs are defensible, but can be conservative and does not rank features by biological importance. A sensitivity curve makes the consequences of cutoff choices visible, but does not by itself identify one uniquely correct cutoff.

A common mistake is to reason, “A feature was selected in 80% of subsamples, so it has an 80% probability of being a true gene.” The frequency is the proportion of the specified subsample fits in which the base procedure selected that feature. It is not a posterior probability of truth, and it does not quantify the chance that the feature is causal. Another mistake is to raise the threshold until a preferred number of features remains and then describe the resulting set as guaranteed to be reliable. Set size alone supplies no such guarantee. State the selection process, show the sensitivity to the cutoff, and describe the bound only with its assumptions and meaning.

Key takeaway: Calibrate a stability threshold by examining how the selected set changes across cutoffs, and use a false-selection bound only when its assumptions and inputs are appropriate. A cutoff is a decision rule for a specified analysis, not a probability that selected features are true.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If a feature is selected in 72 of 120 subsamples, what is its empirical selection frequency? Does this mean there is a 60% probability that it is a true signal?
For fixed feature frequencies, can the selected set at threshold 0.85 contain a feature absent from the set at threshold 0.70? Explain.
With $p=5{,}000$, $q=10$, and $\pi_{\mathrm{thr}}=0.75$, calculate the false-selection bound.
Give two reasons why a cutoff that satisfies the bound should not automatically be called the optimal cutoff.

Solutions. (1) The frequency is $72/120=0.60$. It means the feature was selected in 60% of these subsample fits; it is not a 60% probability that the feature is true. (2) No. If a feature has frequency at least 0.85, it also has frequency at least 0.70, so the higher-threshold set is a subset of the lower-threshold set. (3) The bound is $10^2/((2\times0.75-1)\times5{,}000)=100/2{,}500=0.04$ expected false selections, under the required assumptions. (4) The bound may be conservative and depends on assumptions that may not hold; moreover, the selected-set size and membership may change sharply near that cutoff, and the bound does not establish biological importance or a uniquely best scientific choice.