How Sample Size and Signal Strength Affect Stability

Why Selection Stability Changes

A feature-selection procedure does not act on the underlying signal alone. It acts on a finite sample, with a particular noise level, predictor correlation structure, preprocessing pipeline, and penalty-selection rule. Repeating the analysis on perturbed samples can therefore produce different selected sets. The central question here is how three design features—sample size, signal strength, and predictor correlation—affect that variation.

A larger sample often gives the procedure more information about a signal, and a stronger signal is easier to distinguish from noise. Neither fact guarantees that the same named feature will be selected every time. When genomic predictors are correlated, several features may carry nearly interchangeable information. The procedure may repeatedly select something from that group while switching among its members.

Definition: Fix a fitting and selection procedure, including its preprocessing, penalty-selection rule, and selection threshold. For feature $j$, its selection frequency is the proportion of repeated resamples in which that procedure selects $j$. If $B$ resamples are used and $I_{bj}=1$ when feature $j$ is selected in resample $b$, then $$ \hat{\pi}_j=\frac{1}{B}\sum_{b=1}^{B} I_{bj}. $$ A high $\hat{\pi}_j$ describes reproducibility under the specified resampling and fitting procedure; it is not, by itself, evidence that the feature is causal or even truly associated.

This tutorial assumes that resampling units are appropriate for the study design, and that the same candidate features and analysis rules are used across resamples. In genomic data, for example, repeated measurements from one person must not be split across training and resampling units as though they were independent people. The earlier material on selection stability and subsampling frequencies defines the resampling summaries used here; the focus now is how their values respond to controlled changes in information and signal structure.

A Thresholded-Score Derivation

A simple model isolates the effects of sample size and effect size. Suppose a predictor $x$ is standardized so that $\sum_{i=1}^n x_i^2=n$, and the outcome follows $y_i=\beta x_i+\epsilon_i$, where the errors are independent $N(0,\sigma^2)$. Consider a score-based rule that selects the feature when the absolute standardized score exceeds a fixed cutoff $c$. This is a deliberately simplified selection rule, not an exact formula for the selection probability of every penalized regression fit.

The score is $T=x^\mathsf{T}y/(\sigma\sqrt{n})$. Substituting the model gives a mean contribution $\beta x^\mathsf{T}x/(\sigma\sqrt{n})=\beta\sqrt{n}/\sigma$, while the standardized error contribution has variance one. Thus

T\sim N(\delta,1),\qquad \delta=\frac{\beta\sqrt{n}}{\sigma}.

The rule selects when $T>c$ or $T<-c$. Therefore its selection probability is

P(|T|>c)=1-\Phi(c-\delta)+\Phi(-c-\delta),

where $\Phi$ is the standard normal cumulative distribution function. Holding $\beta$, $\sigma$, and $c$ fixed, increasing $n$ increases $\delta$ in magnitude. For a positive signal, that usually moves more of the score distribution beyond the positive selection cutoff. Holding $n$ and $\sigma$ fixed, increasing a positive $\beta$ has the same directional effect. This justification depends on the stated model, a fixed cutoff, and a known noise scale; a cross-validated penalty in a correlated, high-dimensional design does not generally have this exact probability formula.

Worked Example: Increasing Sample Size for a Fixed Signal

Suppose the standardized-score rule uses $c=1.5$, with $\sigma=1$ and a positive effect $\beta=0.30$. Compare $n=25$ with $n=100$. Assume independent Gaussian errors and the standardized predictor condition above.

At $n=25$, $\delta=0.30\sqrt{25}/1=1.5$. The selection probability is $1-\Phi(1.5-1.5)+\Phi(-1.5-1.5)=1-\Phi(0)+\Phi(-3)$. Using $\Phi(0)=0.5000$ and $\Phi(-3)=0.00135$, this is $0.5014$.

At $n=100$, $\delta=0.30\sqrt{100}=3.0$. The probability is $1-\Phi(1.5-3.0)+\Phi(-1.5-3.0)=1-\Phi(-1.5)+\Phi(-4.5)$, which is approximately $0.9332+0.000003=0.9332$.

Under this specified rule and model, the stronger information in the larger sample raises the feature’s selection probability from about 50.1% to 93.3%. These are model-based probabilities, not guaranteed frequencies for a particular real data set or for a penalized model using a data-selected penalty.

Worked Example: A Weaker Signal at the Same Sample Size

Keep $n=25$, $\sigma=1$, and $c=1.5$, but reduce the positive effect from $\beta=0.60$ to $\beta=0.15$. Assume the same Gaussian model and standardized predictor.

For $\beta=0.60$, $\delta=0.60\sqrt{25}=3.0$, so the calculation from the first example gives $P(|T|>1.5)=1-\Phi(-1.5)+\Phi(-4.5)\approx0.9332$.

For $\beta=0.15$, $\delta=0.15\sqrt{25}=0.75$. The selection probability becomes $1-\Phi(1.5-0.75)+\Phi(-1.5-0.75)=1-\Phi(0.75)+\Phi(-2.25)$. Using $\Phi(0.75)\approx0.7734$ and $\Phi(-2.25)\approx0.0122$, the result is $0.2266+0.0122=0.2388$.

The weaker signal is selected in only about 23.9% of repeated samples under this rule, compared with 93.3% for the stronger signal. This illustrates why weak effects can have low selection frequency even when they are real under the model. It does not establish that any particular low-frequency genomic feature is truly associated.

Correlation and the Identity of the Selected Feature

Predictor correlation changes the interpretation of stability. If two standardized predictors contain similar information, small sample perturbations can change which one appears more useful to the fitted procedure. A useful two-predictor calculation makes this loss of precision visible. In a linear model with two standardized predictors having correlation $\rho$, the predictor cross-product matrix, after scaling by $n$, is

\begin{pmatrix}1&\rho\\ \rho&1\end{pmatrix}.

When this matrix is invertible, its inverse has diagonal entries $1/(1-\rho^2)$. Under independent errors with variance $\sigma^2$, the variance of either unpenalized coefficient estimate is therefore $\sigma^2/[n(1-\rho^2)]$. This result concerns the two-predictor linear model with $n$ sufficient for the fit; it is not a variance formula for a lasso estimator when $p$ is much larger than $n$. It does show why distinguishing two highly correlated predictors is intrinsically difficult: as $|\rho|$ approaches one, the information for separating their individual coefficients diminishes.

Consequently, feature-level stability and group-level stability can differ. If a correlated group contains two plausible substitutes, the selection frequency of either member may be moderate, while the frequency that at least one member is selected is high. These quantities answer different questions: “Is this exact feature reproducibly selected?” versus “Is this signal-bearing group reproducibly represented?”

Worked Example: Stable Group, Unstable Members

Suppose two highly correlated genomic features, A and B, are treated as alternatives by a fixed selection procedure. In 100 subsamples, the procedure selects A in 54, selects B in 46, and never selects both. Assume every subsample follows the same fitting and selection rules.

The feature-level frequencies are $\hat{\pi}_A=54/100=0.54$ and $\hat{\pi}_B=46/100=0.46$. Neither named feature has a high frequency by itself. Define the group event as selecting at least one of A or B. It occurs in $54+46=100$ subsamples, so the group frequency is $100/100=1.00$.

The correct conclusion is that the procedure consistently selects one member of this pair, but does not consistently identify which member. It would be incorrect to call both individual features perfectly stable, or to infer that both have independent effects. The counts describe the behavior of this procedure on these resamples; the correlation and data-generating mechanism are needed to interpret why the substitutions occur.

Reading Controlled Comparisons

To attribute a stability difference to sample size, signal strength, or correlation, compare analyses that hold the other ingredients fixed. That includes the candidate feature set, normalization and filtering, resampling scheme, fitting algorithm, penalty-selection rule, and frequency threshold. If the penalty is retuned within each resample, the resulting frequency includes variation from both the resampled data and the tuning step. If the penalty is held fixed, the comparison answers a different, narrower question.

The following hypothetical results illustrate a controlled comparison. Each frequency is calculated from 100 subsamples using the same selection pipeline. “Group frequency” is the proportion of subsamples selecting at least one feature in a two-feature group. The entries are illustrative, not universal expected values.

Sample size	Signal effect	Within-pair correlation	Target feature frequency	Alternative feature frequency	Group frequency
80	0.25	0.10	0.38	0.09	0.43
160	0.25	0.10	0.68	0.06	0.71
80	0.50	0.10	0.79	0.08	0.83
80	0.50	0.90	0.49	0.44	0.78

The first two rows compare sample size at fixed effect and low correlation: the target frequency rises from 0.38 to 0.68. The first and third compare effect size at fixed sample size and correlation: the target frequency rises from 0.38 to 0.79. The last two compare correlation at fixed sample size and effect: the target’s frequency falls from 0.79 to 0.49, while the group frequency changes less, from 0.83 to 0.78. The intended lesson is not that correlation always lowers a particular feature’s frequency by a fixed amount, but that individual identity can become harder to reproduce even when the group remains detectable.

Diagnostics, Alternatives, and Limitations

A stability comparison is only informative if the resampling procedure reflects the intended source of variation. Inspect the following before interpreting frequencies:

Resampling units: resample independent biological units, not correlated measurements from the same unit as if they were independent.
Pipeline consistency: check that filtering, normalization, feature coding, and penalty tuning are applied consistently. Data-dependent preprocessing performed using information from outside a resample can make the apparent stability misleading.
Selection rule: record whether penalty values are reselected or fixed in each resample, and how ties or near-ties are handled.
Correlation structure: inspect whether low individual frequencies occur among strongly correlated features. Report group-level stability when scientifically defensible groups are specified.
Monte Carlo precision: a frequency estimated from a finite number of resamples has simulation error. For example, if the true resampling probability is near 0.50, the standard deviation of the estimated frequency from $B$ independent resamples is approximately $\sqrt{0.25/B}$; with $B=100$, this is 0.05.

Selection stability is not the same as predictive performance. Cross-validation addresses prediction error under a specified validation design; a model can predict consistently while selecting different correlated features. Stability selection adds a formalized resampling-and-thresholding approach, but its guarantees depend on its assumptions and calibration, and its false-selection bound is not a universal correction for every correlated genomic design. A coefficient-sign summary can provide another view, but stable sign is not equivalent to stable selection or causal effect estimation.

Several limitations matter. More observations do not ensure higher frequency if the analysis pipeline changes with $n$, if additional features enter the candidate set, or if the resampling scheme changes. A larger effect can still be hard to select when noise is high or the penalty is aggressive. Correlation does not always make every feature less stable: if one feature has a sufficiently stronger unique signal, it may remain favored. Finally, selection frequency depends on the chosen procedure and threshold. It is a reproducibility summary, not a property of a feature independent of the analysis.

Common Mistake

Common mistake: “The correlated group is selected in 95% of resamples, so each feature in it is a stable finding.” This reasoning confuses a group event with its members. If the procedure alternates between correlated features, each individual frequency may be much lower than the group frequency. Report the individual frequencies and the group definition separately, and do not treat selection stability alone as evidence of causality.

Fix the estimand of stability.
Decide whether the target is the reproducibility of a named feature, a prespecified correlated group, or the selected model as a whole.

Hold the pipeline interpretable.
Specify the resampling units, preprocessing, fitting and tuning rules, and selection threshold before comparing conditions.

Compare controlled conditions.
Change sample size, effect strength, or correlation while keeping the other conditions fixed, and report the resulting feature- and group-level frequencies.

Qualify the conclusion.
Describe stability as reproducibility under the stated procedure, and consider prediction and scientific interpretation separately.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the thresholded-score model, what happens to $\delta$ when $n$ is multiplied by four while $\beta$ and $\sigma$ stay fixed?
A pair of correlated features has frequencies 0.42 and 0.48, and the group frequency is 0.90. What is the appropriate interpretation?
Why is it not valid to interpret a selection frequency of 0.80 as an 80% probability that a feature is causal?
Name two analysis choices that should be held fixed when comparing stability across sample sizes.

Solutions. (1) Since $\delta=\beta\sqrt{n}/\sigma$, multiplying $n$ by four doubles $\delta$. (2) The procedure often selects at least one member of the group, but neither individual feature is selected in most resamples; the group is more reproducible than its membership. (3) The frequency is conditional on the data perturbations and specified selection procedure, not a probability of causality. (4) Examples include the candidate feature set, preprocessing, resampling scheme, penalty-selection rule, and selection threshold.