Stability Of Subtype-Defining Features

Why Feature Stability Requires Its Own Assessment

A subtype can recur across resamples while the molecular features used to describe it change substantially. Conversely, a small set of features may be repeatedly selected even when the corresponding subgroup is not consistently recovered. These are different questions: subgroup stability concerns whether similar samples form a group again; feature stability concerns whether the molecular characterization of that group is reproducible.

This tutorial focuses on tracking selected features and their contributions across resampling runs. A feature may be a gene, protein, metabolite, or other measured variable. The aim is not to claim that a repeatedly selected feature causes subtype membership. Rather, these summaries describe how consistently the analysis pipeline uses that feature to characterize a recurring subgroup.

Set Up Comparable Runs Before Summarizing Features

Feature summaries are meaningful only when each run refers to a comparable target. First identify the subgroup of interest across runs using the matching or subgroup-stability procedures already established. The label “cluster 2” is not sufficient: labels can permute between runs. Define a matching rule, such as overlap with a reference subgroup, and apply it consistently. If a run does not recover a subgroup meeting the prespecified matching rule, mark it as unsuccessful for that target rather than treating an unrelated cluster as a match.

The feature-analysis pipeline must also be specified. Record which preprocessing, scaling, feature filtering, selection, and model-fitting steps are repeated within each resample. If a step is part of the intended discovery pipeline, estimating it once on all samples and then resampling only the final model can understate instability. Conversely, different feature definitions or measurement platforms across runs prevent direct feature-by-feature comparison unless they are mapped to a common set.

For feature \(j\), let \(I_{jr}=1\) if it is selected in eligible run \(r\), and \(I_{jr}=0\) otherwise. Let \(R^\ast\) be the number of eligible runs in which the target subgroup was successfully matched and feature selection was performed. Then the empirical selection frequency is:

This frequency is conditional on a matched, usable run. Report both \(R^\ast\) and the number of attempted runs: otherwise, a high frequency among a small, unusually easy subset of successful runs may appear more reassuring than it is. Selection frequencies are descriptive stability summaries, not automatically formal confidence levels. Resampling runs often overlap in their samples, so treating their selection indicators as independent Bernoulli observations for a conventional standard error or binomial test is generally unjustified.

Direction and Contribution Are Different From Selection

A selected feature can contribute in opposite directions across runs. For a subtype defined partly by a contrast against other samples, record a signed quantity such as a standardized difference, regression coefficient, or signed loading, using a consistent orientation. A positive value must have the same interpretation in every run. This requires fixing which side of the contrast is the target subtype and, for latent components, aligning sign orientation: multiplying all loadings of a component by \(-1\) leaves the component representation equivalent but reverses every sign.

For a signed feature summary \(\theta_{jr}\), define direction consistency among runs where feature \(j\) is selected as the larger of the proportion with positive values and the proportion with negative values. Also report which direction dominates. This summary deliberately does not treat near-zero estimates as meaningful directions; a prespecified tolerance can classify values with \(|\theta_{jr}|\) below that tolerance as effectively zero. Keep selection frequency and directional consistency separate: a feature selected consistently but with changing signs has a different interpretation from one selected in fewer runs but always in the same direction.

For loading or contribution vectors, first place runs in a common feature order and common scale convention. If \(\mathbf{v}_r\) is a vector of contributions for one matched subtype, its cosine similarity with a reference vector \(\mathbf{v}_0\) is \((\mathbf{v}_r^\mathsf{T}\mathbf{v}_0)/(\|\mathbf{v}_r\|\|\mathbf{v}_0\|)\). Values near 1 indicate similar orientation; values near 0 indicate weak alignment. If component sign is arbitrary, align it to the reference before interpreting the signed cosine. A high cosine measures pattern similarity, not equality of magnitudes. Compare magnitudes separately when the model’s scale is comparable across runs.

Compare Contribution Patterns After Alignment

An individual loading can fluctuate even when the overall pattern is similar, especially when several features are correlated. A vector comparison summarizes whether the multivariate contribution pattern recurs. For example, with two standardized features, a reference contribution vector \(\mathbf{v}_0=(0.8,0.6)\) has norm 1. Suppose a resampled run gives \(\mathbf{v}_r=(0.6,0.8)\), also with norm 1. The cosine similarity is:

The pattern is closely aligned, although the relative contributions of the two features have shifted. If the run instead gives \((-0.6,-0.8)\), its unaligned cosine is \(-0.960\). When component sign is arbitrary, flipping the run’s entire vector yields \((0.6,0.8)\), giving an aligned cosine of \(0.960\). But if the signs encode a prespecified subtype-versus-reference contrast, reversing all signs is not a harmless relabeling: it indicates the opposite contrast and must be reported as such.

Cosine similarity is inappropriate when the feature vectors are not comparable, such as when features were measured on incompatible scales or were selected from different feature spaces without harmonization. In those situations, report selection overlap on the common feature set, or explicitly state that vector comparison is unavailable. Do not fill absent features with zero unless zero has a defensible interpretation in the fitted representation; an unselected feature and a measured zero contribution are not necessarily the same.

A Practical Summary Workflow

Diagnostics, Alternatives, and Limitations

A single stability statistic rarely captures the full feature characterization. Selection frequency is easy to interpret, but it discards effect magnitude and direction. Signed effects reveal direction and scale, but can be unstable when estimates are noisy or close to zero. Rank recurrence can help when absolute loading scales vary, but may obscure meaningful changes in relative contribution. Vector similarities summarize a joint pattern, but can conceal one highly unstable feature within an otherwise consistent vector. Use complementary summaries rather than choosing one number as a universal stability score.

Useful diagnostics include the distribution of selection counts, the fraction of runs with no matched subgroup, sign reversals, large changes in feature ranks, and whether a few resamples dominate the result. Compare feature summaries under reasonable resampling schemes and preprocessing choices, while keeping the target definition fixed. If the target subgroup is matched poorly or the feature universe changes across runs, feature instability may reflect those design problems rather than the feature-selection step alone.

Resampling stability is conditional on the observed data, the chosen pipeline, and the resampling design. It is not independent replication, a guarantee of performance in a new cohort, or proof that a feature is biologically specific. Correlated features may substitute for one another: low individual selection frequencies can coexist with a stable group of interchangeable markers. In that case, report stability at both the individual-feature and feature-set levels, with the set definition made explicit.

Common Pitfall

Wrong reasoning: “This gene was selected in 90% of runs, so it is a strong, directionally consistent subtype marker.” The frequency alone supports only that the selection procedure retained the gene often in the eligible runs. It says nothing about whether its effect was large, whether the sign was consistent, how many runs were excluded, or whether a correlated gene could have replaced it.

Correction: Report the selection frequency with its denominator and subgroup-recovery rate, then separately summarize signed effects or aligned contributions. Treat these as reproducibility descriptions of the fitted pipeline, not as causal evidence or formal validation.