Assessing NMF Component Stability

Why Component Stability Needs More Than One Check

An NMF fit can describe the observed data well and still be hard to reproduce. Small changes in the subjects included, or in the algorithm’s random starting values, may yield different component profiles or assign subjects different component scores. Stability assessment asks whether the fitted patterns persist under specified perturbations. It is a reproducibility check, not a test that a component is biologically true.

The earlier material on Scale and Permutation Indeterminacy in NMF established that component labels and scales are not intrinsically fixed. Consequently, comparing component 1 from one fit with component 1 from another fit before resolving these indeterminacies can produce misleading instability. This tutorial sets out a practical comparison: match components, then summarize feature-profile reproducibility separately from sample-score reproducibility.

For multiple omics blocks, the same logic applies when a model specifies shared subject scores and block-specific feature profiles. Comparisons must use the same feature definitions, preprocessing, block weighting, and component interpretation across fits. If a feature is absent from a resampled fit or the block scales change, a raw similarity value may not be comparable across runs.

Designing the Stability Assessment

There are two useful perturbations. Repeated initializations fit the same subjects and features using different starting values, with the same rank, objective, constraints, and stopping rules. This checks sensitivity to algorithmic starts. Subject resampling refits the model on subsets or bootstrap samples of subjects, keeping the feature space and modeling choices fixed. This checks whether the inferred patterns depend strongly on which subjects are observed.

The comparison requires a common basis. Use a fixed rank \(K\), identical feature preprocessing and objective, and a prespecified resampling scheme. For subject resamples, compare profiles across all features in the common feature space. Compare scores only for subjects present in both fits, or score a common set of subjects using a consistent projection procedure. If bootstrapping duplicates subjects, retain subject identities so that matched observations are handled explicitly rather than treated as new, unrelated people.

Before measuring similarity, put profiles on a common scale. For nonzero nonnegative profile row \(h_k\), one option is \(h_k/\lVert h_k\rVert_2\), where \(\lVert h_k\rVert_2\) is its Euclidean norm. This removes arbitrary positive scaling, leaving the relative feature pattern. For scores, Pearson correlation is useful when the question concerns relative ordering across subjects and score magnitude or offset may differ. Cosine similarity is another choice when direction through the origin is the relevant comparison. The choice must be stated because these quantities answer different questions.

Components are generally permuted between fits. Construct a \(K \times K\) similarity matrix between the reference fit's components and those of the new fit. Pair components so that the total similarity is as large as possible, using the profile similarity as the matching criterion when profiles are the primary object of interest. Each component should be used in at most one pair. If the task is only a stability summary, it is enough to state this one-to-one matching rule; detailed matching algorithms are a separate topic. After matching, report both profile and score similarities. A good profile match does not guarantee stable subject scores.

Summarizing Profiles and Scores Separately

After matching, compute a profile similarity for each matched pair and a score similarity for each pair. Across \(R\) refits, report the distribution of these values, such as the median and range or interquartile range. Also report how often each reference component could be matched with adequate similarity. A single average can conceal a component that is stable in most fits but fails in a few.

For profile comparisons, cosine similarity focuses on the relative feature pattern and is invariant to positive rescaling. For score comparisons, Pearson correlation focuses on whether subjects tend to be high or low on a component in a similar way. Use the same subjects in each paired score comparison, and state how many were shared. If a component is absent, nearly zero, or unmatched in some fits, do not silently omit those failures: report the rate of such events and the rule used to identify them.

Why the Comparison Works—and What It Cannot Establish

The normalization and similarity calculations have a direct justification. If a profile is multiplied by a positive constant \(c\), then its cosine with another profile is unchanged:

Thus the cosine comparison is appropriate for profile shape despite the positive scale indeterminacy of NMF. It does not resolve permutation by itself, which is why matching is performed first. Nor does it make unlike features comparable: feature order, transformation, and block weighting must already be consistent.

Resampling stability depends on the resampling design. A small subset can omit a rare but important subject pattern; a bootstrap can include repeated copies of some subjects. Repeated initializations address a different source of variation and do not show robustness to changes in the sample. Running many starts on one data set and reporting only the best objective can hide the fact that other good fits have different components. Where possible, report stability across starts as well as across subject resamples.

Diagnostics include the distribution of matched similarities, the number of failed or ambiguous matches, and whether conclusions change under reasonable choices of similarity metric or resampling fraction. Inspect component profile plots and score scatterplots for pairs with low similarity. Check that near-zero components are not being treated as stable simply because they match another near-zero component, and that preprocessing or block-weight changes have not caused artificial disagreement. There is no universal cutoff at which a component becomes stable; thresholds depend on purpose and must be prespecified or interpreted descriptively.

Stability also differs from fit quality, rank selection, and external validation. A consistently reproduced component can reflect a systematic artifact, an imposed modeling constraint, or a broad pattern that is not specific to the scientific question. Conversely, a real but rare pattern may be unstable under ordinary resampling. Stability is most informative alongside residual diagnostics, the objective value, scientific interpretation, and independent validation when available.

Common Mistakes and Practical Alternatives

• Comparing component labels directly. Component 1 in one fit need not correspond to component 1 in another. Match components before calculating paired similarities.
• Using raw Euclidean distance without scale control. A pure rescaling can make identical profile shapes look far apart. Normalize profiles or use a scale-invariant measure such as cosine similarity.
• Reporting one stability number. Profiles and scores answer different questions. Give separate summaries and specify the subjects used for score comparisons.
• Equating stability with validity. Reproducibility across the chosen perturbations does not establish causality, biological function, or correct rank.

Alternative summaries include cosine similarity for profiles, correlation for scores, or distances after a clearly specified normalization. A consensus matrix can summarize whether pairs of subjects are repeatedly grouped together, but it answers a clustering-oriented question and does not replace profile-level comparison. A single refit on the full data is also not a stability assessment: it provides no distribution of results under perturbation. Choose measures based on the component property of interest, preserve the distinction between perturbation types, and report enough detail for the comparison to be reproduced.