Why Diagnose Stability?
A factor may look biologically compelling in one fitted multi-omics model yet change substantially when the optimization is restarted or the subjects are resampled. Before interpreting a loading profile as a reproducible cross-omics pattern, check whether that profile—and the subject scores associated with it—survives reasonable changes to the data and fitting procedure. Stability is evidence about reproducibility, not proof that a factor is biologically real or causal.
This worked tutorial uses two kinds of comparison. Repeated fits to the same data, often with different initial values, help reveal sensitivity to the fitting algorithm. Bootstrap fits, which resample subjects and refit the model, show how estimates vary under subject-level sampling. These sources of variation answer different questions and should not be conflated. The earlier material on resampling-based factor stability provides the broader context; here the emphasis is on diagnosing concrete fitted solutions.
Set Up the Comparison
Suppose \(M\) matched omics blocks are measured on the same \(n\) subjects. Write \(X^{(m)}\) for the data matrix for block \(m\), with subjects in rows and its \(p_m\) features in columns. A \(K\)-factor fit represents the data using subject scores \(F\), an \(n \times K\) matrix, and loading matrices \(\Lambda^{(m)}\), each of dimension \(p_m \times K\). For example, a common low-rank part of block \(m\) is \(F(\Lambda^{(m)})^T\). The exact model may include block-specific terms and residuals; stability comparisons must use the same model definition and comparable preprocessing in every fit.
Factor labels are not intrinsically ordered, and a factor can be multiplied by \(-1\) if its scores are also multiplied by \(-1\), without changing the fitted low-rank product. Thus, direct column-by-column comparisons can report large differences even when two fits represent essentially the same solution. Match factor columns across fits and resolve their signs before comparing estimates. A practical matching criterion is the absolute correlation between factor score vectors on common subjects; loading-profile similarities can also be used or incorporated when scores alone do not give a clear match. Choose a one-to-one assignment that maximizes total similarity, rather than matching each factor independently and potentially assigning the same factor twice.
After matching, choose each factor’s sign so that its score correlation with the reference factor is positive. Apply the same sign change to its loading vectors: changing the score sign without changing the loadings would change the fitted product. Only compare loading correlations when features and their scales are the same across fits. If feature sets differ, restrict comparison to common features or use a specifically defined feature-set stability measure; do not treat missing features as zero loadings.
Worked Examples
Worked Example: Aligning Repeated Fits Before Comparing Loadings
A hypothetical joint analysis of standardized expression and protein measurements fits \(K=2\) factors twice to the same subjects, using different starting values. The absolute correlations between reference-fit scores and candidate-fit scores are
The rows are reference factors 1 and 2; the columns are candidate factors 1 and 2. The best one-to-one match is reference factor 1 to candidate factor 1 and reference factor 2 to candidate factor 2, with total similarity \(0.91+0.88=1.79\). The raw signed score correlations are \(0.91\) and \(-0.88\). Therefore, keep candidate factor 1’s sign and reverse both candidate factor 2’s scores and its loading vectors.
Suppose the correlations between the matched, sign-aligned loading profiles, concatenated over the same standardized features in both omics blocks, are \(0.86\) for factor 1 and \(0.83\) for factor 2. These indicate reasonably similar profiles in these two fits, although they do not show that every feature is stable. The correct conclusion is limited: these two repeated fits produce similarly matched factors overall. Feature-level bootstrap checks or additional starts are needed before making stronger claims about reproducibility.
Worked Example: Flagging an Unstable Loading
In a hypothetical bootstrap analysis, all replicates have been matched to the same reference factor and sign-aligned. Consider two standardized feature loadings, each recorded in eight bootstrap replicates. For feature A the loading values are \(0.42, 0.38, 0.45, 0.40, 0.35, 0.44, 0.41, 0.39\). The mean is
The sum of squared deviations from \(0.405\) is \(0.0074\), so the sample standard deviation is
All eight estimates are positive and lie between \(0.35\) and \(0.45\). In this hypothetical set, feature A has a consistently positive loading with modest variation relative to its mean. This is useful evidence of loading stability under the specified bootstrap procedure, not a guarantee that the feature is important or that the factor has a causal interpretation.
For feature B the aligned loadings are \(0.03, -0.02, 0.08, 0.01, -0.07, 0.06, -0.01, 0.02\). Their mean is \(0.10/8=0.0125\). The values vary on both sides of zero, and the mean is small compared with their spread. Its profile contribution is therefore not reliably positive in these replicates. Reporting feature B as a firm positive marker would overstate the evidence. The diagnosis concerns this feature’s loading; it does not, by itself, establish that the entire factor is unstable.
Worked Example: Finding a Subject Score Sensitive to Fitting
A hypothetical factor is fit four times to the same data, with different starting values. After matching and sign alignment, subject A’s score is \(1.2, 1.0, 1.1, 1.3\), while subject B’s score is \(0.8, -0.4, 1.0, -0.2\). For subject A, the mean score is \(4.6/4=1.15\). Its squared deviations sum to \(0.05\), giving
Subject B’s mean is \(1.2/4=0.30\). Its squared deviations from \(0.30\) sum to \(1.48\), so
Both standard deviations are in factor-score units, but subject B’s estimates also change sign across fits. This points to substantial subject-level sensitivity to fitting, despite the factors having been aligned. It is a reason not to give subject B a firm placement on the factor based on this fit alone. Because the data were not resampled, these numbers diagnose sensitivity to the fitting procedure, not population sampling uncertainty.
What the Stability Summaries Establish
For a loading vector with matched features, the Pearson correlation across feature entries summarizes profile similarity, while the proportion of bootstrap replicates in which a loading retains its aligned sign summarizes sign consistency. If interpretation focuses on the largest loadings, a top-feature overlap can be more informative. For sets \(A\) and \(B\) of selected features, the Jaccard overlap is \(|A\cap B|/|A\cup B|\); it is \(1\) for identical nonempty sets and \(0\) for disjoint sets. These summaries answer different questions: correlation reflects overall profile shape, sign consistency concerns direction, and overlap concerns selection of prominent features.
For subject scores, summarize the distribution across fits or bootstrap replicates for each subject, for example with a standard deviation, range, or bootstrap percentile interval. A narrow range suggests the subject’s score is insensitive to the particular perturbation examined; frequent changes in sign or ordering relative to other subjects indicate a less secure placement. Bootstrap intervals describe variation under the resampling and fitting procedure. They are not automatically confidence intervals with exact coverage, and an interval for an individual score is not a probability statement about the parameter.
Assumptions, Diagnostics, and Failure Modes
A meaningful comparison requires more than the same number of factors. The fits should use the same subjects or a clearly defined set of common subjects for repeated-fit score correlations; comparable feature definitions and scaling for loading comparisons; and the same factor count, model specification, and fitting target. Bootstrap resampling should resample matched subjects as units so that the measurements from different omics blocks remain paired. If samples are clustered, repeated measures, or otherwise dependent, resampling individual rows as if they were independent is inappropriate; the resampling scheme must respect the sampling design.
Check matching diagnostics before interpreting stability. A factor with two nearly equal candidate matches may have an ambiguous identity, especially if its scores or loading profile are similar to another factor. A low match similarity can mean that the factor changed, disappeared, or was split or combined; it can also mean the chosen matching criterion is inadequate. Do not conceal this ambiguity by forcing a match and then reporting a single loading correlation. Inspect score and loading similarities together, and report unmatched or ambiguous factors explicitly.
Other warning signs include wide bootstrap distributions, frequent aligned sign reversals for particular loadings, changing top-feature sets, and factor scores that vary substantially for a subset of subjects. Check whether instability is concentrated in a block, correlated with missingness or influential subjects, or caused by a handful of weakly measured features. Residual and predictive diagnostics address model fit, not factor reproducibility; a model can fit adequately while its individual factor interpretations remain unstable.
Comparison and Limits
| Approach | What it probes | Main caution |
|---|---|---|
| Repeated fits with different starts | Dependence on initialization or optimization | Does not measure subject-sampling uncertainty |
| Subject-level bootstrap refits | Variation under resampling subjects and refitting | Depends on the sampling design, model, and bootstrap procedure |
| Feature-set perturbations | Sensitivity to included features | Changes the measured feature space; compare only appropriately defined profiles |
No one summary is a universal stability test. In particular, a high profile correlation can hide changes in a few scientifically important features, while a low correlation can be driven by many small, unimportant entries. Conversely, sign consistency for one loading does not establish that the factor is stable as a whole. State the comparison, alignment method, feature scale, and summaries used, then describe the specific evidence rather than declaring a factor simply “stable.”
A common mistake is to compare raw factor columns and conclude that a factor is unstable because its loading signs or column number changed. That reasoning ignores sign and permutation ambiguity: equivalent solutions can differ in those ways without changing their fitted values. The correction is to match factors, align signs consistently for scores and loadings, and only then assess similarity. The opposite mistake is to treat a high matched correlation as proof of biological validity. Alignment removes representational differences; it does not remove uncertainty, establish causality, or validate the scientific interpretation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two fits have signed score correlations \(0.87\) and \(-0.84\) for their matched factors. What sign adjustment is required for the second factor before comparing loadings?
- Why can repeated fits with different starts not, by themselves, quantify subject-sampling uncertainty?
- A loading changes sign in many aligned bootstrap replicates and has a mean close to zero. What is the cautious interpretation?
- Why should matched omics measurements be resampled by subject rather than by individual feature values?
- Can a high correlation between two aligned loading profiles prove that the factor is biologically valid? Explain.
Solutions. (1) Multiply both the candidate factor’s scores and its loading vectors by \(-1\); this makes the score correlation positive and preserves the fitted product. (2) The data remain unchanged across those fits, so the comparison probes fitting sensitivity, not variability from drawing different subjects. (3) The loading’s direction and magnitude are not consistently supported by these bootstrap fits; avoid describing it as a firm positive or negative feature contribution. (4) Resampling subjects preserves the within-subject pairing across omics blocks; resampling features independently would break that matched data structure. (5) No. High correlation supports similarity of the aligned profiles under the comparison, but does not establish biological validity, causality, or stability under other perturbations.