Diagnosing Confounded Latent Directions

Why Diagnose Latent Directions?

A fitted multi-omics factor model can reproduce the observed data well while leaving the meaning of its individual factors uncertain. Two directions may describe nearly the same feature pattern, account for similar amounts of fitted variation, or change substantially when the model is fit from a different initialization. These patterns suggest that the data do not clearly distinguish those directions. They do not, by themselves, show that the model is wrong or that a particular biological interpretation is false.

The practical question is therefore not simply whether a factor solution can be computed. It is whether its individual directions are sufficiently separated to support stable interpretation. This tutorial develops three complementary checks: correlations between loading vectors, similarity in factor contributions, and sensitivity across initializations. The earlier material on rotational invariance, scale and sign constraints, and identifiability provides important context: some differences between solutions reflect conventions or equivalent representations, not substantive instability.

Model Setup and Assumptions

For clarity, write the fitted component for view \(v\) as \(F W^{(v)T}\), where \(F\) is the matrix of sample scores shared across views and \(W^{(v)}\) is the loading matrix for view \(v\). For factor \(k\), write its score vector as \(f_k\) and its loading vector in view \(v\) as \(w_k^{(v)}\). A fitted value for sample \(i\) and feature \(j\) in view \(v\) is the sum of the corresponding factor products, plus any residual or other model terms specified by the model.

The diagnostics below are most directly interpretable when the same features, preprocessing, model specification, and scale constraints are used for all fits being compared. To interpret squared norms as contributions, scores and loadings must also be on a consistent scale. If scores or loadings are rescaled between fits, the individual norms can change even when their product—and thus the fitted component—does not. Likewise, correlations should be calculated over the same feature set within a view, or over a deliberately constructed and comparably scaled feature set. A correlation across incomparable feature scales can be misleading.

Loading Correlations and Factor Contributions

Within a view, compare two factors’ loading vectors across the same features. The Pearson correlation measures whether their deviations from their respective feature-average loadings have similar patterns. For vectors \(a\) and \(b\) of length \(p\), define \(\bar a=p^{-1}\sum_{j=1}^{p}a_j\) and \(\bar b=p^{-1}\sum_{j=1}^{p}b_j\). If both vectors have nonzero variance, their Pearson correlation is:

A large absolute correlation indicates that the two loading profiles vary similarly across the chosen features, allowing for opposite signs. Sign matters: the sign of a factor can be reversed without changing its fitted component if both its score and loading signs are reversed. Thus, when diagnosing similarity of directions, \(|r|\) is often more informative than \(r\) alone. Pearson correlation is not the same as cosine similarity: cosine similarity does not subtract the vector means. State which measure is used, especially if loading profiles have nonzero means. If a vector has no variation across features, its Pearson correlation is undefined.

A second check compares how much fitted variation each factor contributes. Under the specific condition that the columns of \(F\) are mutually orthogonal, factor \(k\)'s contribution to the fitted sum of squares across all views can be defined as:

Here \(\|f_k\|^2=\sum_i f_{ik}^2\), and \(\|w_k^{(v)}\|^2\) is the sum of squared loadings for factor \(k\) in view \(v\). With orthogonal score columns, the cross-products between fitted factor components vanish in the total squared norm, so the fitted sum of squares is the sum of the \(C_k\). If the score columns are not orthogonal, cross-terms can contribute; the formula remains a component norm, but the components do not generally form an additive partition of total fitted sum of squares. Contributions also depend on scale conventions and preprocessing. Similar contributions alone do not establish that two factors are confounded.

Why Similar Contributions Can Signal Weak Separation

When a model’s fitted variation can be reassigned between two directions with little change to fit, separate interpretations may be fragile. A short calculation illustrates the role of separation in a simple orthogonal variance decomposition. Suppose two orthogonal directions have contributions \(\lambda_1\) and \(\lambda_2\), and rotate the pair by angle \(\theta\). The contribution assigned to the first rotated direction is:

Using \(\cos^2\theta=(1+\cos 2\theta)/2\) and \(\sin^2\theta=(1-\cos 2\theta)/2\), this becomes:

The dependence on orientation is proportional to the gap \(\lambda_1-\lambda_2\). If \(\lambda_1=\lambda_2\), then \(q_1(\theta)\) is constant for every \(\theta\): the contribution alone cannot distinguish orientations in the pair. If the contributions are close, orientation changes can alter the allocation only slightly. This calculation is a justification for treating near-equal contributions as a warning sign in settings where the model permits such rotations; it is not a universal theorem that any two similarly sized factors are confounded. Other model constraints and differences in their loading patterns can still distinguish them.

Sensitivity to Initialization

Many fitting procedures use starting values. If different initializations yield different individual factors, first account for sign and ordering: factor labels can be permuted, and signs can be reversed under sign symmetry. Compare fits only after matching factors, for example by finding the assignment that maximizes total absolute loading correlation over shared features. Then inspect the matched correlations, contributions, and fitted reconstructions. A low match after sign and permutation adjustment is stronger evidence of sensitivity than a simple mismatch in factor labels.

Individual factors may change even when the subspace they span is stable. This happens when the data identify a low-dimensional space well but do not identify a unique orientation within it. In that case, reporting a single loading vector as a uniquely determined signature overstates what the fit supports. Comparing the subspaces—for example, through their principal angles—can help distinguish unstable orientation from an unstable span. That comparison supplements rather than replaces the loading and contribution checks.

Diagnostics, Alternatives, and Limitations

No single diagnostic is a universal test for confounding. Interpret several checks together and in the context of the model and preprocessing:

• Loading correlation: high absolute correlation within a view flags overlapping profiles. Check the number and identity of features, whether the correlations are defined, and whether the result is driven by a few extreme loadings.
• Contribution similarity: comparable contributions are more concerning when loading profiles also overlap or individual directions change across fits. Contribution size is scale-dependent and may not be additive if score columns are correlated.
• Initialization sensitivity: compare after accounting for sign and permutation. Different objectives, preprocessing, or constraints invalidate a direct like-for-like comparison. If fitted values and subspaces agree while axes differ, orientation ambiguity is a plausible explanation.
• Subspace comparison: principal angles compare spans rather than particular axes. A stable span with unstable factors supports a different interpretation from both unstable axes and an unstable span.

These checks are diagnostic, not a formal hypothesis test with a universal cutoff. Loading correlations can be unstable when computed over few features, and similarities can be distorted by strong feature dependence or heterogeneous scaling. Near-equal contributions depend on the model’s scaling and decomposition. Repeated-start variability can also arise because an optimizer has not found comparable solutions; such evidence should not be attributed automatically to confounding. A full assessment may require checking fit quality and optimization behavior, while recognizing that those are distinct questions from whether the directions are interpretable.

The main alternatives answer different questions. A loading-correlation matrix gives a quick, feature-level view of overlap; contribution comparisons summarize relative fitted magnitude; repeated fits assess dependence on starting values; and subspace comparisons assess stability of a span. These approaches are complementary, not interchangeable. None alone selects the number of factors, establishes biological meaning, or demonstrates that a factor causes an outcome. Choosing the number of factors is a separate model-selection question.

Common Pitfall

A common mistake is to see two factors with similar contributions and conclude that one should be removed, or to see a high loading correlation and claim that the factors are identical. The first conclusion confuses contribution magnitude with evidence for a particular model dimension; the second ignores score patterns, other views, constraints, and uncertainty. Instead, report the diagnostics, state how factors were matched and scaled, and distinguish evidence about individual directions from evidence about their joint span. Use cautious language such as “the fitted directions appear weakly separated under these diagnostics,” rather than treating a threshold as a definitive decision rule.