Comparing Competing Factor Solutions

Why Compare Factor Solutions?

A factor analysis does not usually deliver one uniquely determined answer. Two candidate solutions may both represent the same multi-omics data reasonably well while differing in factor number, allocation of variation, or detail in their loading profiles. The practical task is not simply to select the solution with the smallest residual sum of squares. It is to assess whether improvements in fit are meaningful, whether important residual structure remains, whether the factors admit useful interpretations, and whether those interpretations recur under resampling.

The comparison is most informative when both candidates use the same subjects, features, transformations, missing-data treatment, and evaluation scheme. If those differ, a numerical fit difference may reflect preprocessing rather than the factor solutions. This tutorial uses a common matrix representation; the same reasoning can be applied block by block when features come from several omics assays.

For a fair comparison, first verify that each candidate was fitted to the same analysis target. If features were standardized, use the same scaling parameters; if they were centered, use the same centering convention. Align loading profiles before comparing factors: factor order can be permuted, and multiplying a factor's scores and loadings by \(-1\) leaves the fitted matrix unchanged. A rotation can also change individual loading columns while preserving the fitted low-rank component. Consequently, compare factors only after reasonable matching and alignment, and distinguish the fitted subspace from a particular rotated representation.

Fit: Improvement, Generalization, and Complexity

A direct measure of in-sample discrepancy is the residual sum of squares, \(RSS = \sum_{i=1}^{n}\sum_{j=1}^{p} R_{ij}^{2}\). If features are on comparable scales, a per-cell mean squared error is \(RSS/(np)\); its square root is an RMSE in the analyzed feature units. These measures summarize overall discrepancy but do not show where it occurs. A more flexible solution can lower training RSS simply by adapting more closely to the observed data.

Evaluate generalization by fitting on training subjects and computing squared prediction errors on held-out subjects, using a prediction procedure defined before examining the held-out results. For a held-out set \(\mathcal{H}\) of subject-feature pairs, define \(MSE_{\mathcal{H}} = |\mathcal{H}|^{-1}\sum_{(i,j)\in\mathcal{H}}(X_{ij}-\widehat{X}_{ij})^2\). If the scientific goal is to predict new subjects, split by subject, not by individual cells: putting some cells from the same subject in training and others in validation can leak subject-specific information. All preprocessing that estimates parameters from the data must be learned from the training set and applied to validation data.

The tendency of training error to decrease with added factors has a simple justification in an unconstrained low-rank least-squares fit. Any rank-at-most-\(K\) fit is also an allowed rank-at-most-\(K+1\) fit, by adding a zero component. Thus the minimum training RSS over the larger feasible set cannot exceed the minimum over the smaller set. This argument does not guarantee that every optimization algorithm finds the global minimum, and added constraints or different likelihoods can change the comparison. It does explain why a training-fit gain alone is weak evidence for adding a factor.

Information criteria can penalize complexity, but they require care. A likelihood-based criterion is meaningful only when both solutions define comparable likelihoods for the same observations and use compatible likelihood assumptions; a parameter count must also reflect the fitted model and its constraints. AIC or BIC calculated from incompatible likelihoods, different preprocessing, or informal effective parameter counts should not be treated as an objective ranking. Held-out fit, likelihood criteria, and residual diagnostics answer related but different questions.

Residual Structure and Interpretation

Residual checks ask whether the candidate has left systematic structure unexplained. Examine residual variances by feature and by omics block, residual correlations between features, and residual patterns against fitted scores or relevant covariates. A few large residuals may be influential; a cluster of correlated residuals can indicate a missing shared pattern or model mismatch. A global RSS can conceal both. Conversely, a residual pattern need not mean that another factor should be added: it may reflect technical effects, an unsuitable likelihood, an outlier, or a feature-specific process outside the model's intended scope.

Interpretability is another comparison dimension, not a substitute for fit. Ask whether the loading profiles identify coherent feature sets, whether those profiles are distinguishable from one another, and whether the interpretation depends on an arbitrary rotation or threshold. In multi-omics analyses, check whether a proposed signal is supported across blocks or is driven by one block's scale or feature count. A factor's association with a biological label is descriptive unless the design and analysis justify a causal interpretation.

For two solutions, record conclusions at more than one level. A broad conclusion may be that a coordinated immune-related pattern is present. A narrower conclusion may assign a particular feature set to a specific factor, or claim that a factor combines two pathways. The broad conclusion can remain credible even when the narrower allocation changes between solutions.

Resampling Stability and Robust Conclusions

Resampling asks whether a candidate's salient structure recurs when the observed subjects or features are perturbed. For subject resampling, refit each solution in bootstrap samples and compare its factors with the reference fit. Before computing profile similarities, match factors, account for sign changes, and use a consistent scaling. If factors cannot be matched, record that rather than forcing a misleading correspondence. Stability measures can include loading-profile correlations, agreement among selected high-loading features, or the frequency with which a factor is recovered. These quantities describe reproducibility under the chosen resampling scheme; they do not establish biological truth.

When comparing two solutions, stability should be considered alongside fit and interpretation. A factor that appears in nearly every resample with a similar profile is stronger evidence for a reproducible pattern than a factor that appears only occasionally. But high stability can occur for a systematic artifact, and low stability can reflect correlated or weakly identified factors that exchange or split across fits. Resampling diagnostics from the earlier material on factor stability are useful here, provided the factor matching and perturbation scheme are appropriate to the comparison.

A Disciplined Comparison

A practical comparison keeps the evidence separate before making an overall judgment. Do not combine incomparable measures into an arbitrary score. Instead, state what each measure shows, note where the candidates agree or disagree, and explain which conclusions remain supported under both.

Alternative approaches emphasize different goals. A variance-explained or scree-style summary is compact but can favor additional factors and says little about held-out prediction. Cross-validation evaluates prediction for a defined target but can be noisy, and its result depends on the split and preprocessing pipeline. Likelihood-based information criteria account for fit and a complexity penalty under their model assumptions. Posterior predictive checks assess whether simulated data from a fitted probabilistic model reproduce relevant features of the observed data. None replaces careful inspection of residuals, interpretability, or stability. The appropriate comparison depends on whether the main objective is prediction, compact representation, or interpretable and reproducible structure.

Common Pitfall and Limitations

A common mistake is to conclude, “The model with more factors has lower RSS, so it is the better solution.” This reasoning confuses the ability to fit the training data with evidence that the added factor captures a useful, generalizable pattern. Training RSS is expected not to increase when the fitting procedure can use a larger nested low-rank class. The correction is to check held-out performance, residual structure, factor coherence, and resampling stability, and to state what remains uncertain.

No comparison can remove dependence on modeling choices. Results can change with feature scaling, missingness, sample size, likelihood, prior or penalty, and rotation. Held-out performance estimates performance for the validation design, not universal predictive ability. Bootstrap stability concerns the selected perturbations and sample; it does not guarantee external reproducibility. If two factors are highly correlated or weakly identified, their separate labels may not be stable even when their combined subspace is. Report robust conclusions at the level the evidence supports, and avoid treating one solution as uniquely correct merely because it has the most attractive fit statistic.