Why Compare Ranks Predictively?
A factor model with more factors can usually fit the observed data at least as well as a smaller model. That fact alone does not show that the additional factors capture reproducible structure: they may instead fit noise, technical variation, or idiosyncrasies of the particular samples used to estimate the model. Predictive criteria address this problem by asking whether a candidate rank describes data it was not fitted to, or by penalizing the improvement in fit for the added complexity.
For each candidate rank \(K\), fit the same specified model to the same training data and compare its performance using a criterion suited to the data and prediction target. Here, rank \(K\) is the number of latent factors. The earlier material on choosing the number of factors motivates considering plausible ranks; this tutorial focuses on how predictive scores and information criteria compare them.
Held-Out Likelihood and Reconstruction Error
Let \(Y\) denote the observed multi-omics data, with one matrix for each view. For a validation set \(Y_{\mathrm{val}}\), the held-out log likelihood at rank \(K\) is the log predictive density assigned to those observations by a model fitted using training data \(Y_{\mathrm{train}}\). A larger value is better. If models use different observation distributions, the likelihood must be computed on the same observed outcomes and with comparable conventions.
The expression is schematic: the predictive distribution must reflect how the validation data could be predicted from the training data. For example, to test prediction of one omics view from another, use the available view to infer a validation sample's score and evaluate the withheld view. To test prediction for entirely new samples, a model must specify how new sample scores are predicted or distributed. Fitting scores using the very outcomes scored as “held out” would make the evaluation optimistic.
A reconstruction-error criterion instead compares observed validation values with their reconstructed or predicted values. For continuous, standardized features, a common choice is mean squared error (MSE). Lower error is better. With multiple views, state how feature scales and views are weighted: an unscaled sum of squared errors can be dominated by a large or high-variance view. Standardizing using training data or evaluating each view separately can help, but the choice should match the scientific prediction target.
Worked Example: Choosing a Rank by Held-Out Log Likelihood
Suppose a hypothetical three-view study uses five-fold validation, with folds separated by sample. Scores are average held-out log predictive densities per validation sample, averaged across folds; higher is better. Assume each fold uses the same held-out samples and scoring definition for all ranks.
| Rank \(K\) | Mean held-out log score | Standard error across folds |
|---|---|---|
| 1 | -12.4 | 0.30 |
| 2 | -10.8 | 0.25 |
| 3 | -10.2 | 0.20 |
| 4 | -10.1 | 0.22 |
The best mean score is at \(K=4\), because \(-10.1\) is larger than \(-10.2\). However, the improvement over \(K=3\) is only \(0.1\) log-score units per sample, smaller than the reported fold-level standard errors. Those standard errors are not, by themselves, a formal test of the difference; paired fold differences and their variability are more informative because the same folds were scored at each rank. A defensible practical conclusion is that ranks 3 and 4 predict similarly in this validation, so one might prefer rank 3 for parsimony unless other evidence supports rank 4. The score does not establish that either rank is biologically true.
Worked Example: Validation Reconstruction Error
In a hypothetical continuous-data analysis, all features are standardized using training-set means and standard deviations. The analyst evaluates 100 held-out feature values, with the same validation values and weights for each rank. The sums of squared prediction residuals are 120, 91, 84, and 88 for \(K=1,2,3,4\), respectively.
For \(K=3\), the validation MSE is \(\mathrm{MSE}_{\mathrm{val}}(3)=\sum_{j=1}^{100}(y_j-\hat y_j)^2/100=84/100=0.840\) squared standardized units. The corresponding MSEs are 1.20, 0.910, 0.840, and 0.880 for ranks 1 through 4. The smallest is at \(K=3\), so this criterion favors rank 3. Rank 4 has lower training error in many factor-fitting procedures, but here it predicts the held-out entries less accurately than rank 3. The units are squared standardized units, not the original measurement units.
This conclusion depends on the evaluation target. If prediction of a particular view matters most, calculate and report that view's error rather than allowing numerous features in another view to dominate a pooled score.
Information Criteria and the Fit–Complexity Trade-Off
Information criteria use a fitted likelihood and a penalty that increases with model complexity. Let \(\ell_K\) be the maximized log likelihood for rank \(K\), \(p_K\) the number of free parameters under the specified model, and \(n\) the number of independent sampling units represented in the likelihood. Common definitions are:
For either criterion, smaller is preferred among comparable candidate models. The fit term \(-2\ell_K\) rewards better likelihood, while the penalty discourages adding parameters. AIC is motivated as an approximate correction for the optimism of in-sample fit when estimating expected predictive discrepancy. BIC arises from a large-sample approximation to a marginal likelihood under regularity conditions. They have different aims and need not select the same rank.
The parameter count must reflect the actual fitted model and its constraints. It is not generally correct to count every entry in a loading matrix and score matrix as an independent parameter: factor representations have invariances and may have constrained or shared parameters. In a probabilistic model, count the free parameters of the likelihood model as defined, not latent quantities treated as random observations. Some latent or penalized models need a model-specific effective degrees of freedom rather than a simple integer count. Also use the appropriate independent sample size: the number of cells in a multi-omics matrix is not automatically \(n\), since features from the same sample are not independent replicates.
Worked Example: AIC and BIC Can Prefer Different Ranks
Consider hypothetical fits to the same 120 independent samples, with comparable maximized likelihoods and model-specific free-parameter counts supplied by the model specification. The values are \((\ell_K,p_K)=(-520,18),(-480,30),(-465,41)\) for \(K=1,2,3\).
For \(K=2\), \(\mathrm{AIC}(2)=-2(-480)+2(30)=960+60=1020\). For \(K=3\), \(\mathrm{AIC}(3)=930+82=1012\); for \(K=1\), it is \(1040+36=1076\). Thus AIC selects \(K=3\).
For BIC, \(\log(120)\approx4.787\). The values are \(\mathrm{BIC}(1)=1040+18(4.787)\approx1126.2\), \(\mathrm{BIC}(2)=960+30(4.787)\approx1103.6\), and \(\mathrm{BIC}(3)=930+41(4.787)\approx1126.3\). BIC selects \(K=2\). The likelihood improves as rank increases, but BIC's larger penalty at this sample size offsets the additional fit from rank 3. Neither calculation is meaningful if the likelihoods are computed on different outcomes or the parameter counts are inconsistent with the fitted models.
Why Complexity Must Be Counted Carefully
A useful dimension calculation illustrates why raw factor entries overcount complexity. Consider an \(n\times q\) matrix \(M\) of rank exactly \(K\), represented as \(M=UV^\mathsf{T}\), where \(U\) is \(n\times K\) and \(V\) is \(q\times K\), both full column rank. The factors contain \(nK+qK\) entries. But for every invertible \(K\times K\) matrix \(A\),
Thus \(K^2\) degrees of freedom describe alternative factorizations of the same \(M\), rather than different rank-\(K\) matrices. Subtracting these redundant degrees gives the dimension \(K(n+q-K)\). This is a justification for the dimension of the set of rank-\(K\) matrices under the full-rank assumption; it is not a universal formula for \(p_K\) in a probabilistic multi-omics factor model. Residual variances, covariates, constraints, shared and view-specific loadings, and the treatment of scores all change the relevant count.
Diagnostics, Alternatives, and Limitations
Rank comparisons can fail even when the arithmetic is correct. Inspect the following before trusting a selected rank:
- Leakage: Estimate scaling, feature selection, imputation, and other data-dependent preprocessing within each training fold. Using validation outcomes to make these choices can make scores too favorable.
- Misaligned splits: Split at the level relevant to the scientific target. If repeated observations from one person appear in training and validation, apparent performance on new people can be misleading.
- Unequal influence of views: Report view-specific scores or justify any aggregation. A pooled error can hide poor prediction in a smaller but important assay.
- Unstable scores: Compare paired fold-level differences and examine their variability. A tiny mean improvement may not be practically distinguishable from validation noise.
- Model mismatch: Gaussian squared-error scoring is not automatically suitable for counts or binary data. Use an observation model and predictive score appropriate to the outcomes.
Held-out likelihood directly assesses predictive density when it is correctly defined, but requires a credible validation design and can be computationally expensive. Reconstruction error is easy to interpret for continuous measurements, but depends on scale and may not correspond to a probability model. AIC and BIC are convenient summaries of fit and complexity, but rely on likelihood assumptions and regularity approximations; latent-factor singularities or boundary constraints can make standard asymptotic arguments unreliable. Criteria can disagree without any being calculated incorrectly because they target different trade-offs.
A common mistake is to select the rank with the smallest training reconstruction error. The reasoning is that a smaller residual must indicate the better factor model. The correction is that adding factors can reduce training error by fitting sample-specific noise; use held-out performance, a justified complexity penalty, or both. Also avoid interpreting a selected \(K\) as a count of distinct biological mechanisms: the answer depends on the model, data, preprocessing, and criterion.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For candidate ranks, held-out mean log scores are \(-8.2\), \(-7.5\), and \(-7.6\). Which rank has the largest score, and what further information would help judge whether its advantage is convincing?
- A validation set has 50 standardized entries and a sum of squared prediction residuals of 40. Calculate its MSE and state the units.
- For a model with \(\ell=-90\), \(p=12\), and \(n=40\), calculate AIC and BIC to one decimal place.
- Why can counting every entry in \(U\) and \(V\) overstate the dimension of the rank-\(K\) matrix \(UV^\mathsf{T}\)?
Solutions. (1) Rank 2 has the largest score, \(-7.5\). Paired fold-level differences and their variability, using identical folds and scoring rules, help assess whether the improvement over the other ranks is stable. (2) \(\mathrm{MSE}=40/50=0.800\) squared standardized units. (3) \(\mathrm{AIC}=-2(-90)+2(12)=204.0\). Since \(\log(40)\approx3.689\), \(\mathrm{BIC}=180+12(3.689)\approx224.3\). (4) For any invertible \(K\times K\) matrix \(A\), \(UA\) and \(VA^{-\mathsf{T}}\) produce the same matrix. These equivalent factorizations contribute \(K^2\) redundant degrees of freedom, giving dimension \(K(n+q-K)\) under the stated full-rank conditions.