Why Rank Selection Requires More Than a Small Loss
In non-negative matrix factorization (NMF), the rank \(K\) is the number of nonnegative components used to approximate a data matrix. A larger \(K\) can represent more detail, but it also introduces more parameters and may split a broad pattern into several components. A small reconstruction loss is therefore not, by itself, evidence that a rank is scientifically appropriate.
Let \(X\) be a nonnegative matrix with \(n\) samples in rows and \(p\) features in columns. At candidate rank \(K\), NMF represents it by \(W\) and \(H\), where \(W\) contains sample scores and \(H\) contains feature profiles. The entries of both factors are nonnegative. The fitting objective should match the data and analysis: for example, a count-oriented divergence for count matrices, as covered in the earlier material on NMF objectives for count matrices. Candidate ranks must be compared using the same preprocessing, objective, and treatment of any regularization.
This tutorial assumes the rows and columns of \(X\), its preprocessing, and the loss function have been chosen appropriately for the analysis. Candidate fits should use comparable convergence criteria and multiple initializations, because NMF optimization is generally nonconvex: a run can stop at a local solution rather than the best solution available at that rank. Components also have scale and permutation indeterminacy. Consequently, component comparisons require a consistent normalization and matching convention rather than relying on the raw factor entries.
Reconstruction Curves and Their Limits
A reconstruction curve plots the optimized training loss \(L_K\) against candidate rank \(K\). With a fixed objective, smaller loss means a closer fit according to that objective. The curve often decreases as rank increases, so the relevant question is not simply which rank gives the smallest loss. Instead, examine how much improvement each extra component provides and whether that improvement is worth the added complexity for the intended use.
For a loss where smaller is better, one useful descriptive quantity is the relative gain from rank \(K-1\) to \(K\):
A sharp reduction followed by much smaller gains may suggest an elbow. The calculation is a summary, not a formal test: there is no universal cutoff for \(G_K\), and a gradual curve may have no clear elbow. The scale and interpretation of \(L_K\) depend on the chosen objective, so the curve should be reported with the objective and any normalization stated.
For globally optimized unregularized reconstruction under a fixed loss, the best achievable training loss cannot increase when rank increases. To see why, take any rank-\(K\) factors \(W,H\) and append a zero column to \(W\) and a zero row to \(H\). The product remains \(WH\), now represented at rank \(K+1\). Thus the rank-\(K+1\) feasible set includes representations available at rank \(K\), and minimizing the same loss over the larger set cannot yield a larger minimum. In practice, reported losses can rise slightly if optimization is imperfect, initialization differs, or the objective and penalties have changed. Such a rise is a reason to check fitting and comparability, not evidence that the mathematical nesting argument failed.
Worked Example: Finding a Flattening Point
Suppose a hypothetical count-matrix analysis uses one fixed count-appropriate objective, with loss reported per matrix entry. After multiple starts, the best comparable losses are \(L_1=0.240\), \(L_2=0.160\), \(L_3=0.126\), \(L_4=0.121\), and \(L_5=0.119\). Calculate the relative gains and describe what the curve suggests.
From rank 1 to rank 2, the gain is \(G_2=(0.240-0.160)/0.240=0.080/0.240=0.333\), or \(33.3\%\). From rank 2 to rank 3, \(G_3=(0.160-0.126)/0.160=0.034/0.160=0.213\), or \(21.3\%\). From rank 3 to rank 4, \(G_4=(0.126-0.121)/0.126=0.005/0.126=0.0397\), or \(3.97\%\). From rank 4 to rank 5, \(G_5=(0.121-0.119)/0.121=0.002/0.121=0.0165\), or \(1.65\%\).
The gain drops substantially after rank 3, so ranks 3 or 4 are reasonable candidates for closer assessment; the curve alone does not decide between them. For instance, a remaining structured residual pattern at rank 3 could make rank 4 useful, while redundant or poorly reproducible components at rank 4 could favor rank 3. These conclusions are conditional on comparable optimization and the same preprocessing and objective.
Residual Structure and Component Redundancy
Residuals are the differences between observations and their fitted values, \(R=X-WH\). Their patterns can indicate that a chosen rank, objective, or other modeling choice does not adequately represent the data. For example, a coherent set of features with residuals of the same sign, or a sample subgroup with consistently large residuals, can suggest remaining structure. Random-looking residual variation is generally more reassuring than a visible pattern, but it does not prove the model is correct. Residual displays should respect the data scale and observation model: raw differences for counts, for example, can be difficult to compare across features with very different expected magnitudes.
A separate warning sign is component redundancy: two components may capture nearly the same feature profile, sample-score pattern, or both. Since NMF permits rescaling a component by multiplying its scores by a positive constant and dividing its feature profile by that constant, normalize profiles before comparing them. For example, normalize each nonzero row of \(H\) to have sum one. Similarity of normalized profiles alone is not conclusive: two components could have similar features but occur in different samples. Consider their score patterns as well. Similar profiles and similar score patterns can suggest that a larger rank has divided one pattern into near-duplicates.
Worked Example: Checking Whether Two Components Are Duplicates
In a hypothetical rank-3 fit, two normalized feature profiles over three features are \(h_1=(0.80,0.60,0)\) and \(h_2=(0.78,0.625,0)\). Their cosine similarity is the dot product divided by the product of their Euclidean lengths. Both lengths are 1 to rounding, so \(\cos(h_1,h_2)=0.80(0.78)+0.60(0.625)=0.624+0.375=0.999\). The feature profiles are extremely similar.
Now suppose their sample-score vectors across four samples are \(w_1=(1,2,4,3)\) and \(w_2=(4,3,1,2)\). These vectors have the same mean, \(2.5\). Their centered versions are \((-1.5,-0.5,1.5,0.5)\) and \((1.5,0.5,-1.5,-0.5)\), so their Pearson correlation is \(-1\). Thus, despite nearly identical feature profiles, they are active in opposite sample patterns in this example. Calling them duplicates based only on the feature-profile cosine would be misleading. This check raises a question for investigation; it does not establish that either component is scientifically meaningful.
Resampling and Repeated Fits
NMF can depend on initialization, and the data themselves are only one observed sample from a broader process. Repeated starts and resampling address different sources of variation. Multiple starts at the same rank assess whether optimization repeatedly finds comparable fits on the observed matrix. Resampling samples or features assesses whether similar structure recurs when the data are perturbed. Comparisons must account for component permutation and scale; otherwise the same component can appear to change merely because its label or magnitude changed.
For rank selection, use reproducibility as supporting evidence rather than as a separate definition of truth. A rank that repeatedly gives similar fit and broadly similar component patterns is more defensible than one whose apparent structure changes drastically across reasonable perturbations. However, a stable component can reflect a stable artifact, such as a batch effect, and unstable components may still represent weak real structure. The detailed mechanics of component stability and alignment are distinct questions; here, the key point is to use reproducibility as one part of a rank comparison.
Worked Example: Using Repeated Fits to Compare Two Ranks
Suppose a hypothetical study compares \(K=3\) and \(K=4\) across 10 resamples. For each rank, investigators record whether a component pattern can be matched across runs under a prespecified similarity rule. At \(K=3\), all three patterns are matchable in 8 of 10 resamples; at \(K=4\), all four are matchable in 3 of 10. The corresponding fractions are \(8/10=0.80\) and \(3/10=0.30\).
This summary favors rank 3 on reproducibility in this particular comparison, especially if rank 4’s extra pattern is the one often unmatched. It does not prove that rank 3 is correct: the matching rule, resampling scheme, and number of resamples affect the summary, and the rank-4 fit might still capture a useful weak pattern. A responsible conclusion combines this evidence with the reconstruction curve and residual checks, and states the resampling design.
Putting Diagnostics Together
Rank selection is strongest when several kinds of evidence tell a compatible story. A practical comparison can proceed as follows.
State the data transformation, objective, any regularization, and the purpose of the representation. Do not compare losses from incompatible objectives as if they were on one scale.
Use several starts per rank and consistent convergence checks. Retain enough information to distinguish a poor local fit from a genuinely larger minimum loss.
Look for diminishing gains and for residual structure that remains consequential at lower ranks. Do not turn a visual elbow into a universal threshold.
Ask whether extra components contribute distinct sample and feature patterns, and whether those patterns persist across starts or suitable resamples.
Explain which trade-off matters for the analysis, identify plausible alternatives, and report evidence that would make the choice sensitive.
Other approaches can complement these diagnostics. Held-out prediction or masked-entry reconstruction can assess how well a model predicts observations not used in fitting, provided the holdout scheme is appropriate to the data structure and the evaluation uses a suitable loss. Information criteria require a defensible likelihood and parameter-count treatment; they are not automatically available from an arbitrary NMF objective. A reconstruction elbow is quick and transparent but can be subjective. Residual checks can reveal a poor representation but do not select a rank on their own. Resampling provides evidence about reproducibility, but can be computationally intensive and sensitive to the perturbation design. No approach removes the need to specify what kind of representation is useful.
Common Pitfalls and Limitations
- Choosing the rank with the smallest training loss. Training loss usually improves as rank grows, so this rule tends to favor larger models. Compare gains and complexity, and check whether added components are useful and reproducible.
- Treating the elbow as an objective answer. Some curves have no clear elbow, and its location can depend on the loss scale. Report it as a heuristic, not a test or proof.
- Calling similar feature profiles duplicates without checking scores. Similar profiles can be used across different samples. Compare both sides of the factorization after accounting for scale and permutation.
- Assuming a stable fit is necessarily valid. Repeatedly recovered structure can be a reproducible artifact. Check residual patterns and the study design as well.
- Comparing ranks fitted under different conditions. Changes in preprocessing, loss, weights, or regularization can change the objective. A resulting loss difference cannot be attributed to rank alone.
NMF rank is an approximation choice, not necessarily the number of uniquely existing biological processes. Different ranks can be useful for different purposes: a compact representation may aid summary, while a larger rank may preserve finer distinctions. For joint multi-omics fits, also verify that the objective and block scaling give the intended influence to each data block; otherwise a rank comparison may mostly reflect the dominant block. State the selected rank, the candidate range, the diagnostics used, and the main limitation of the choice.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- If the best training loss decreases at every rank, why is the rank with the smallest loss not automatically preferred?
- A reconstruction curve has relative gains of 25%, 7%, and 6% over three successive increments. What does this pattern suggest, and what can it not establish?
- Two components have very similar normalized feature profiles. What additional comparison is important before calling them redundant?
- What different evidence do repeated initializations and sample resampling provide?
- Why can a small increase in reported loss at a larger rank occur even though the feasible-set argument predicts a non-increasing optimum?
Solutions. (1) A larger rank can reduce training loss while adding unnecessary or unstable complexity; inspect improvement, residual structure, redundancy, reproducibility, and the analysis goal. (2) The largest improvement is followed by smaller gains, suggesting a possible flattening region. The gains do not prove an objectively correct rank or provide a universal cutoff. (3) Compare the corresponding sample-score patterns too, after accounting for NMF scale and permutation indeterminacy. Similar profiles may occur in different samples. (4) Repeated initializations assess optimization sensitivity on the same data; sample resampling assesses sensitivity to perturbations of the observed data. Neither guarantees scientific validity. (5) Numerical optimization may find a poorer local solution, starts or convergence may be inadequate, or the compared objectives or penalties may differ. The nesting result applies to the global optimum under the same objective.