Scale and Permutation Indeterminacy in NMF

Why Factor Scales and Labels Need a Convention

An NMF fit represents a nonnegative data matrix as a product of two nonnegative factor matrices. The columns of one factor and rows of the other describe paired parts of that representation. But the numerical entries in those factors are not uniquely determined by the fitted product: a component can be rescaled, and components can be listed in a different order, without changing the reconstruction. Consequently, comparing raw factor entries across fits can be misleading unless scale and ordering are handled explicitly.

This indeterminacy is not, by itself, evidence that a fitting algorithm has failed. It is a mathematical symmetry of the factorization. A normalization convention can choose one representative from a family of equivalent factorizations, making factor magnitudes interpretable on that convention’s scale. It does not prove that the factorization is otherwise unique, nor does it make the data’s measurement units irrelevant.

Here rows of \(X\) index samples and columns index features. Thus \(W_{ik}\) is the weight of component \(k\) for sample \(i\), while \(H_{kj}\) is the loading of feature \(j\) on component \(k\). The fitted entry is \((WH)_{ij}=\sum_{k=1}^r W_{ik}H_{kj}\). The invariances below require nonnegative factors and transformations that preserve nonnegativity. They preserve the product exactly, whether or not that product is a perfect fit to \(X\).

Positive Rescaling Leaves the Product Unchanged

For any component \(k\), multiply its column of \(W\) by a positive constant \(c\), and divide its corresponding row of \(H\) by the same constant. The component’s outer product is unchanged because the two scale factors cancel:

The condition \(c>0\) is important. It keeps both factors nonnegative and makes the transformation invertible. A negative scale can violate nonnegativity, while setting \(c=0\) destroys the component rather than merely changing its scale.

For all components at once, let \(D\) be an \(r \times r\) diagonal matrix with strictly positive diagonal entries \(d_1,\ldots,d_r\). Define \(W'=WD\) and \(H'=D^{-1}H\). Then

where \(I_r\) is the \(r \times r\) identity matrix. Both transformed factors remain nonnegative. If the loss function depends only on the reconstruction \(WH\), as does a loss based on reconstruction error or a divergence between \(X\) and \(WH\), its value is unchanged by this transformation. An objective with additional terms that depend directly on \(W\) or \(H\) need not have this scale invariance.

Permuting Components Leaves the Product Unchanged

Component numbers are labels, not intrinsic rankings. Let \(P\) be an \(r \times r\) permutation matrix: it rearranges coordinates and satisfies \(PP^T=P^TP=I_r\). Apply the same permutation to the columns of \(W\) and the corresponding rows of \(H\), by setting \(W'=WP\) and \(H'=P^TH\). Then

The transpose appears because the row order of \(H\) must track the reordered columns of \(W\). Permuting only one factor generally changes the product. Rescaling and permutation can also be combined: \(W'=WDP\) and \(H'=P^TD^{-1}H\) give \(W'H'=WH\).

Normalizing Factors Without Changing Reconstruction

A common convention is to give each nonzero column of \(W\) unit sum. For component \(k\), let \(s_k=\sum_{i=1}^m W_{ik}\). If \(s_k>0\), divide that column by \(s_k\) and multiply row \(k\) of \(H\) by \(s_k\). In matrix form, let \(D\) have diagonal entries \(d_k=1/s_k\). Then \(\widetilde W=WD\) has unit-sum columns, and \(\widetilde H=D^{-1}H\) has rows multiplied by the corresponding original column sums. The product remains unchanged because \(\widetilde W\widetilde H=WH\).

Under this convention, each column of \(\widetilde W\) describes a normalized profile across samples, while the compensating scale is carried by the corresponding row of \(\widetilde H\). A useful scale-invariant summary of the \(k\)th component is its total reconstructed mass. Nonnegativity lets us write that mass as a product of sums:

After normalizing column \(k\) of \(W\) to sum to one, this component mass equals the sum of row \(k\) of \(\widetilde H\). This can make component magnitudes comparable under the stated convention, provided the features have interpretable, compatible scales. It does not make magnitudes automatically comparable across different measurement units or differently preprocessed data blocks.

Choosing and Checking a Convention

Unit-sum columns of \(W\) are not the only possible convention. One could instead normalize rows of \(H\), for example to unit sum, and transfer their original row sums to columns of \(W\). That choice assigns scale to the other factor. The appropriate convention depends on which factor’s profile is intended to be normalized; what matters for comparison is to use the same convention and state it clearly.

Diagnostics should distinguish a harmless representation change from a substantive difference in the fit. If two reconstructions agree but their factor magnitudes differ by reciprocal scales, scale indeterminacy is a plausible explanation; if their component order differs, permutation is a plausible explanation. If the products differ, these invariances do not explain the difference. Even after normalizing and matching an ordering, distinct factorizations can still produce the same or similar reconstruction. Scale and permutation conventions resolve only these particular ambiguities, not every source of non-uniqueness.

Common Pitfalls and Limitations

• Comparing raw entries across unconstrained fits. A larger entry in \(W\) can be offset by a smaller entry in its paired row of \(H\). Compare factors only after choosing a consistent normalization.
• Permuting only one factor. Reordering columns of \(W\) without reordering the corresponding rows of \(H\) changes the component pairings and generally changes \(WH\).
• Normalizing without compensating. Dividing a column of \(W\) by its sum but leaving its matching row of \(H\) unchanged changes the reconstruction unless that component is zero.
• Treating normalization as proof of uniqueness. A fixed scale and component order do not rule out other factorizations that are not related by these transformations.
• Ignoring units and data preparation. A convention makes factor scales consistent within a chosen representation. It cannot make a loading measured on one feature scale directly comparable to one measured on another without considering units and preprocessing.

The main alternative to normalizing factors is to leave them in the scale produced by the fitting algorithm. That can be acceptable when only the fitted product matters, but raw factors from separate runs are then especially hard to compare. Normalization improves interpretability by choosing where scale is recorded; it does not improve reconstruction by itself. In particular, a close match between normalized factors should not be interpreted as proof that the underlying components are scientifically identical.