Orthogonality-Constrained NMF

Why Control Component Overlap?

A nonnegative matrix factorization represents a nonnegative data matrix \(X\) as an approximate product of two nonnegative matrices. The columns of the feature factor describe component profiles across features; the rows of the score factor describe how components are expressed across samples. Components may overlap: the same feature can have positive weight in several profiles, and a sample can have positive scores for several components. This is often scientifically plausible, but sometimes a model is easier to interpret if components are less overlapping.

Orthogonality is one way to discourage or prohibit that overlap. It is a modeling choice, not a general requirement of NMF. In particular, nonnegative orthogonal components have a stronger interpretation than merely “uncorrelated” signed vectors: their positive supports cannot intersect. Whether that is sensible depends on whether shared features or shared sample activity are plausible in the application.

What Orthogonality Means With Nonnegative Factors

For two nonnegative vectors \(u\) and \(v\) of the same length, their inner product is \(u^Tv=\sum_j u_jv_j\). Every summand is nonnegative. Therefore, \(u^Tv=0\) if and only if \(u_jv_j=0\) at every coordinate \(j\). In words, the vectors cannot both be positive at the same coordinate. A hard constraint on \(W\) thus assigns each feature to at most one component with positive weight. A hard constraint on \(H\) means each sample has a positive score for at most one component.

This is a much more restrictive condition than orthogonality in unconstrained methods, where positive and negative products can cancel. For example, the signed vectors \((1,1)\) and \((1,-1)\) have inner product zero despite sharing both coordinates. Such cancellation is impossible for nonnegative vectors. Orthogonality of \(W\) and orthogonality of \(H\) are also different modeling assumptions: the former excludes shared features, while the latter excludes co-activity in samples.

Hard constraints are sometimes written \(W^TW=I\) or \(HH^T=I\), where \(I\) is the identity matrix. These equations impose both orthogonality and unit norms. The off-diagonal zero conditions alone impose orthogonality; diagonal values need not be one. Because NMF has scale indeterminacy—\(WH=(WD)(D^{-1}H)\) for a positive diagonal matrix \(D\)—a norm convention must be made explicit if unit norms are imposed. The hard support restriction follows from the off-diagonal conditions in either formulation.

Soft Orthogonality in a Penalized Objective

A common soft approach adds a penalty to the squared-error NMF objective. For example, if overlap in sample scores is of interest, one may use

Here \(\lVert A\rVert_F^2\) is the sum of the squared entries of \(A\), and \(\lambda\geq0\) controls the trade-off. The first term measures reconstruction error; the second penalizes pairs of score rows that have large inner products. When \(\lambda=0\), there is no orthogonality penalty. Increasing \(\lambda\) makes overlap more costly relative to reconstruction error. A finite penalty generally encourages smaller overlaps but does not guarantee exact orthogonality.

The displayed penalty uses unnormalized inner products, so it reflects both the angle and the magnitudes of the score rows. Its value also depends on scale. To make that penalty interpretable, one must use a consistent scale convention—for example, normalize each active column of \(W\) to unit norm and place its amplitude in the corresponding row of \(H\). A normalized-correlation penalty is another option when the aim is to penalize direction rather than magnitude, but it is undefined for a zero row and requires a rule for inactive components. The penalty’s definition and scaling convention should be reported alongside \(\lambda\); its numerical value is not meaningful in isolation.

A Small Penalized Optimization

The trade-off can also be solved within a simple restricted family, making the effect of \(\lambda\) explicit. Keep \(X\) and \(W=I\) as above, and restrict the score matrix to \(H(t)=\begin{pmatrix}1&1\\t&0\end{pmatrix}\), where \(t\geq0\). The first row is fixed, while the second component’s score on the first sample is allowed to vary. This is a one-parameter constrained problem, not a claim about all possible NMF factorizations.

Choosing the Constrained Factor and Checking the Fit

The factor to constrain should follow the scientific question. Penalizing \(HH^T\)’s off-diagonal entries targets overlap of component scores across samples. Penalizing \(W^TW\)’s off-diagonal entries targets overlap of feature profiles. Applying both penalties is possible, but it can impose a severe restriction: each component would need its own feature support and its own sample support. For multi-omics data, where biological processes may affect several features or coexist in a sample, that restriction may be implausible.

Orthogonality can simplify descriptions by encouraging components to have distinct supports. It does not establish biological independence, causal separation, or statistical uncorrelatedness of measured variables. Components remain products of a chosen factorization model, and the penalty’s effect is relative to the reconstruction term and the data’s scaling. Results should therefore be interpreted as a property of the fitted representation.

Useful diagnostics include the achieved pairwise inner products (or correlations, if that is the penalty used), the reconstruction error, and the factor norms under the chosen scale convention. Examine these across a sensible range of \(\lambda\), rather than judging a fit only by its penalized objective. If overlap barely changes as \(\lambda\) increases, optimization may not have reached a good solution, or the model may be constrained by other choices. If overlap falls but reconstruction error rises sharply, the imposed separation may be distorting the data’s structure. Zero or nearly zero components also need attention: they can make normalized overlaps undefined or appear artificially small.

A soft penalty is not the only alternative. A hard constraint guarantees orthogonality but can make the feasible set restrictive and may prevent a good reconstruction. Ordinary NMF leaves overlap unrestricted and may fit better, while sparse NMF encourages small or zero factor entries through a different penalty; sparsity does not itself ensure orthogonality. These approaches answer different questions and should not be treated as interchangeable.

Practical Limitations

Hard orthogonality may be incompatible with shared biological signals, and a large soft penalty can force components apart even when the data support shared activity. Penalty values are not portable across datasets with different scales, factor normalizations, or objective conventions. Orthogonality constraints also do not remove the general nonconvexity of NMF, so different initializations can yield different local solutions. Compare fits under consistent scaling and initialization practices, report the factor being constrained and the exact penalty, and interpret separation as a modeling preference rather than a discovered law of the system.