Smooth NMF for Ordered Features

Why Smoothness Can Be Useful

In nonnegative matrix factorization, a matrix \(X\) is approximated by \(WH\), where \(W\) and \(H\) have nonnegative entries. When the rows of \(X\) represent measurements with a meaningful order, it may be reasonable to expect each component’s weights across those rows to change gradually. For example, neighboring wavelengths or genomic positions may carry related signal. A smoothness penalty expresses that expectation as part of the fitting objective rather than treating it as a certainty about the data.

Ordering can also occur along the columns of \(X\). If columns are measurements taken at ordered time points, the corresponding columns of \(H\) contain the component scores at those times. Penalizing neighboring score differences can favor trajectories that change gradually. The choice of factor to penalize therefore follows from the dimension in which the ordered measurements occur.

The penalty favors small changes, not necessarily identical values. Its strength is controlled by a nonnegative tuning parameter: setting that parameter to zero removes the smoothness preference, while increasing it makes rough adjacent changes more costly. The appropriate strength depends on the measurement scale and the scientific aim; smoothness is not automatically beneficial just because an order exists.

Difference Matrices and the NMF Objective

For \(m\) ordered rows, define the \((m-1)\times m\) first-difference matrix \(D_m\) to have \(-1\) and \(1\) in consecutive positions in each row, with zero elsewhere. Thus, its first row calculates the second entry minus the first, its second row calculates the third minus the second, and so on. Multiplication by \(D_m\) calculates all adjacent differences at once.

The matrix \(L_m\) is the Laplacian of a path connecting each row to its immediate neighbor. The squared Frobenius norm \(\|D_mW\|_F^2\) sums the squared adjacent differences over every component. Equivalently, it is a quadratic form involving \(L_m\). For ordered columns, \(D_nH^\mathsf{T}\) calculates adjacent differences along the columns of \(H\); its squared Frobenius norm has the corresponding quadratic-form representation.

The factor \(1/2\) is a convenient convention for derivatives. For example, writing \(W_{i,:}\) for row \(i\) of \(W\), the row penalty satisfies \(\|D_mW\|_F^2=\sum_{i=1}^{m-1}\|W_{i+1,:}-W_{i,:}\|_2^2\). Likewise, the column penalty is \(\|HD_n^\mathsf{T}\|_F^2=\sum_{j=1}^{n-1}\|H_{:,j+1}-H_{:,j}\|_2^2\). These sums make clear which neighboring measurements are being compared.

Deriving the Smoothness Contribution

The reconstruction term is the usual squared-error NMF objective. To see how the added terms affect the fit, consider the gradient with respect to each factor while holding the other fixed. The identity \(\|A\|_F^2=\operatorname{tr}(A^\mathsf{T}A)\), together with the derivative of a quadratic form, gives the smoothness contribution for \(W\) as \(\lambda_W L_mW\). For \(H\), it is \(\lambda_H HD_n^\mathsf{T}D_n\). The complete unconstrained gradients are therefore:

For example, the \(W\) penalty can be derived directly:

The nonnegativity constraints still apply. These gradients describe the smoothness terms’ contribution to a constrained optimization method; they do not by themselves specify a complete algorithm or guarantee a global optimum. The reconstruction loss and the constraints must also be handled by the chosen NMF optimizer.

When an Ordering Has Unequal Gaps

The basic difference matrix treats every adjacent pair equally. That is reasonable when measurements are equally spaced or when adjacency itself, rather than distance, is the intended notion of neighborhood. If positions are unequally spaced and the scientific goal is to penalize rapid change per unit distance, the penalty should account for the gaps.

Let \(t_1,\ldots,t_m\) be increasing measurement positions and let \(\Delta_i=t_{i+1}-t_i>0\). A piecewise-linear derivative penalty uses \(\sum_i\|W_{i+1,:}-W_{i,:}\|_2^2/\Delta_i\). It is a weighted first-difference penalty: short intervals receive more weight for the same change, because that change represents a steeper slope.

Assumptions, Diagnostics, and Alternatives

A smoothness penalty does not establish that neighboring measurements really have similar underlying signals. Its justification is a modeling assumption: nearby positions are expected to have related factor values. Before applying it, verify that the ordering is meaningful, that the same notion of adjacency applies throughout the ordered dimension, and that factor scales make the penalty interpretable. For ordered genomic features, for instance, physical distance and boundaries between chromosomes may matter; an arbitrary list order is not a scientific basis for a smoothness penalty.

Factor scale deserves special attention. The product \(WH\) is unchanged if a column of \(W\) is multiplied by a positive constant while the corresponding row of \(H\) is divided by that constant. If only \(W\) is penalized, this rescaling can shrink its smoothness cost without changing the reconstruction. If only \(H\) is penalized, the reverse rescaling can shrink that cost. Thus a one-factor smoothness penalty may be evaded unless the factor scale is identified, for example by an appropriate normalization constraint. When both factors are penalized, their penalties can help determine the scale, but this does not remove the need to inspect the modeling choices and possible zero-penalty cases.

Useful diagnostics include plotting each fitted factor profile against the true measurement order, comparing adjacent differences before and after regularization, and checking whether important abrupt changes have been erased. Compare the reconstruction error and smoothness contribution across plausible values of \(\lambda_W\) or \(\lambda_H\). A fit with a very small roughness value is not automatically good: it may simply be over-smoothed. Also check whether missing measurements, irregular gaps, or discontinuities make the assumed sequence of neighbors inappropriate.

Smoothness and sparsity serve different goals. The earlier material on sparse NMF uses penalties to encourage small or zero factor entries; a first-difference penalty instead discourages neighboring entries from differing. A dense, slowly changing profile can have low roughness but not be sparse. Smoothing the data before NMF is another possible approach, but it changes \(X\) before the factorization and is not equivalent to penalizing the fitted factors. First-difference penalties are also not the only smoothness choice: a second-difference penalty targets changes in slope and can allow an approximately linear trend, whereas a first-difference penalty tends to favor locally constant profiles. The appropriate choice depends on what kinds of profiles are plausible.