Deriving the Frobenius-Norm NMF Objective

From Reconstruction to an Objective

In nonnegative matrix factorization, a nonnegative data matrix is approximated by the product of two nonnegative, lower-dimensional factors. The rank and reconstruction-error ideas from the earlier material on rank, reconstruction error, and approximation lead to a practical question: how should the discrepancy between the data and the product be quantified? The squared Frobenius loss answers this question by summing the squared residuals over all matrix entries.

Let \(X\) be an \(n \times p\) data matrix, with \(n\) samples and \(p\) features, and let \(r\) be the chosen factorization rank. Use \(W\) for the \(n \times r\) sample-score factor and \(H\) for the \(r \times p\) feature-loading factor. Then \(WH\) is an \(n \times p\) reconstruction of \(X\). In NMF, the entries of \(X\), \(W\), and \(H\) are nonnegative. The loss is evaluated on the residual matrix \(R=WH-X\), whose entries are the reconstructed values minus the observed values.

The factor \(1/2\) is a convenient convention because it cancels the \(2\) that appears when a squared residual is differentiated. Some sources instead define the objective as \(L(W,H)=\|X-WH\|_F^2\). These objectives have the same minimizers, but their gradients differ by a factor of \(2\). Throughout the derivation and examples below, \(F\) denotes the half-scaled objective; the unscaled version is stated explicitly when relevant.

Assumptions and Scope of the Loss

The objective is defined for a complete, numeric matrix \(X\) and factors with compatible dimensions. It measures all entries on a common squared-error scale: a residual twice as large contributes four times as much to the loss. It also gives each matrix entry the same weight. If entries have different measurement scales or precisions, the unweighted objective may let high-magnitude or noisy features dominate.

No probability model is needed to define or minimize \(F\). If one additionally models \(X_{ij}=(WH)_{ij}+\varepsilon_{ij}\), with independent errors \(\varepsilon_{ij}\sim N(0,\sigma^2)\) of common variance, then minimizing the squared residual sum is equivalent to maximizing the Gaussian likelihood for fixed \(\sigma^2\), subject to the factor constraints. That statistical interpretation requires the stated error assumptions; using the loss alone does not establish them. In particular, the optimization criterion does not imply that observations are independent or that residuals are Gaussian or homoscedastic.

Deriving the Gradients

The factors enter the reconstruction through a matrix product, so each gradient must account for how a change in one factor changes that product. The derivation can be done entry by entry or with a differential. The differential makes the dimensions and the contributions of the two factors especially clear.

In detail, using the cyclic property of the trace, which allows cyclic rearrangement of compatible matrix products inside a trace, gives

The gradient is defined by writing the differential as the matrix inner product \(\operatorname{tr}((\nabla_W F)^T dW)+\operatorname{tr}((\nabla_H F)^T dH)\). Matching terms yields the gradients. Substituting \(R=WH-X\) gives equivalent expanded forms:

If instead the objective is the unscaled \(L=\|X-WH\|_F^2\), its gradients are \(\nabla_W L=2(WH-X)H^T\) and \(\nabla_H L=2W^T(WH-X)\). Omitting or adding the factor of \(2\) is not harmless if one reports gradients for a specific objective, even though it does not change which feasible points minimize the two objectives.

Dimension Checks and a Numerical Calculation

Dimension checks are a useful way to detect incorrect transposes. With \(X\) of size \(n \times p\), \(W\) of size \(n \times r\), and \(H\) of size \(r \times p\), the reconstruction \(WH\) and residual \(R\) must each be \(n \times p\). The gradient with respect to a matrix must have the same dimensions as that matrix: \(\nabla_W F\) is \(n \times r\), and \(\nabla_H F\) is \(r \times p\).

Checking a Gradient by a Small Perturbation

A gradient formula can be checked by changing one factor entry slightly and comparing the resulting change in the objective with the predicted first-order change. This is a numerical check, not a substitute for the derivation: the step must be small enough that the quadratic remainder is modest, yet not so small that rounding error dominates.

Interpreting Gradients on the Nonnegative Domain

The formulas above are ordinary derivatives of the smooth function \(F(W,H)\), considered in the surrounding unconstrained matrix space. NMF restricts feasible factors to be nonnegative. At a factor entry strictly above zero, a sufficiently small change in either direction is feasible; at a zero entry, a negative change is not. Consequently, a gradient is a local rate of change, but it is not always a feasible direction.

This distinction matters when assessing an optimization result. A zero gradient in the unconstrained sense is a useful stationarity condition for an interior point, but at a boundary point a constrained optimum need not have a zero ordinary gradient. Conversely, a small loss is not proof that the factorization is optimal: it only describes reconstruction quality at the current factors and rank. The constrained optimality conditions require additional reasoning beyond differentiating the loss.

Diagnostics, Alternatives, and Limitations

Several checks help reveal when the objective or its implementation is being misused:

• Check dimensions first. Confirm \(WH\) has the same dimensions as \(X\), and that each gradient has the dimensions of its factor. An incorrect transpose can sometimes produce a valid multiplication with the wrong interpretation.
• Check the residuals, not only their sum of squares. A moderate total loss can conceal a few large errors or systematic patterns across samples or features. Inspecting residuals can show whether particular entries dominate.
• Check scales and missingness. The unweighted loss treats every observed entry as present and equally weighted. Missing values cannot simply be inserted as zeros without changing the target; unequal scales or precision call for a different modeling choice.
• Check nonnegativity and the objective convention. The factor constraints are part of NMF, and a calculation using the unscaled loss has gradients twice those for \(F\). State which convention is used when comparing gradient values.

The Frobenius objective is a natural choice when squared additive deviations are an appropriate measure of reconstruction error. It is not universally appropriate for every nonnegative data type: counts, compositional measurements, and data with strongly unequal variances may call for other loss formulations or explicit weighting. Those alternatives change the objective and its derivatives, so their formulas should not be mixed with the Frobenius gradients derived here.

A common mistake is to reason, “The loss is a sum of squared residuals, so the gradient with respect to \(W\) is just \(2(WH-X)\).” This is wrong because \(W\) is not the reconstructed matrix: it affects the reconstruction through multiplication by \(H\). The chain rule contributes \(H^T\), giving \((WH-X)H^T\) for the half-scaled loss, or twice this for the unscaled loss. Similarly, differentiating with respect to \(H\) contributes \(W^T\) on the left.