Why the Loss Function Matters
Nonnegative matrix factorization represents a nonnegative data matrix \(X\) by a product \(WH\), whose entries are fitted values for the observed entries of \(X\). The factorization alone does not specify what counts as a good approximation: the loss function sets the discrepancy that the fit is asked to reduce. For data whose entries vary greatly in magnitude, measuring discrepancies only by absolute differences can give a different result from measuring relative differences.
The beta-divergence family provides a continuous way to vary that discrepancy. It includes squared error, generalized Kullback–Leibler (KL) divergence, and Itakura–Saito divergence as particular cases or limits. The parameter \(\beta\) is a choice about how differences across scales are weighted; it is not, by itself, a claim that a particular probability model generated the data.
The fitted value is kept strictly positive in this definition. This avoids undefined logarithms and powers at \(y=0\), and it is the usual domain for evaluating the loss. In an NMF objective, one commonly sums the entrywise losses \(d_\beta(X_{ij}\mid (WH)_{ij})\), provided the fitted entries are positive. Although the factors \(W\) and \(H\) are nonnegative, their product can have zero entries; those entries therefore require attention when using losses whose domain requires \(y>0\).
Special Cases and Limits
At \(\beta=2\), substitution into the general expression gives half the squared difference. At \(\beta=1\), the expression is generalized KL divergence, also called the I-divergence. At \(\beta=0\), it is the Itakura–Saito divergence. The limiting cases matter because the general formula has a zero denominator at \(\beta=0\) and \(\beta=1\); the definitions there are limits, not direct substitutions.
Here is a derivation of the KL limit. Write \(\beta=1+t\) and let \(t\) approach zero. Using \(a^t=1+t\log a+o(t)\) for \(a>0\), the numerator of \(d_{1+t}(x\mid y)\) becomes
The denominator is \((1+t)t=t+o(t)\), so the ratio tends to \(x\log(x/y)-x+y\). This argument applies for \(x>0\); the value at \(x=0\) follows by the continuous extension, which gives \(d_1(0\mid y)=y\).
For the Itakura–Saito limit, take \(x>0\) and expand around \(\beta=0\). The numerator has first-order term \(\beta\{\log(x/y)+1-x/y\}\), while the denominator has first-order term \(-\beta\). Their ratio tends to \(x/y-\log(x/y)-1\). When \(x\) tends down to zero, this limit diverges, explaining why \(d_0(0\mid y)=+\infty\).
For \(\beta\ne 0,1\) and positive \(x,y\), put \(t=x/y\). Then \(d_\beta(x\mid y)=y^\beta f_\beta(t)\), where
If \(0<t<1\), the derivative is negative; if \(t>1\), it is positive. Therefore \(f_\beta\) has its minimum at \(t=1\), where \(f_\beta(1)=0\). Since \(y^\beta>0\), the claimed nonnegativity and equality condition follow. The limiting cases have the same property: their derivatives with respect to \(t=x/y\) are \(\log t\) for KL and \(1/t-1\) for Itakura–Saito, each changing from negative to positive at \(t=1\).
Worked Examples
Worked Example: Squared Error at \(\beta=2\)
Suppose, in a hypothetical NMF fit, four nonnegative abundance measurements are \(X=(4,1,2,3)\) and the corresponding fitted values are \(\hat X=(3,2,2,1)\), in arbitrary abundance units. Calculate the summed beta-divergence at \(\beta=2\).
All fitted values are positive, so the loss is defined. Since \(d_2(x\mid y)=\frac12(x-y)^2\), the four contributions are
The total loss is 3 squared abundance units. The entry with the largest absolute residual contributes the most here. Some implementations omit the factor \(1/2\); that doubles the objective value but does not change which fit minimizes it.
Worked Example: Generalized KL Loss
For a second hypothetical fit, consider two observed abundances \(x=(4,2)\) and fitted values \(y=(3,1)\). Both fitted values are positive, and the observations are nonnegative, so the generalized KL loss is defined.
For the first entry, \(d_1(4\mid 3)=4\log(4/3)-4+3\approx 0.1507\). For the second, \(d_1(2\mid 1)=2\log(2)-2+1\approx 0.3863\). Therefore,
The total is approximately 0.5370 abundance units. Each logarithm uses a ratio, so its argument is dimensionless; the loss itself scales with the abundance measurements.
Worked Example: Scale Invariance and a Zero Observation
Suppose hypothetical positive measurements are \(x=(2,8)\), with fitted values \(y=(1,4)\). At \(\beta=0\), each observed-to-fitted ratio is 2, so
The sum is \(0.6137\). If every observation and fitted value is multiplied by 10, both ratios remain 2, and the sum remains \(0.6137\). But if an observed value is zero while its fitted value is positive, the Itakura–Saito loss is infinite. This loss is therefore unsuitable without modification when the data include zeros that must be fit with finite objective value.
Choosing a Regime and Checking the Fit
The choice of \(\beta\) should reflect what differences matter for the application and what values occur in the data. At \(\beta=2\), squared error penalizes absolute residuals, so a fixed absolute discrepancy at a large value can be important. At \(\beta=1\), generalized KL is finite for zero observations with positive fits and is often used when discrepancies are naturally assessed relative to fitted abundance. It is not automatically a full probability model: an objective alone does not establish that the entries are independent Poisson observations. At \(\beta=0\), Itakura–Saito is invariant to common rescaling of observations and predictions, making it useful when relative or multiplicative discrepancies are central and observations are positive.
More generally, the scaling identity for positive \(x,y\) is
for \(c>0\). The endpoint identities follow directly from their formulas; for the general case, each term in the numerator acquires a factor \(c^\beta\). Thus \(\beta=0\) is scale invariant, while other choices cause the numerical loss to change with the measurement scale. A change of units can affect the relative influence of blocks or features when their losses are combined.
A useful diagnostic is to compute entrywise loss contributions and inspect them by feature and magnitude. If a few high-abundance entries dominate under squared error, that may be intended, or it may indicate that absolute residuals are not the target. If beta values near zero are selected but the matrix contains zeros, infinite contributions signal a domain mismatch rather than a numerical optimizer failure. Also check that all fitted entries are positive where required, and verify that rescaling the data does not silently change the intended interpretation of the objective.
| Choice | Typical discrepancy emphasis | Boundary or scale consideration |
|---|---|---|
| \(\beta=2\), squared error | Absolute residual size | Finite at zero observations; loss scales quadratically under common rescaling |
| \(\beta=1\), generalized KL | Relative discrepancy with a finite zero-observation contribution | Finite for \(x=0,y>0\); loss scales linearly |
| \(\beta=0\), Itakura–Saito | Multiplicative or relative discrepancy | Scale invariant, but infinite at \(x=0,y>0\) |
Intermediate values can provide a compromise, but do not eliminate the need to assess the scale and zeros in the data. These losses are alternatives, not interchangeable labels for the same fit: changing \(\beta\) changes the objective and can change the resulting factors. Likewise, a beta-divergence objective does not on its own address missingness, outliers, dependence, or whether negative-valued measurements are scientifically meaningful. It is defined for nonnegative observations and should not be applied to signed data without a justified transformation or model.
Common Mistakes
- Using the general formula at an excluded boundary. For \(\beta<0\), \(x^\beta\) is undefined at \(x=0\). The correct finite-domain formula requires \(x>0\); if zero observations are included, the extended-real value is \(+\infty\).
- Assuming all zero observations are handled alike. For \(\beta>0\), \(d_\beta(0\mid y)=y^\beta/\beta\); for \(\beta\leq 0\), the value is infinite. Check the chosen loss against the data before fitting.
- Calling every beta-divergence a likelihood. A divergence defines a loss. A probabilistic interpretation requires additional distributional and sampling assumptions.
- Comparing objective values across different scales or beta choices as if they were directly comparable. Their scale behavior differs, so a smaller numerical value need not indicate a better scientifically relevant fit.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For \(x=0\), \(y=2\), and \(\beta=2\), calculate \(d_2(x\mid y)\). What is \(d_0(x\mid y)\)?
- For \(x=6\) and \(y=3\), calculate the Itakura–Saito divergence to three significant figures.
- What happens to a beta-divergence objective at \(\beta=0\) if all observations and fitted values are multiplied by the same positive constant?
- Why does a finite generalized KL loss at a zero observation not prove that the data arose from independent Poisson sampling?
Solutions. (1) \(d_2(0\mid2)=\frac12(0-2)^2=2\). At \(\beta=0\), the observed value is zero and the fitted value is positive, so \(d_0(0\mid2)=+\infty\). (2) \(d_0(6\mid3)=6/3-\log(6/3)-1=1-\log2\approx0.307\). (3) The objective is unchanged because each entrywise ratio \(x/y\) is unchanged, so \(d_0(cx\mid cy)=d_0(x\mid y)\). (4) A loss specifies an optimization criterion, not a sampling model; independent Poisson sampling would require separate assumptions about the distribution and dependence of the observations.