Generalized KL Divergence for NMF

Why Use a Different Reconstruction Loss?

Nonnegative matrix factorization represents a nonnegative data matrix $X$ by a product $WH$, where the entries of both factors are constrained to be nonnegative. The choice of reconstruction loss determines which discrepancies between $X$ and $WH$ are penalized most. Squared error penalizes the square of each absolute residual. Generalized Kullback–Leibler (KL) divergence instead gives a loss that depends on the observed value relative to the fitted value. This can be useful when values are nonnegative counts or intensities and proportional discrepancies matter.

The loss is defined entry by entry and then summed over the matrix. Its behavior at zero matters: omitting or mishandling a zero can change the objective, and a positive observation paired with a zero fitted value has an infinite loss. The divergence is therefore not simply a formula to apply mechanically; its domain and zero conventions are part of its definition.

Definition: Let $X$ be an $m\times n$ matrix with nonnegative entries, and let $Y$ be a nonnegative matrix of the same dimensions. The generalized KL divergence from $Y$ to $X$ is $$ D_{\mathrm{KL}}(X\|Y)=\sum_{i=1}^{m}\sum_{j=1}^{n}\left[x_{ij}\log\left(\frac{x_{ij}}{y_{ij}}\right)-x_{ij}+y_{ij}\right]. $$ Here $\log$ is the natural logarithm. For each entry, define $0\log(0/y)=0$ when $y\geq 0$. If $x_{ij}=y_{ij}=0$, that entry contributes 0; if $x_{ij}=0$ and $y_{ij}>0$, it contributes $y_{ij}$; and if $x_{ij}>0$ and $y_{ij}=0$, it contributes $+\infty$.

In NMF, $Y=WH$, with $W$ of size $m\times r$, $H$ of size $r\times n$, and both factors nonnegative. The generalized KL NMF objective is to minimize $D_{\mathrm{KL}}(X\|WH)$ over such factors. A finite objective requires $ (WH)_{ij}>0$ wherever $x_{ij}>0$; fitted values may be zero at observed zeros. The divergence is nonnegative and is zero exactly when $X=Y$, but it is not a distance: in particular, exchanging $X$ and $Y$ generally changes its value.

Zero Entries and Entrywise Contributions

For a single observed value $x$ and fitted value $y$, write the contribution as $d(x,y)=x\log(x/y)-x+y$. The zero conventions follow from the limiting behavior of this expression. If $x=0$ and $y>0$, the logarithmic term is defined as zero, leaving $d(0,y)=y$. If both values are zero, the contribution is zero. If $x>0$, taking $y$ toward zero makes $\log(x/y)$ grow without bound, so $d(x,y)$ tends to infinity.

These cases have distinct interpretations. A zero observation is not automatically ignored: predicting a positive value for it incurs a penalty equal to that fitted value. A positive observation cannot be matched by an exact zero prediction at finite loss. In computation, strictly positive initial factors commonly produce positive fitted values, but the mathematical definition still matters when the data or fitted reconstruction contains zeros.

Worked Example: Summing Entrywise KL Contributions

Suppose a hypothetical nonnegative data matrix and candidate reconstruction are $$ X=\begin{pmatrix}12&0\\8&5\end{pmatrix}, \qquad Y=\begin{pmatrix}10&2\\10&4\end{pmatrix}. $$ Calculate $D_{\mathrm{KL}}(X\|Y)$, rounding contributions and the total to four significant figures.

The four entrywise contributions are as follows. For $x=12,y=10$, $12\log(12/10)-12+10=12\log(1.2)-2\approx 0.1879$. For $x=0,y=2$, the zero convention gives $d(0,2)=2$. For $x=8,y=10$, $8\log(8/10)-8+10=8\log(0.8)+2\approx 0.2149$. For $x=5,y=4$, $5\log(5/4)-5+4=5\log(1.25)-1\approx 0.1157$.

Adding the unrounded contributions gives $D_{\mathrm{KL}}(X\|Y)\approx 0.1878587+2+0.2148516+0.1157178=2.518$. The divergence is finite because every positive entry of $X$ has a positive fitted value in $Y$. The observed zero still contributes 2 because the reconstruction assigns it a value of 2.

How the Loss Weights Residuals

Generalized KL divergence is not a uniform penalty on absolute residuals. To see its behavior near a good fit, hold $y>0$ fixed and write $x=y+\delta$. For small $\delta$, the Taylor expansion of $\log(1+\delta/y)$ gives

\begin{aligned} d(y+\delta,y) &=(y+\delta)\log(1+\delta/y)-\delta\\ &=\frac{\delta^2}{2y}-\frac{\delta^3}{6y^2}+O\left(\frac{\delta^4}{y^3}\right). \end{aligned}

Thus, for small residuals, the leading term is approximately $(x-y)^2/(2y)$: squared residuals are weighted inversely by the fitted value. For the same small absolute residual, a discrepancy around a small fitted value generally contributes more than one around a large fitted value. This is a local approximation, not an exact description for large discrepancies or near $y=0$.

Worked Example: Equal Absolute Residuals, Different KL Losses

Compare two hypothetical scalar observations with the same fitted value $y=10$: $x=12$ and $x=8$. Both have absolute residual 2, so each has squared error $(x-y)^2=4$.

Under generalized KL divergence, the overprediction-side observation $x=12$ contributes $d(12,10)=12\log(1.2)-2\approx 0.1879$. The underprediction-side observation $x=8$ contributes $d(8,10)=8\log(0.8)+2\approx 0.2149$.

The absolute residuals and squared errors are identical, but the KL contributions differ. This asymmetry is a feature of the divergence, not an arithmetic inconsistency. It also cautions against saying that KL loss depends only on relative error: its exact entrywise formula depends on both $x$ and $y$, and its local weighting is a statement near $x=y$.

The comparison with squared error can be summarized this way:

Property	Squared error	Generalized KL divergence
Entrywise loss	$(x-y)^2$	$x\log(x/y)-x+y$, with zero conventions
Near a good fit	Exactly the squared residual	Approximately $(x-y)^2/(2y)$
Behavior at an observed zero	Contribution $y^2$	Contribution $y$
Positive observation with fitted value zero	Finite contribution $x^2$	Infinite contribution

The scale behavior also differs. For a positive scalar $c$, $D_{\mathrm{KL}}(cX\|cY)=cD_{\mathrm{KL}}(X\|Y)$, whereas the sum of squared residuals scales by $c^2$. Consequently, changing the scale of a data block can affect comparisons between loss functions and between blocks; the numerical value of a KL objective is not scale-free.

Interpretation and Assumptions

The divergence is a valid mathematical loss for nonnegative matrices. Its use is particularly interpretable for count data under a specific model. If the entries are conditionally independent Poisson random variables with means $y_{ij}>0$, then the negative log-likelihood, up to a term depending only on the observed data, equals $D_{\mathrm{KL}}(X\|Y)$. For integer observations, the negative log-likelihood is $$ -\ell(Y;X)=\sum_{i,j}\left[y_{ij}-x_{ij}\log(y_{ij})+\log(x_{ij}!)\right]. $$ Subtracting the divergence leaves a quantity that depends on $X$ but not on $Y$, so minimizing the divergence over candidate means is equivalent to maximizing this Poisson likelihood. When $x_{ij}=0$, the likelihood contribution is $y_{ij}$, consistent with the divergence convention.

This likelihood interpretation requires the Poisson assumptions, including conditional independence and the specified means. It does not follow merely because the matrix contains nonnegative numbers. The generalized KL objective can also be used as a discrepancy for non-count data, but then it should not automatically be described as a Poisson likelihood. It is unsuitable without modification for data containing negative values, as can happen after centering or some standardization procedures.

For NMF, the factors restrict the fitted matrix $WH$ to be nonnegative and of rank at most $r$. The divergence scores how well a candidate reconstruction fits under the chosen loss; it does not establish that a factorization is unique, that its components have a causal interpretation, or that the data-generating process is Poisson. For a fixed $X$, comparing reconstructions using the same objective is meaningful, but raw objective values from differently scaled data sets need not be comparable.

Checking the Objective and Its Limitations

Before optimizing or interpreting a generalized KL objective, check the data and the fitted values directly. Useful diagnostics include:

Domain check: confirm that $X$ and $WH$ are nonnegative. A negative entry is outside the definition given here.
Positive-observation check: verify that every positive $x_{ij}$ has a positive fitted value. An infinite contribution signals a zero fit at a positive observation.
Contribution check: inspect entrywise losses, especially large contributions and zero observations assigned substantial fitted values. A finite total can still be dominated by a small number of entries.
Scale check: check whether a few large-scale rows, columns, or blocks dominate the sum. The loss is additive, so a block with larger magnitudes can have a larger total contribution.
Model check: if using a Poisson interpretation, assess whether conditional independence and the mean structure are plausible. Strong overdispersion or dependence weakens that interpretation.

Generalized KL divergence is not universally preferable to squared error. Squared error may be a more natural choice when additive Gaussian-like errors with roughly constant variance are plausible. KL loss is often attractive when proportional discrepancies or a Poisson mean model are relevant. Other nonnegative-data losses exist, but selecting among them requires assumptions about the data and the scientific goal; the smallest numerical objective is not by itself evidence that one loss is appropriate.

Common pitfall: A common mistake is to call an observed zero a missing value and drop its contribution. That is incorrect for the stated objective: if $x=0$ and $y>0$, then $d(0,y)=y$. Another mistake is to assume that equal absolute residuals must have equal KL loss; the worked comparison shows otherwise. Apply the zero conventions, calculate the entrywise contributions, and state the loss assumptions before interpreting the result.

Practical Workflow

Check the matrix domain.
Confirm that the observations are nonnegative and clarify how zeros are represented. Do not apply this objective directly to negative, centered data.

Form a nonnegative reconstruction.
For NMF, use $Y=WH$ with compatible dimensions and nonnegative factors. Ensure positive fitted values wherever observations are positive if a finite divergence is required.

Evaluate and inspect the loss.
Compute each contribution using the zero conventions, sum them, and inspect large terms rather than relying only on the total.

Interpret in context.
Describe the objective as a reconstruction discrepancy, or as a Poisson likelihood-equivalent criterion only when the corresponding model assumptions are justified.

Check Your Understanding

Answer each question, then compare with the short solutions below.

What is the generalized KL contribution when the observed value is zero and the fitted value is 3?
What happens to the divergence contribution when the observed value is positive and the fitted value is zero?
Near a good fit, what weight does generalized KL divergence place on the squared residual?
Why does a Poisson likelihood interpretation require more than nonnegative observations?

Solutions. (1) $d(0,3)=3$, by the zero convention. (2) The contribution is infinite, since $x\log(x/y)$ diverges as $y$ tends to zero for $x>0$. (3) For $x$ close to $y>0$, $d(x,y)\approx (x-y)^2/(2y)$, so the leading residual-square weight is $1/(2y)$. (4) The likelihood interpretation assumes a specified probabilistic model, including conditionally independent Poisson observations with means given by the fitted entries; nonnegativity alone does not imply those assumptions.

Property	Squared error	Generalized KL divergence
Entrywise loss	\((x-y)^2\)	\(x\log(x/y)-x+y\), with zero conventions
Near a good fit	Exactly the squared residual	Approximately \((x-y)^2/(2y)\)
Behavior at an observed zero	Contribution \(y^2\)	Contribution \(y\)
Positive observation with fitted value zero	Finite contribution \(x^2\)	Infinite contribution