Hellinger Loss for Density Estimation

Why Use Hellinger Loss?

Density estimators are often compared by integrated squared error or integrated absolute error. Squared Hellinger loss offers another scale: it compares the square roots of the densities. This can be useful when densities vary greatly in size, and it gives direct control of integrated absolute error without requiring a uniform bound on the densities.

There is an important qualification. A Hellinger guarantee does not, by itself, imply a useful bound on integrated squared error. For that conversion, we will need a uniform upper bound on both densities. The distinction matters when interpreting risk bounds for an estimator: the loss in which a guarantee is stated determines what can be concluded.

Definition: Let $p$ and $q$ be nonnegative densities with integral one with respect to the same measure $\mu$. The squared Hellinger loss used here is $$ H^2(p,q)=\int \bigl(\sqrt{p(x)}-\sqrt{q(x)}\bigr)^2\,d\mu(x). $$ The corresponding Hellinger distance is $H(p,q)=\sqrt{H^2(p,q)}$. Some sources define squared Hellinger loss with an additional factor of $1/2$; this tutorial uses the unhalved convention above.

The loss is nonnegative and is zero exactly when the densities agree $\mu$-almost everywhere. By Cauchy–Schwarz, $\int\sqrt{pq}\,d\mu\leq 1$, so an equivalent expression is

H^2(p,q)=2-2\int\sqrt{p(x)q(x)}\,d\mu(x).

Thus the loss is small when the two densities have substantial overlap on the square-root scale. The convention matters for numerical constants: using the alternative factor $1/2$ halves every squared-Hellinger value and changes the constants in inequalities accordingly.

Worked Comparisons on a Simple Support

Worked Example: Hellinger, Absolute, and Squared Error

Let $p(x)=1$ on $[0,1]$. Define $q(x)=1.5$ on $[0,0.5)$, $q(x)=0.5$ on $[0.5,1]$, and both densities to be zero outside $[0,1]$. These are densities because $\int p=1$ and $\int q=0.5(1.5)+0.5(0.5)=1$.

On the first half of the interval, the square-root difference is $1-\sqrt{1.5}$; on the second it is $1-\sqrt{0.5}$. Therefore

\begin{aligned} H^2(p,q) &=0.5(1-\sqrt{1.5})^2+0.5(1-\sqrt{0.5})^2\\ &\approx 0.0252551+0.0428932\\ &\approx 0.0681483. \end{aligned}

The integrated absolute error is $\int|p-q|=0.5(0.5)+0.5(0.5)=0.5$. The integrated squared error is $\int(p-q)^2=0.5(0.5^2)+0.5(0.5^2)=0.25$. These quantities are distinct, even though each compares the same pair of densities. Since both densities are bounded above by $M=1.5$, the bounded-density inequality proved below gives $0.25\leq 4(1.5)(0.06815)=0.40890$, using the displayed rounded Hellinger loss.

Hellinger Loss and Integrated Absolute Error

For densities $p$ and $q$, define their integrated absolute error as $\|p-q\|_1=\int|p-q|\,d\mu$. Their total variation distance is $\operatorname{TV}(p,q)=\tfrac12\|p-q\|_1$. The next result gives both an upper and a lower comparison with squared Hellinger loss.

Theorem: For any two densities $p,q$ with respect to a common measure, $$ \frac12 H^2(p,q)\leq \operatorname{TV}(p,q)\leq H(p,q), $$ and consequently $$ H^2(p,q)\leq \|p-q\|_1\leq 2H(p,q). $$

Proof. For nonnegative numbers $a,b$, we have $(\sqrt a-\sqrt b)^2\leq |a-b|$. Indeed, $|a-b|=|\sqrt a-\sqrt b|(\sqrt a+\sqrt b)$, and $\sqrt a+\sqrt b\geq|\sqrt a-\sqrt b|$. Integrating this pointwise inequality gives $H^2(p,q)\leq\|p-q\|_1$, or $H^2(p,q)/2\leq\operatorname{TV}(p,q)$.

For the other direction, factor the absolute difference and apply Cauchy–Schwarz:

\begin{aligned} \|p-q\|_1 &=\int|\sqrt p-\sqrt q|(\sqrt p+\sqrt q)\,d\mu\\ &\leq \left(\int(\sqrt p-\sqrt q)^2\,d\mu\right)^{1/2} \left(\int(\sqrt p+\sqrt q)^2\,d\mu\right)^{1/2}. \end{aligned}

The second integral is $2+2\int\sqrt{pq}\,d\mu\leq4$, because $\int\sqrt{pq}\,d\mu\leq(\int p\,d\mu)^{1/2}(\int q\,d\mu)^{1/2}=1$. Hence $\|p-q\|_1\leq2H(p,q)$, which gives $\operatorname{TV}(p,q)\leq H(p,q)$. $\square$

The two sides express different directions of control. A small Hellinger loss guarantees a small integrated absolute error, with a square root in the bound. Conversely, small integrated absolute error guarantees small squared Hellinger loss. These inequalities do not say the losses are numerically equal or interchangeable without changing their scale.

Worked Example: Checking the Absolute-Error Bounds

For the densities in the first example, $H^2\approx0.0681483$, so $H\approx\sqrt{0.0681483}\approx0.2610523$. The theorem gives

0.0681483\leq \|p-q\|_1=0.5\leq 2(0.2610523)\approx0.522105.

Equivalently, $0.0340742\leq\operatorname{TV}(p,q)=0.25\leq0.2610523$. Both bounds hold. Notice that the upper bound is informative but not an equality: the theorem converts a loss guarantee into a bound, not an exact formula for the other loss.

When Hellinger Loss Controls Squared Error

Integrated squared error is $\|p-q\|_2^2=\int(p-q)^2\,d\mu$. The identity $(p-q)^2=(\sqrt p-\sqrt q)^2(\sqrt p+\sqrt q)^2$ shows why a bound on the densities is relevant: it limits the second factor.

Theorem: Suppose $p$ and $q$ are densities and $p(x)\leq M$, $q(x)\leq M$ for $\mu$-almost every $x$, for some finite $M$. Then $$ \|p-q\|_2^2\leq 4M H^2(p,q). $$

Proof. The hypotheses imply $(\sqrt p+\sqrt q)^2\leq(2\sqrt M)^2=4M$ almost everywhere. Pointwise, therefore,

(p-q)^2 =(\sqrt p-\sqrt q)^2(\sqrt p+\sqrt q)^2 \leq 4M(\sqrt p-\sqrt q)^2.

Integrating both sides with respect to $\mu$ gives $\int(p-q)^2\,d\mu\leq4M\int(\sqrt p-\sqrt q)^2\,d\mu=4M H^2(p,q)$. The assumptions ensure this conclusion even when the densities are defined on a support with infinite measure. $\square$

Worked Example: Translating a Hellinger Guarantee

Suppose a density estimator $\hat f$ and the target density $f$ are both bounded above by $M=2$, and a deterministic guarantee establishes $H^2(\hat f,f)\leq0.00625$. The theorem gives

\int(\hat f-f)^2\leq4(2)(0.00625)=0.05.

Thus the integrated squared error is at most $0.05$, and its square root is at most $\sqrt{0.05}\approx0.223607$. Without using the density bound, the absolute-error theorem still gives $\int|\hat f-f|\leq2\sqrt{0.00625}\approx0.158114$, and hence $\operatorname{TV}(\hat f,f)\leq\sqrt{0.00625}\approx0.079057$. The squared-error conclusion depends on the uniform bound; the absolute-error conclusion does not.

Interpreting Estimator Risk

A density estimator must itself be a nonnegative density for these Hellinger comparisons to apply. In particular, an estimator that can take negative values is not a density until it has been converted into one by a procedure that preserves nonnegativity and unit integral. Once $\hat f$ is a density almost surely, the deterministic inequalities above can be applied for each realized sample.

For example, suppose $E[H^2(\hat f,f)]\leq r$, where $f$ is the target density and the expectation is over the data. The absolute-error inequality and Cauchy–Schwarz imply

\begin{aligned} E\!\left[\int|\hat f-f|\right] &\leq 2E[H(\hat f,f)]\\ &\leq 2\sqrt{E[H^2(\hat f,f)]} \leq 2\sqrt r. \end{aligned}

The second inequality follows by applying Cauchy–Schwarz to the random variable $H(\hat f,f)$. If both $\hat f$ and $f$ are bounded above by the same fixed $M$ almost surely, then the bounded-density result also yields $E[\int(\hat f-f)^2]\leq4Mr$. A risk statement in expectation therefore transfers to another expected loss, provided the needed assumptions hold.

Worked Example: Why a Density Bound Cannot Be Dropped

Let $f(x)=1$ on $[0,1]$ and zero elsewhere. For $0<\alpha<1$ and $\delta>0$, define $g$ to equal $1-\alpha$ on $[0,1]$, $\alpha/\delta$ on $[2,2+\delta]$, and zero elsewhere. Its integral is $(1-\alpha)+(\alpha/\delta)\delta=1$, so $g$ is also a density.

Take $\alpha=0.01$ and $\delta=10^{-6}$. On $[0,1]$, the squared-root difference is $(1-\sqrt{0.99})^2$; on $[2,2+\delta]$, it is $\alpha/\delta$. Thus

\begin{aligned} H^2(f,g) &=(1-\sqrt{0.99})^2+\delta\frac{0.01}{10^{-6}}\\ &\approx0.000025126+0.01\\ &\approx0.010025126. \end{aligned}

But their integrated squared error is

\begin{aligned} \int(f-g)^2 &=1(0.01)^2+\delta\left(\frac{0.01}{10^{-6}}\right)^2\\ &=0.0001+100\\ &=100.0001. \end{aligned}

The Hellinger loss is small while the squared error is large. The spike in $g$ reaches height $10{,}000$, so there is no fixed, small common upper bound on these densities. This example illustrates why a Hellinger guarantee alone cannot be converted into a useful integrated squared-error guarantee.

Common Pitfalls and Practical Use

Mixing Hellinger conventions. Some definitions include a factor of $1/2$, while this tutorial does not. Check the convention before comparing values or constants.
Treating the losses as equal. The inequalities compare scales; they do not identify Hellinger, absolute, and squared error. In particular, the absolute-error bound involves $H=\sqrt{H^2}$, not $H^2$.
Converting to squared error without checking boundedness. The bound $\|p-q\|_2^2\leq4M H^2(p,q)$ requires both densities to be bounded by the same finite $M$. A Hellinger guarantee alone is not enough.
Applying density losses to a signed estimate. Square roots of a negative estimate are not defined as real-valued functions. Confirm that the estimator is a nonnegative, normalized density before using these results.

Hellinger loss is useful because it gives a stable way to compare densities and a direct route to integrated absolute error, even when the densities are not uniformly bounded. When interpreting a rate or risk bound, state the loss convention, identify whether the guarantee is deterministic or in expectation, and check the assumptions required for any conversion to squared error.

Check Your Understanding

Answer each question using the unhalved Hellinger convention defined above.

For two densities $p$ and $q$, what is the definition of $H^2(p,q)$, and what alternative convention is sometimes used?
If $H^2(p,q)\leq r$, give an upper bound on $\|p-q\|_1$ and on $\operatorname{TV}(p,q)$.
State the assumptions needed to conclude $\|p-q\|_2^2\leq4M H^2(p,q)$.
Suppose $E[H^2(\hat f,f)]\leq r$. What upper bound follows for $E[\int|\hat f-f|]$, assuming $\hat f$ is a density almost surely?
Why does a small Hellinger loss not necessarily imply a small integrated squared error when the densities are unbounded?