What Is a Density Estimator Trying to Estimate?
Density estimation uses an i.i.d. sample to estimate the distribution that generated the observations. The target is a density function, not a parameter vector or a collection of labels. How accuracy is measured depends on the loss: two losses can assess the same estimate differently, and a loss that is finite for every pair of densities need not be finite for every pair under another criterion.
Work on \(\mathbb{R}^d\) with Lebesgue measure. Suppose \(X_1,\ldots,X_n\) are i.i.d. with density \(f\), where \(f\geq 0\), \(f\) is measurable, and \(\int_{\mathbb{R}^d} f(x)\,dx=1\). An estimator is a random function constructed from the sample. To define integrated risks rigorously, take \(\hat f_n(x)\) to be jointly measurable in the sample and \(x\). For a density estimator, require that with probability one, \(\hat f_n\) is nonnegative and integrates to one.
The density requirement matters: a random function that takes negative values or does not integrate to one may still be useful as an intermediate estimate, but it is not itself a density estimator under this definition. The losses below compare the estimate and target over the full domain, or at one specified point.
Four Ways to Measure Error
The integrated squared error (ISE) squares the difference at each location and accumulates it over the domain. It is a squared \(L^2\) distance and can be infinite, so a standard sufficient condition for finite ISE is that both \(f\) and \(\hat f_n\) belong to \(L^2\) with probability one.
The integrated absolute error (IAE) accumulates the absolute difference instead. Because \(f\) and \(\hat f_n\) are densities, each has integral one. Therefore IAE is at most two and is finite without an \(L^2\) assumption.
The squared Hellinger loss compares square roots of the densities. Here the convention is the unnormalized squared distance: there is no factor of \(1/2\). Its value lies between zero and two for any pair of densities. Some references use a factor of \(1/2\), so the convention should always be checked when comparing results.
A pointwise loss assesses error at a fixed location \(x\). For example, the squared pointwise loss and its risk are
The pointwise risk can be infinite if the relevant second moment does not exist. Unlike integrated losses, it only concerns one location. There is also a version issue: densities that agree almost everywhere represent the same distribution, but may have different assigned values at an individual point. Pointwise targets therefore require a chosen version of \(f\); assumptions such as continuity can make the intended value unambiguous.
Worked Example: Comparing Losses for Two Uniform Densities
Let \(f\) be uniform on \([0,1]\), and let \(g\) be uniform on \([0,2]\). Their density values are \(f(x)=1\) on \([0,1]\) and zero elsewhere, and \(g(x)=1/2\) on \([0,2]\) and zero elsewhere. Treat \(g\) as a fixed estimate.
Integrated squared error. On \([0,1]\), the difference is \(1-1/2=1/2\); on \((1,2]\), it is \(0-1/2=-1/2\). Thus \(\operatorname{ISE}(g,f)=1(1/2)^2+1(1/2)^2=1/2\).
Integrated absolute error. Each interval of length one contributes \(1/2\), so \(\operatorname{IAE}(g,f)=1(1/2)+1(1/2)=1\).
Squared Hellinger loss. On \([0,1]\), the squared difference of square roots is \((1-1/\sqrt{2})^2\). On \((1,2]\), it is \((1/\sqrt{2}-0)^2=1/2\). Therefore \(H^2(g,f)=(1-1/\sqrt{2})^2+1/2=2-\sqrt{2}\).
The three values differ because the losses weight local discrepancies differently. All are finite here; the same need not be true of ISE for arbitrary densities.
Risks Average Over Samples
A realized loss depends on the particular sample through \(\hat f_n\). Its risk averages that loss over repeated samples from the distribution with density \(f\). The expectation is with respect to the joint distribution of \(X_1,\ldots,X_n\), while the integral in an integrated loss is with respect to \(x\). Keeping these two operations distinct prevents a common confusion: a risk is not the loss for one observed data set.
For nonnegative losses, expectation and integration can be exchanged under joint measurability, even if the resulting quantity is infinite. This is an application of Tonelli's theorem.
Proof. Let \(S=(X_1,\ldots,X_n)\) and let \(P_S\) be its sampling distribution. Joint measurability makes \((s,x)\mapsto\phi(\hat f_n(s,x),f(x))\) a nonnegative measurable function on the product of the sample space and \(\mathbb{R}^d\). Tonelli's theorem for the product measure \(P_S\times dx\) gives \[ \int\!\left[\int \phi(\hat f_n(s,x),f(x))\,dx\right]dP_S(s) = \int\!\left[\int \phi(\hat f_n(s,x),f(x))\,dP_S(s)\right]dx. \] The left side is the expected integrated loss, and the right side is the integral of the pointwise expected loss. This proves the identity without requiring either side to be finite. \(\square\)
For instance, choosing \(\phi(a,b)=(a-b)^2\) shows that expected ISE equals the integral of pointwise squared risk, including when both are infinite. This identity does not make the two criteria interchangeable in every practical question: pointwise risk answers what happens at a specified \(x\), while integrated risk aggregates across the domain.
Worked Example: Calculating Risk From Two Possible Estimates
Let the true density be uniform on \([0,1]\), and let \(X_1\) be the first observation. Define an estimator to be \(g_1\), the uniform density on \([0,1]\), if \(X_1\leq 1/2\); otherwise define it to be \(g_2\), the uniform density on \([0,2]\). Since \(X_1\) is uniform on \([0,1]\), each estimate is selected with probability \(1/2\).
If \(g_1\) is selected, all four losses considered here are zero. If \(g_2\) is selected, the calculations in the preceding example give ISE \(1/2\), IAE \(1\), and squared Hellinger loss \(2-\sqrt{2}\). Therefore \[ R_{\mathrm{ISE}}=\tfrac12(0)+\tfrac12(\tfrac12)=\tfrac14,\quad R_{\mathrm{IAE}}=\tfrac12(0)+\tfrac12(1)=\tfrac12,\quad R_{\mathrm{H}}=\tfrac12(0)+\tfrac12(2-\sqrt{2})=1-\tfrac{\sqrt{2}}{2}. \]
At \(x=1/2\), both chosen density values are defined by their interval formulas. The target value is \(f(1/2)=1\); the estimates are \(g_1(1/2)=1\) and \(g_2(1/2)=1/2\). Thus the squared pointwise risk there is \[ R_{1/2}=\tfrac12(1-1)^2+\tfrac12(1/2-1)^2=\tfrac18. \] This example shows why a random estimator's risk requires averaging over its sampling-dependent outcomes.
How the Losses Compare
The losses are not unrelated. For any two densities \(f\) and \(g\), define \(H(f,g)=\sqrt{H^2(f,g)}\). Then
Proof. For nonnegative numbers \(a,b\), \(|\sqrt a-\sqrt b|\leq\sqrt{|a-b|}\). Squaring and applying this pointwise with \(a=f(x)\), \(b=g(x)\), then integrating, gives \(H^2(f,g)\leq\int|f-g|\,dx\).
For the other inequality, factor the difference: \[ \int |f-g|\,dx =\int|\sqrt f-\sqrt g|(\sqrt f+\sqrt g)\,dx. \] Cauchy–Schwarz bounds this by \[ H(f,g)\left(\int(\sqrt f+\sqrt g)^2\,dx\right)^{1/2}. \] The integral in parentheses is \(2+2\int\sqrt{fg}\,dx\). Cauchy–Schwarz gives \(\int\sqrt{fg}\,dx\leq(\int f)^{1/2}(\int g)^{1/2}=1\), so the parenthesized integral is at most \(4\). The square root is at most \(2\), proving \(\int|f-g|\,dx\leq 2H(f,g)\). \(\square\)
These bounds compare integrated absolute and Hellinger distances, but they do not turn either into ISE: the latter uses squared density differences and may be infinite even when the other two losses are finite. Expected versions of the displayed bounds also follow by taking expectations, with the rightmost bound involving \(E[H]\), not necessarily the square root of \(E[H^2]\).
Worked Example: A Density With Infinite Integrated Squared Error
On \((0,1)\), define \(f(x)=1/(2\sqrt{x})\), and set it to zero outside that interval. It is a density because \[ \int_0^1 \frac{1}{2\sqrt{x}}\,dx =\left[\sqrt{x}\right]_0^1=1. \] Let \(g\) be the uniform density on \([0,1]\), so \(g(x)=1\) there.
For \(0<x\leq1/16\), \(f(x)\geq2\), and hence \(f(x)-1\geq f(x)/2\). It follows that \[ (f(x)-g(x))^2\geq \frac{f(x)^2}{4}=\frac{1}{16x}. \] Since \(\int_0^{1/16}1/(16x)\,dx=\infty\), the integrated squared error between these densities is infinite.
Their integrated absolute error is nevertheless finite: for any two densities, \(\int|f-g|\leq\int(f+g)=2\). Their squared Hellinger loss is finite as well, since it is at most the integrated absolute error by the result just proved. Thus being a valid density does not imply being square-integrable, and finite ISE requires additional conditions.
Common Pitfalls and Practical Meaning
- Assuming all losses are finite. Every pair of densities has finite integrated absolute and squared Hellinger losses, but ISE may be infinite. Check square-integrability when finite ISE is required.
- Confusing loss with risk. Loss is computed for a realized estimate; risk averages over the sampling distribution. A single observed estimate does not by itself give its theoretical risk.
- Treating pointwise error as a full-distribution comparison. Agreement at one \(x\) does not imply agreement elsewhere. Moreover, pointwise evaluation depends on the chosen versions of the densities.
- Overlooking the Hellinger convention. A factor of \(1/2\) changes the numerical loss and risk. State whether the squared distance is normalized.
The loss is part of the statistical question, not merely a calculation chosen after estimation. Integrated losses assess the estimate across the domain; pointwise risk focuses attention on a location. Different choices can therefore prioritize different kinds of accuracy. A meaningful comparison states the target density, the loss convention, and the assumptions needed for the risk to exist or be finite.
Check Your Understanding
Answer each question using the definitions and assumptions stated above.
- What properties must a sample-dependent function satisfy to be a density estimator?
- Which of the four losses can be infinite for a pair of densities without additional assumptions? Give the relevant sufficient condition for finiteness.
- For the squared Hellinger convention used here, what is the range of the loss for two densities, and where might a factor-of-two difference arise?
- Why does joint measurability matter when exchanging expectation and integration in an expected integrated loss?
- Why can two versions of the same density give different pointwise losses at a specified \(x\)?