Pointwise Risk and Bias–Variance

Why Risk at One Location Deserves Its Own Definition

A density estimator is a random function, but a particular scientific question may concern its value at just one location $x$. For example, a density estimate near a clinical measurement threshold may matter more than its accuracy in a distant tail. The error at that location is random across repeated samples, so its average squared size gives a natural measure of performance.

This pointwise measure is related to integrated squared error, which aggregates squared error over locations. They answer different questions: pointwise risk describes performance at a specified $x$, whereas integrated risk accounts for errors throughout the domain. A small risk at one location does not imply a small integrated risk, and an integrated summary can conceal where errors occur.

Definition: Let $f$ be a specified version of the true density, and let $\hat f_n$ be a random density estimator. At a fixed, nonrandom location $x$, its pointwise mean squared error, or pointwise risk under squared-error loss, is $$ R_n(x)=E\left[(\hat f_n(x)-f(x))^2\right], $$ provided this expectation exists. Its pointwise bias is $b_n(x)=E[\hat f_n(x)]-f(x)$, and its pointwise variance is $\operatorname{Var}(\hat f_n(x))$.

The location $x$ is held fixed while the data vary. Thus, $R_n(x)$ is an expectation over repeated samples, not an average over locations. For the definition and decomposition below, assume $\hat f_n(x)$ has a finite second moment. The target $f(x)$ is a fixed finite number for the chosen density version.

The Pointwise Bias–Variance Decomposition

At one location, the squared error can be split into a random deviation from the estimator's own mean and the estimator's systematic offset from the target. Taking expectations makes the cross term vanish.

Theorem (pointwise bias–variance decomposition): If $E[\hat f_n(x)^2]<\infty$ and $f(x)$ is finite, then $$ R_n(x)=\operatorname{Var}(\hat f_n(x))+\left(E[\hat f_n(x)]-f(x)\right)^2 =\operatorname{Var}(\hat f_n(x))+b_n(x)^2. $$

Proof. Write $\hat f_n(x)-f(x)=(\hat f_n(x)-E[\hat f_n(x)])+b_n(x)$. Squaring and taking expectations gives

\begin{aligned} R_n(x) &=E\left[(\hat f_n(x)-E[\hat f_n(x)]+b_n(x))^2\right]\\ &=E\left[(\hat f_n(x)-E[\hat f_n(x)])^2\right] +2b_n(x)E[\hat f_n(x)-E[\hat f_n(x)]]+b_n(x)^2\\ &=\operatorname{Var}(\hat f_n(x))+b_n(x)^2, \end{aligned}

because $E[\hat f_n(x)-E[\hat f_n(x)]]=0$. The finite second moment ensures these expectations exist. $\square$

The equality is exact at each fixed location. It does not say that either contribution is constant across $x$: both the bias and variance can change with location. In particular, an estimator can be unbiased at one location and biased at another.

Worked Example: An Unbiased Estimate in a Two-Component Model

Let $g_0$ be the uniform density on $[0,1]$ and $g_1$ the uniform density on $[2,3]$. Suppose the target density is $f_p=(1-p)g_0+pg_1$, where $0<p<1$. Observe $n$ independent draws, and let $Z_i=1$ if draw $i$ came from component $g_1$, and $Z_i=0$ otherwise. The disjoint supports make the component observable from the draw. Estimate $p$ by $\hat p=n^{-1}\sum_{i=1}^n Z_i$, and estimate the density by $\hat f_n=(1-\hat p)g_0+\hat p g_1$.

At $x=2.5$, $g_0(2.5)=0$ and $g_1(2.5)=1$, so $f_p(2.5)=p$ and $\hat f_n(2.5)=\hat p$. Since each $Z_i$ is Bernoulli with success probability $p$, $E[\hat p]=p$ and $\operatorname{Var}(\hat p)=p(1-p)/n$. Therefore the bias is zero and $$ R_n(2.5)=\frac{p(1-p)}{n}. $$

For $p=0.3$ and $n=10$, the variance and risk are $0.3(0.7)/10=0.021$, with squared bias $0$. At $x=0.5$, the estimate is $1-\hat p$ and the target is $1-p$; its variance and risk are again $0.021$. The same sample-size and parameter values need not give the same risk at every location for other component densities, as the next examples illustrate.

Worked Example: Shrinkage Adds Bias at a Fixed Location

Use the same two-component model, with $p=0.2$ and $n=4$. Define a shrunken weight $\tilde p=\tfrac12\hat p+\tfrac14$, and estimate the density by $\tilde f_n=(1-\tilde p)g_0+\tilde p g_1$. Since $0\leq\hat p\leq1$, we have $1/4\leq\tilde p\leq3/4$; thus this is still a density-valued estimator.

At $x=2.5$, the estimated value is $\tilde p$ and the target is $p$. Its expectation is $$ E[\tilde p]=\tfrac12(0.2)+\tfrac14=0.35, $$ so its bias is $0.35-0.2=0.15$, and its squared bias is $0.0225$. Its variance is $$ \operatorname{Var}(\tilde p)=\left(\tfrac12\right)^2\operatorname{Var}(\hat p) =\tfrac14\frac{0.2(0.8)}{4}=0.01. $$ The pointwise risk is consequently $0.0225+0.01=0.0325$. At $x=0.5$, both estimate and target are the complements of these weights, so the bias changes sign but its square, the variance, and the risk remain the same. This calculation shows why variance alone is not a measure of accuracy when an estimator is biased.

Pointwise Risk and Integrated Squared Error

Integrated squared error measures the error of an entire estimated function: $$ \operatorname{ISE}(\hat f_n)=\int(\hat f_n(x)-f(x))^2\,dx. $$ Its expectation is the integrated risk. To relate it to pointwise risk, the integrand must be measurable as a function of the sample and location. The following result uses nonnegativity, so Tonelli's theorem applies even before finiteness is known.

Theorem: Suppose $(\omega,x)\mapsto\hat f_n(\omega,x)$ is jointly measurable, $f$ is measurable, and integration is with respect to Lebesgue measure. Then $$ E[\operatorname{ISE}(\hat f_n)]=\int E[(\hat f_n(x)-f(x))^2]\,dx=\int R_n(x)\,dx, $$ where either side may initially be infinite. If this value is finite, the expected integrated squared error is the integral of pointwise risk.

Proof. The function $q(\omega,x)=(\hat f_n(\omega,x)-f(x))^2$ is jointly measurable and nonnegative. Tonelli's theorem permits exchanging expectation and integration for such a function: $$ E\left[\int q(\omega,x)\,dx\right]=\int E[q(\omega,x)]\,dx. $$ The left side is the expected integrated squared error, and the integrand on the right is $R_n(x)$ by its definition. This proves the identity, including the possibility that both sides are infinite. $\square$

Combining this identity with the pointwise decomposition gives the integrated bias–variance expression whenever the relevant integrals are defined: $$ E[\operatorname{ISE}(\hat f_n)] =\int \operatorname{Var}(\hat f_n(x))\,dx+\int b_n(x)^2\,dx. $$ This is an aggregation over locations; it is not the risk at any one selected $x$. If a density estimate has zero risk at one point but positive risk over a region, that isolated success need not make its integrated risk small.

Worked Example: Zero Risk at a Crossing Point, Positive Integrated Risk

Consider densities on $[0,1]$ given by $g_0(x)=1$ and $g_1(x)=x+\tfrac12$, and zero outside that interval. Both integrate to one: $\int_0^1 1\,dx=1$, and $\int_0^1(x+\tfrac12)\,dx=\tfrac12+\tfrac12=1$. The second density is nonnegative. Let $f_p=(1-p)g_0+pg_1$. As in the first example, suppose component labels are observed, and estimate $p$ by the sample fraction $\hat p$ from $n$ independent labels. The density estimate is $\hat f_n=(1-\hat p)g_0+\hat p g_1$.

At any $x\in[0,1]$, subtracting the target from the estimate gives $$ \hat f_n(x)-f_p(x)=(\hat p-p)(g_1(x)-g_0(x))=(\hat p-p)(x-\tfrac12). $$ The estimate is unbiased for the target at every location, so its pointwise risk is $$ R_n(x)=\frac{p(1-p)}{n}(x-\tfrac12)^2. $$ In particular, $R_n(0.5)=0$, because the two component densities agree there. Yet the integrated risk is $$ \int_0^1R_n(x)\,dx =\frac{p(1-p)}{n}\int_0^1(x-\tfrac12)^2\,dx =\frac{p(1-p)}{12n}, $$ which is positive for $0<p<1$. For $p=0.4$ and $n=10$, this integrated risk is $0.24/120=0.002$, while the pointwise risk at $x=0.5$ is exactly zero. At $x=0.9$, it is $0.024(0.4)^2=0.00384$. The risk profile, not just one value, determines the integrated summary.

Density Versions and Common Pitfalls

A density is defined only up to changes on sets of Lebesgue measure zero: changing its value at a single point leaves the distribution and all integrals unchanged. Consequently, integrated squared error does not change if a density version is modified at one point, but pointwise risk at that point may change. A pointwise statement therefore concerns a specified version of the density and estimator. Smoothness or continuity assumptions can make a particular version natural, but they must be stated rather than silently presumed.

Mistake: “The estimator is unbiased, so its risk is zero.” Unbiasedness sets $b_n(x)=0$, but risk still equals $\operatorname{Var}(\hat f_n(x))$, which can be positive.
Mistake: “Small risk at the location of interest means small integrated error.” Risk can be concentrated elsewhere. Integrated risk aggregates it over the domain.
Mistake: “Integrated risk determines risk at every point.” An integral does not specify the value of its integrand at a particular point; isolated changes can leave the integral unchanged.
Mistake: “The density value at a point is version-free.” It is not. Pointwise comparisons require a specified density version, unlike distributional statements that depend only on integrals.

Pointwise risk is useful when accuracy at a particular threshold, mode, or scientifically chosen value matters. It also clarifies the local tradeoff: a method may reduce variance at $x$ while increasing squared bias there, so the relevant comparison is their sum. Integrated risk remains appropriate when overall function error is the target, but its aggregation can hide both local strengths and local weaknesses.

Check Your Understanding

Use the definitions and results above to answer these questions.

At a fixed $x$, an estimator has bias $0.1$ and variance $0.04$. What is its pointwise risk?
Why does unbiasedness not imply zero pointwise risk?
In the crossing-density example, why is risk zero at $x=0.5$ even though the integrated risk is positive?
Which measurability and sign properties permit the exchange of expectation and integration in the integrated-risk identity?
Why can changing a density at one point affect pointwise risk without changing integrated squared error?

Pointwise Risk and Bias–Variance

What You'll Learn