Integrated Squared Error Decomposition

From Squared Error to an Integrated Risk

A density estimator is random because it is built from random data. Its error therefore varies both across the domain where the density is defined and across possible data sets. Integrated squared error summarizes the error over the domain for one realized estimator; integrated mean squared error averages that quantity over the estimator’s randomness. The distinction matters: the first is random, while the second is a number (or, without suitable integrability, possibly infinity).

The exact decomposition in this tutorial separates integrated mean squared error into two contributions: integrated squared bias and integrated variance. It is an identity, not an approximation. To make it valid, we must ensure the random estimator is measurable and that exchanging expectation and integration is justified.

Definition: Let $D\subseteq\mathbb{R}^d$ be a measurable domain, let $dx$ denote Lebesgue measure, and let $f:D\to[0,\infty)$ be a target density. A random density estimator $\hat f(x)$ is a jointly measurable function of $x\in D$ and the underlying outcome $\omega$. Its integrated squared error is $$ \operatorname{ISE}(\hat f,f)=\int_D \bigl(\hat f(x)-f(x)\bigr)^2\,dx. $$ Its integrated mean squared error is $$ \operatorname{MISE}(\hat f,f)=E\!\left[\operatorname{ISE}(\hat f,f)\right]. $$ Where the relevant moments are finite, the pointwise bias is $E[\hat f(x)]-f(x)$, and the pointwise variance is $\operatorname{Var}(\hat f(x))$.

The definition does not require the estimator to be a valid density for every outcome, although many density estimators are. For the decomposition, the central object is its squared error as a random function of $x$. In the examples below, each realized estimator is nonnegative and integrates to one, so it is a density in every realization.

The Pointwise Identity

At a fixed location $x$, the target $f(x)$ is a nonrandom number and $\hat f(x)$ is a random variable. Write $m(x)=E[\hat f(x)]$, when this expectation exists. Subtracting and adding $m(x)$ separates the error into a centered random fluctuation and a nonrandom bias:

\hat f(x)-f(x) = \bigl(\hat f(x)-m(x)\bigr)+\bigl(m(x)-f(x)\bigr).

Upon squaring, the cross term has expectation zero because $E[\hat f(x)-m(x)]=0$. This gives the usual bias–variance identity at each $x$.

Result: If $E[\hat f(x)^2]<\infty$, then $$ E\!\left[\bigl(\hat f(x)-f(x)\bigr)^2\right] = \bigl(E[\hat f(x)]-f(x)\bigr)^2+\operatorname{Var}(\hat f(x)). $$

For completeness, this identity follows by expanding the square around the mean. The finiteness of the second moment ensures that the expectations in the calculation are defined.

\begin{aligned} E\!\left[\bigl(\hat f(x)-f(x)\bigr)^2\right] &=E\!\left[\bigl(\hat f(x)-m(x)+m(x)-f(x)\bigr)^2\right]\\ &=E\!\left[(\hat f(x)-m(x))^2\right] +2(m(x)-f(x))E[\hat f(x)-m(x)] +(m(x)-f(x))^2\\ &=\operatorname{Var}(\hat f(x))+(m(x)-f(x))^2. \end{aligned}

This calculation is pointwise. Integrating it over $D$ requires an additional justification: it is not enough merely to write an expectation inside an integral and assume the order can be exchanged.

Exact Integrated Decomposition

A convenient sufficient set of conditions is that $f$ is measurable and square-integrable on $D$, that $\hat f$ is jointly measurable in $x$ and the data outcome, and that the expected integrated squared error is finite. The last condition can be written directly as an integral of a nonnegative function. Tonelli’s theorem allows the expectation and integral to be exchanged for that function, even before finiteness is known; the assumed finite value then ensures the quantities in the decomposition are finite.

Theorem: Suppose $f\in L^2(D)$, $\hat f(x)$ is jointly measurable, and $$ \int_D E\!\left[\bigl(\hat f(x)-f(x)\bigr)^2\right]\,dx<\infty. $$ Then $E[\hat f(x)]$ and $\operatorname{Var}(\hat f(x))$ are finite for almost every $x$, and $$ \operatorname{MISE}(\hat f,f) = \int_D \bigl(E[\hat f(x)]-f(x)\bigr)^2\,dx + \int_D \operatorname{Var}(\hat f(x))\,dx. $$

Proof. Joint measurability implies that $(x,\omega)\mapsto(\hat f(x,\omega)-f(x))^2$ is measurable and nonnegative. Tonelli’s theorem therefore gives

E\!\left[\int_D(\hat f(x)-f(x))^2\,dx\right] = \int_D E\!\left[(\hat f(x)-f(x))^2\right]\,dx.

The right-hand side is finite by assumption. Consequently, $E[(\hat f(x)-f(x))^2]$ is finite for almost every $x$. Since $f\in L^2(D)$, $f(x)$ is finite almost everywhere. For almost every such $x$, the inequality $\hat f(x)^2\leq 2(\hat f(x)-f(x))^2+2f(x)^2$ shows $E[\hat f(x)^2]<\infty$. The pointwise identity thus applies for almost every $x$.

Both terms in that identity are nonnegative and measurable as functions of $x$. Integrating the identity and using the preceding Tonelli equality gives

\begin{aligned} \operatorname{MISE}(\hat f,f) &=\int_D E\!\left[(\hat f(x)-f(x))^2\right]\,dx\\ &=\int_D \bigl(E[\hat f(x)]-f(x)\bigr)^2\,dx +\int_D\operatorname{Var}(\hat f(x))\,dx. \end{aligned}

The left-hand side is finite, so the two nonnegative integrals on the last line are finite as well. This proves the claimed equality. $\square$

The same equality can hold with infinite values when appropriate pointwise moments exist and the nonnegative terms are interpreted as extended integrals. The finite-risk condition in the theorem is useful because it avoids ambiguity about undefined differences of infinities and guarantees a finite decomposition.

Worked Examples

Worked Example: An Unbiased Random Density

Let $D=[0,1]$, let the target density be $f(x)=1$, and let $S$ take the values $-1$ and $1$, each with probability $1/2$. Define $$ \hat f(x)=1+\frac12 S(2x-1),\qquad 0\leq x\leq1. $$ Find the integrated squared bias, integrated variance, and MISE.

Check that each realization is a density. Since $-1\leq 2x-1\leq1$, $\hat f(x)\geq1-\frac12=\frac12$. Also, $$ \int_0^1\hat f(x)\,dx =1+\frac12S\int_0^1(2x-1)\,dx =1, $$ because $\int_0^1(2x-1)\,dx=0$.

The mean is $E[\hat f(x)]=1$, so the bias is zero at every $x$. The variance is $$ \operatorname{Var}(\hat f(x)) =E\!\left[\left(\frac12S(2x-1)\right)^2\right] =\frac14(2x-1)^2, $$ because $S^2=1$. Therefore the integrated squared bias is $0$, while $$ \int_0^1\operatorname{Var}(\hat f(x))\,dx =\frac14\int_0^1(2x-1)^2\,dx =\frac14\left(\frac13\right) =\frac1{12}. $$ The decomposition gives $\operatorname{MISE}=0+\frac1{12}=\frac1{12}$. Directly, for either value of $S$, the squared error integrates to $\frac14\int_0^1(2x-1)^2\,dx=\frac1{12}$, confirming the result.

Worked Example: A Deterministic Biased Density

Again let $D=[0,1]$ and $f(x)=1$, but use the fixed estimator $\hat f(x)=2$ for $0\leq x\leq\frac12$ and $\hat f(x)=0$ for $\frac12<x\leq1$. Calculate its MISE and the two terms in the decomposition.

This estimator is nonnegative and integrates to $2(\frac12)+0(\frac12)=1$. It has no randomness, so $\operatorname{Var}(\hat f(x))=0$ everywhere. Its bias is $2-1=1$ on the first half and $0-1=-1$ on the second half. Thus, $$ \int_0^1\bigl(\hat f(x)-f(x)\bigr)^2\,dx =\int_0^{1/2}1^2\,dx+\int_{1/2}^1(-1)^2\,dx =\frac12+\frac12=1. $$ Since the estimator is deterministic, this is also its MISE. The integrated squared bias is $1$, the integrated variance is $0$, and their sum is $1$.

This example shows that having a density as the estimate does not imply a small integrated error. Normalization constrains the total area, but the estimator can still place that area incorrectly across the domain.

Worked Example: Unequal Randomization and Both Error Components

Let $f(x)=1$ on $[0,1]$. With probability $1/4$, let $\hat f$ equal $2$ on the left half and $0$ on the right half. With probability $3/4$, reverse those values. Find both integrated terms and verify the MISE directly.

Each realization is a density, since it is nonnegative and has area $2(1/2)=1$. On the left half, $\hat f(x)$ equals $2$ with probability $1/4$ and $0$ with probability $3/4$. Its mean is $(1/4)2+(3/4)0=1/2$, so its bias relative to $f(x)=1$ is $-1/2$. Its second moment is $(1/4)4=1$, and its variance is $1-(1/2)^2=3/4$.

On the right half the probabilities are reversed. The mean is $(3/4)2+(1/4)0=3/2$, the bias is $1/2$, and the second moment is $(3/4)4=3$. Its variance is $3-(3/2)^2=3/4$. Hence the integrated squared bias and integrated variance are $$ \int_0^1(E[\hat f(x)]-1)^2\,dx =\frac12\left(\frac14\right)+\frac12\left(\frac14\right) =\frac14, $$ $$ \int_0^1\operatorname{Var}(\hat f(x))\,dx =\frac12\left(\frac34\right)+\frac12\left(\frac34\right) =\frac34. $$ Their sum is $1$. Directly, either realized estimator differs from $1$ by magnitude $1$ throughout $[0,1]$, so its integrated squared error is $1$; its expectation is also $1$.

Why the Conditions and Decomposition Matter

The decomposition identifies two distinct sources of expected integrated error. Integrated squared bias measures the systematic discrepancy between the average estimator and the target. Integrated variance measures the average squared fluctuation of the estimator around its mean. These are integrated over the domain, so locations with different error levels all contribute according to their Lebesgue measure. The identity itself does not say which term is larger or how either term changes with a tuning choice; it provides the exact accounting that makes those questions precise.

A common mistake is to write $$ E\!\left[\int_D(\hat f-f)^2\,dx\right] =\int_D E[(\hat f-f)^2]\,dx $$ without checking measurability or integrability, and then to treat the result as automatically finite. Tonelli’s theorem does justify exchanging the order for a nonnegative measurable squared error, but it permits the common value to be infinite. Finiteness requires an additional condition, such as the theorem’s finite expected integrated squared error assumption. For signed integrands, an exchange generally requires an integrability condition for Fubini’s theorem; the proof here avoids that issue by applying Tonelli to nonnegative quantities.

Another mistake is to confuse the two error measures. For a realized estimator, $\operatorname{ISE}(\hat f,f)$ is random. The decomposition is for its expectation, $\operatorname{MISE}(\hat f,f)$, and its terms are integrals of expected pointwise quantities. A particular data set can yield an ISE above or below the MISE without contradicting the identity.

Key takeaway: Under joint measurability and finite expected integrated squared error, the MISE is exactly the integrated squared bias plus the integrated variance. Tonelli’s theorem justifies the integration step for squared error; the finiteness condition ensures the decomposition consists of finite quantities.

Check Your Understanding

Use the definitions and conditions above to answer these questions.

What is the difference between $\operatorname{ISE}(\hat f,f)$ and $\operatorname{MISE}(\hat f,f)$?
In the pointwise identity, why does the cross term have expectation zero?
Which theorem justifies exchanging the expectation and integral of the squared error, and why is nonnegativity relevant?
Can the decomposition’s terms sum to infinity if the pointwise moments exist? What condition in the stated theorem rules this out?
For a deterministic estimator, which component of the decomposition is zero, and why?