Risk Functions Of Biased And Unbiased Estimators

Why Compare Risk Functions?

An estimator is not judged by its bias alone. Under squared-error loss, its risk at a parameter value combines variability across possible data sets with any systematic displacement from the target. A biased estimator can therefore have lower risk than an unbiased estimator at some parameter values: a reduction in variance may more than compensate for the squared bias.

The comparison is pointwise: it is made separately at each value of the unknown parameter. Two estimators can trade places as the parameter changes, and an estimator that has smaller risk in one region need not have smaller risk everywhere. The examples below make that distinction concrete.

Risk, Bias, and the Comparison Criterion

Definition: Let $X$ have distribution $P_\theta$, where $\theta\in\Theta$, and let $\delta(X)$ be an estimator of the real-valued parameter $\theta$. Under squared-error loss $L(\theta,\delta)=(\delta-\theta)^2$, the risk function is $$ R(\theta,\delta)=E_\theta[(\delta(X)-\theta)^2]. $$ When the relevant expectations are finite, the bias is $b_\delta(\theta)=E_\theta[\delta(X)]-\theta$, and the variance is $\operatorname{Var}_\theta(\delta(X))$. The estimator is unbiased on $\Theta$ if $b_\delta(\theta)=0$ for every $\theta\in\Theta$.

The risk is an expectation over the data distribution at a fixed parameter value; it is not an average over parameter values. For any estimator with finite second moment, write its error as the centered random fluctuation plus its bias:

\delta(X)-\theta = \bigl(\delta(X)-E_\theta[\delta(X)]\bigr) + b_\delta(\theta).

Squaring and taking expectation makes the cross term vanish, because the centered fluctuation has expectation zero. Thus

R(\theta,\delta) = \operatorname{Var}_\theta(\delta(X)) + b_\delta(\theta)^2.

Theorem: Suppose $U$ and $V$ are two estimators with finite second moments under $P_\theta$, and $U$ is unbiased for every $\theta\in\Theta$. Then $$ R(\theta,V)<R(\theta,U) \quad\Longleftrightarrow\quad \operatorname{Var}_\theta(U)-\operatorname{Var}_\theta(V)>b_V(\theta)^2. $$ In particular, a biased estimator has smaller risk at $\theta$ precisely when its variance reduction relative to $U$ is greater than its squared bias.

Proof. Unbiasedness gives $R(\theta,U)=\operatorname{Var}_\theta(U)$. The risk decomposition gives $R(\theta,V)=\operatorname{Var}_\theta(V)+b_V(\theta)^2$. Therefore

R(\theta,V)<R(\theta,U) \quad\Longleftrightarrow\quad \operatorname{Var}_\theta(V)+b_V(\theta)^2 < \operatorname{Var}_\theta(U),

which is equivalent, after subtracting $\operatorname{Var}_\theta(V)$, to the stated condition. Equality of risks occurs when the variance reduction equals the squared bias; reversing the inequality gives larger risk for $V$. $\square$

This criterion is local in $\theta$. To claim that $V$ has lower risk throughout a parameter space, the inequality must hold at every $\theta$ in that space. It is not enough to check it at one convenient value.

Worked Comparisons

Worked Example: A Bernoulli Observation and a Constant Estimate

Suppose $X\sim\operatorname{Bernoulli}(p)$, with $p\in[0,1]$. Compare the unbiased estimator $U=X$ with the constant estimator $V=1/2$.

Since $E_p[X]=p$, $U$ is unbiased. Its variance, and hence its risk, is

R(p,U)=\operatorname{Var}_p(X)=p(1-p).

The constant estimator has bias $1/2-p$ and variance zero, so its risk is

R(p,V)=\left(\frac12-p\right)^2.

The constant estimator has lower risk when

\left(p-\frac12\right)^2<p(1-p) \quad\Longleftrightarrow\quad 2p^2-2p+\frac14<0.

The roots of the quadratic are $p=1/2\pm1/(2\sqrt{2})$, so $V$ has lower risk for $1/2-1/(2\sqrt{2})<p<1/2+1/(2\sqrt{2})$. At $p=1/2$, its risk is zero while $U$'s risk is $1/4$. At either endpoint $p=0$ or $p=1$, $U$'s risk is zero while $V$'s is $1/4$. The risk curves therefore cross: neither estimator has lower risk everywhere.

Worked Example: A Uniform Location Model

Let $X\sim\operatorname{Uniform}(\theta-1,\theta+1)$, for $\theta\in\mathbb{R}$. Compare $U=X$ with the constant estimator $V=0$. The distribution is symmetric about $\theta$, so $E_\theta[X]=\theta$ and $U$ is unbiased. A uniform random variable on an interval of length 2 has variance $2^2/12=1/3$. Therefore

R(\theta,U)=\frac13.

For $V=0$, the error is always $-\theta$, so its bias is $-\theta$, its variance is zero, and

R(\theta,V)=\theta^2.

It follows directly that $V$ has lower risk when $\theta^2<1/3$, or $|\theta|<1/\sqrt{3}$. The risks are equal at $\theta=\pm1/\sqrt{3}$. Outside that interval, $U$ has lower risk. This example shows why the parameter value matters even when one estimator is unbiased at every parameter value.

Worked Example: A Biased Estimator with Lower Risk Throughout a Finite Model

Let the parameter space be $\Theta=\{0,1\}$. Observe $Y\in\{0,1\}$, where $Y=\theta$ with probability $0.8$ and $Y\ne\theta$ with probability $0.2$. Compare $U=(Y-0.2)/0.6=(5Y-1)/3$ with $V=Y$. Both estimate the numerical value of $\theta$, and actions are allowed to be any real number.

First, $E_\theta[Y]=0.2+0.6\theta$. Hence

E_\theta[U] = \frac{E_\theta[Y]-0.2}{0.6} = \theta,

so $U$ is unbiased. The variance of $Y$ is $0.2(0.8)=0.16$ at either parameter value. Since $U=(Y-0.2)/0.6$, scaling gives

\operatorname{Var}_\theta(U) = \frac{0.16}{0.6^2} = \frac{0.16}{0.36} = \frac49.

Because $U$ is unbiased, $R(\theta,U)=4/9$ for both $\theta=0$ and $\theta=1$. In contrast, $V=Y$ is wrong by exactly 1 when the observation is flipped and is correct otherwise. Its squared-error risk is therefore

R(\theta,V) = 0.2(1)^2+0.8(0)^2 = 0.2 = \frac15 < \frac49.

For completeness, $E_\theta[V]-\theta=0.2$ when $\theta=0$ and $-0.2$ when $\theta=1$, so $V$ is biased at both parameter values. Its variance is $0.16$, and its risk decomposition checks the direct calculation: $0.16+0.2^2=0.16+0.04=0.20$. Thus this biased estimator has strictly lower risk at every point of this finite parameter space.

How to Read a Risk Comparison

A risk function records performance separately for each possible parameter value. The Bernoulli and uniform examples have crossing risk functions: each estimator is preferable in some region, and neither wins everywhere. The finite-model example has a different outcome: the biased estimator has lower risk at both parameter values. These are distinct conclusions, and a calculation at one value cannot establish the other.

For two estimators $\delta_1$ and $\delta_2$, a direct comparison is often clearest:

R(\theta,\delta_1)-R(\theta,\delta_2).

If this difference is positive at a particular $\theta$, then $\delta_2$ has lower risk there. To make a claim across all of $\Theta$, determine the sign over the entire parameter space, including its boundaries when they belong to the space. Tables or plots can help visualize the comparison, but they do not replace checking the sign of the risk difference.

Common Mistake

Wrong reasoning: “$U$ is unbiased and $V$ is biased, so $U$ must have smaller squared-error risk.” Unbiasedness sets the bias term to zero; it says nothing by itself about the size of the variance.
Correction: Compare $\operatorname{Var}_\theta(U)$ with $\operatorname{Var}_\theta(V)+b_V(\theta)^2$ at the parameter value of interest. A biased estimator wins there if its variance reduction exceeds its squared bias. Then check separately whether the comparison holds elsewhere in the parameter space.

The practical importance is that a preference for unbiasedness is not the same as a proof of lower risk. Under squared-error loss, the relevant quantity is the full risk function. Reporting the parameter region of improvement—and any region where the ranking reverses—makes the decision criterion explicit.

Check Your Understanding

Use the definitions and comparisons above to answer the following.

If an estimator has bias $0.3$ and variance $0.1$ at a parameter value, what is its squared-error risk there?
In the Bernoulli example, why does the constant estimator have lower risk near $p=1/2$, but not at $p=0$?
For two estimators whose risks cross, can either one be said to have lower risk everywhere? Explain.
In the uniform location example, find the parameter values at which the two risks are equal.
In the finite model, verify the risk of $V=Y$ from its bias and variance at $\theta=1$.