The Bias-Variance Identity For Scalar Estimators

Why Separate Error Into Two Parts?

An estimator can miss a parameter in two different ways. Its values may vary substantially across repeated samples, or its average value may be systematically displaced from the parameter. Mean squared error combines these effects into one measure, but separating them makes the source of error visible. Under a finite-moment condition, the separation is exact: mean squared error equals variance plus squared bias.

This is an algebraic identity for a scalar estimator under a specified sampling distribution. It does not require normality, unbiasedness, or independence among observations. The key requirement is that the estimator has a finite second moment, so its mean, variance, and mean squared error are well-defined and finite.

Definitions and Conditions

Let $\theta$ be a fixed scalar parameter, let $X$ denote the data, and let $T=T(X)$ be a measurable scalar estimator. For a given value of $\theta$, expectations and probabilities are taken under the sampling distribution $P_\theta$. The estimator's sampling variation is described by its distribution across repeated samples from that model.

Definition: If $E_\theta[T^2]<\infty$, the bias of $T$ at $\theta$ is $b_\theta(T)=E_\theta[T]-\theta$, its variance is $\operatorname{Var}_\theta(T)=E_\theta[(T-E_\theta[T])^2]$, and its mean squared error is $\operatorname{MSE}_\theta(T)=E_\theta[(T-\theta)^2]$. Under scalar quadratic loss $L(\theta,T)=(T-\theta)^2$, this mean squared error is also the risk of the estimator at $\theta$.

The assumption $E_\theta[T^2]<\infty$ guarantees that all three quantities in the definition are finite. In particular, $E_\theta[|T|]<\infty$ follows from the Cauchy–Schwarz inequality, since $E_\theta[|T|]\leq \sqrt{E_\theta[T^2]}$. The parameter $\theta$ is a fixed finite number, not a random variable in these definitions.

Theorem: For any fixed $\theta$ and measurable scalar estimator $T$ satisfying $E_\theta[T^2]<\infty$, $$ \operatorname{MSE}_\theta(T) = \operatorname{Var}_\theta(T) + \big(E_\theta[T]-\theta\big)^2. $$

Deriving the Identity

The proof adds and subtracts the estimator's expectation inside its error. The centered part, $T-E_\theta[T]$, has expectation zero. That is why the cross-product vanishes when the square is expanded.

Center the estimator.
Write $T-\theta=(T-E_\theta[T])+(E_\theta[T]-\theta)$. The second term is a fixed number under $P_\theta$.

Expand the square.
Square the two-term expression. The result contains a centered square, a constant square, and a cross-product.

Take expectations.
The cross-product has expectation zero because $E_\theta[T-E_\theta[T]]=0$. The other two terms are variance and squared bias.

Here is the complete calculation. Finite second moment ensures the terms are integrable, so the expectations in the expansion exist.

\begin{aligned} \operatorname{MSE}_\theta(T) &=E_\theta[(T-\theta)^2]\\ &=E_\theta\left[\left\{T-E_\theta[T]+E_\theta[T]-\theta\right\}^2\right]\\ &=E_\theta[(T-E_\theta[T])^2] +2(E_\theta[T]-\theta)E_\theta[T-E_\theta[T]] +(E_\theta[T]-\theta)^2\\ &=\operatorname{Var}_\theta(T)+(E_\theta[T]-\theta)^2. \end{aligned}

The middle term is zero because $E_\theta[T-E_\theta[T]]=E_\theta[T]-E_\theta[T]=0$. This proves the identity. $\square$

The theorem holds separately at each parameter value for which the moment condition holds. It does not say that variance or bias must be constant in $\theta$; either may change across the parameter space. It says that at a particular $\theta$, the total squared-error risk is the sum of these two nonnegative contributions.

Worked Examples

Worked Example: An Unbiased Bernoulli Estimator

Suppose $X\sim\operatorname{Bernoulli}(p)$, where $0\leq p\leq 1$, and use $T=X$ to estimate $p$. Calculate the bias, variance, and mean squared error, then check the identity directly.

Since $X$ is Bernoulli, $E_p[X]=p$, so the bias is $E_p[X]-p=0$. Also $X^2=X$, because $X$ is either zero or one. Therefore

\operatorname{Var}_p(X)=E_p[X^2]-(E_p[X])^2=p-p^2=p(1-p).

The identity gives $\operatorname{MSE}_p(X)=p(1-p)+0^2=p(1-p)$. Direct calculation confirms it: when $X=1$, the squared error is $(1-p)^2$, and when $X=0$, it is $p^2$. Thus

\begin{aligned} E_p[(X-p)^2] &=p(1-p)^2+(1-p)p^2\\ &=p(1-p)\big((1-p)+p\big)\\ &=p(1-p). \end{aligned}

This estimator is unbiased, so its mean squared error is exactly its variance. At the endpoints $p=0$ and $p=1$, both are zero, as the observation then equals the target with probability one.

Worked Example: A Biased Estimator in a Binomial Model

Let $X\sim\operatorname{Binomial}(2,p)$, with $0\leq p\leq1$, and estimate $p$ using $T=X/3$. This estimator is biased. Find its bias, variance, and mean squared error, and verify the decomposition by summing over the possible observations.

The binomial mean and variance are $E_p[X]=2p$ and $\operatorname{Var}_p(X)=2p(1-p)$. Hence

E_p[T]=\frac{2p}{3},\qquad b_p(T)=\frac{2p}{3}-p=-\frac{p}{3}, \qquad \operatorname{Var}_p(T)=\frac{2p(1-p)}{9}.

The identity therefore gives

\operatorname{MSE}_p(T) =\frac{2p(1-p)}{9}+\left(-\frac p3\right)^2 =\frac{2p-p^2}{9}.

For a direct check, the probabilities of $X=0,1,2$ are $(1-p)^2$, $2p(1-p)$, and $p^2$, respectively. The corresponding estimates are $0,1/3,2/3$. Thus

\begin{aligned} E_p[(T-p)^2] &=(1-p)^2p^2+2p(1-p)\left(\frac13-p\right)^2 +p^2\left(\frac23-p\right)^2\\ &=p^2-2p^3+p^4 +2p(1-p)\left(\frac19-\frac{2p}{3}+p^2\right) +p^2\left(\frac49-\frac{4p}{3}+p^2\right)\\ &=\frac{2p-p^2}{9}. \end{aligned}

The result agrees with the decomposition. For $0<p\leq1$, the squared bias is positive, so the mean squared error is larger than the variance alone. At $p=0$, both are zero.

Worked Example: A Biased Affine Estimator Under a Normal Model

Suppose $X\sim N(\theta,\sigma^2)$, where $\sigma^2$ is finite and known, and estimate $\theta$ with $T=(X+c)/2$, where $c$ is a fixed constant. Find the three components of the identity. Then evaluate them when $\theta=2$, $c=0$, and $\sigma^2=4$.

Linearity of expectation gives $E_\theta[T]=(\theta+c)/2$. Therefore $b_\theta(T)=(c-\theta)/2$. Multiplying a random variable by $1/2$ multiplies its variance by $1/4$, while adding a constant does not change variance. Thus

\operatorname{Var}_\theta(T)=\frac{\sigma^2}{4}, \qquad \operatorname{MSE}_\theta(T)=\frac{\sigma^2}{4}+\frac{(c-\theta)^2}{4}.

For the specified values, the bias is $-1$, the variance is $1$, and the mean squared error is $1+(-1)^2=2$. This also follows directly: $T=X/2$ has mean $1$ and variance $1$, so $E_\theta[(T-2)^2]=E_\theta[((T-1)-1)^2]=1+1=2$; the cross-product has expectation zero because $E_\theta[T-1]=0$.

What the Identity Does—and Does Not—Say

A direct consequence is that mean squared error is at least as large as either component:

\operatorname{MSE}_\theta(T)\geq \operatorname{Var}_\theta(T), \qquad \operatorname{MSE}_\theta(T)\geq b_\theta(T)^2.

Both inequalities follow because the other term in the decomposition is nonnegative. In particular, an unbiased estimator has $b_\theta(T)=0$, so its mean squared error equals its variance. Conversely, a small variance by itself does not guarantee small mean squared error: a large bias can make the total error large. Nor does the identity assert that a biased estimator is necessarily better or worse than an unbiased one; that comparison requires evaluating the relevant quantities under the model and parameter values of interest.

The condition on moments matters. If $E_\theta[T^2]=\infty$, the standard finite variance and mean squared error in the theorem need not exist. It is not valid to manipulate undefined expressions such as “infinity minus infinity” and claim the same finite decomposition. Before applying the identity, check that the estimator has a finite second moment at the parameter value being considered.

Common Mistake: Treating Bias as Random Error

A frequent error is to write mean squared error as variance plus bias, omitting the square. That mixes quantities with different units: variance and mean squared error have squared units, while bias has the same units as the estimator. The correct term is squared bias, $b_\theta(T)^2$, which has squared units and is always nonnegative.

A related mistake is to use $E_\theta[T^2]$ as the variance. The variance is $E_\theta[(T-E_\theta[T])^2]$, or equivalently $E_\theta[T^2]-(E_\theta[T])^2$ when the second moment is finite. For an unbiased estimator of $\theta$, $E_\theta[T]=\theta$, so $E_\theta[T^2]$ still includes the contribution $\theta^2$; it is not generally the variance or the mean squared error.

Key takeaway: For a scalar estimator with finite second moment, $\operatorname{MSE}_\theta(T)=\operatorname{Var}_\theta(T)+b_\theta(T)^2$. Unbiasedness removes the squared-bias term; it does not remove sampling variability.

Check Your Understanding

For each question, keep the parameter value fixed and distinguish the estimator's expectation from its realized value.

State the moment condition needed for the finite bias-variance identity, and explain why it ensures the quantities in the identity exist.
An estimator has variance $5$ and bias $-2$ at a particular parameter value. What is its mean squared error there?
Why does an unbiased estimator's mean squared error equal its variance?
Suppose an estimator has zero variance at a parameter value. What does the identity say about its mean squared error at that value?
Explain why the formula cannot automatically be applied using ordinary finite variance when the estimator has an infinite second moment.