The Cramér-Rao Bound for Biased Estimators

Why Bias Changes the Cramér–Rao Bound

The usual Cramér–Rao lower bound is often introduced for unbiased estimators. But an estimator can be biased and still have a variance worth assessing. In that case, the relevant quantity is not simply the variance of an unbiased estimator: the bound also depends on how the estimator’s expected value changes with the parameter. This result is aligned with the SOA VEE Mathematical Statistics syllabus.

Suppose $X$ has a probability model indexed by a scalar parameter $\theta$, and let $T=T(X)$ be an estimator with finite variance. Write its expectation as $g(\theta)=E_\theta[T]$. The estimator’s bias is $b(\theta)=g(\theta)-\theta$. Unless $b(\theta)=0$ for every $\theta$, $T$ is biased.

Definition: Under the regularity conditions below, the Cramér–Rao variance bound for an estimator $T$ with differentiable expectation $g(\theta)$ is $$ \operatorname{Var}_\theta(T)\geq \frac{[g'(\theta)]^2}{I(\theta)}. $$ Here $I(\theta)$ is the Fisher information in the observed data. Since $g(\theta)=\theta+b(\theta)$, the same bound can be written as $$ \operatorname{Var}_\theta(T)\geq \frac{[1+b'(\theta)]^2}{I(\theta)}. $$

This is a lower bound on variance, not on mean squared error. The mean squared error is $E_\theta[(T-\theta)^2]=\operatorname{Var}_\theta(T)+b(\theta)^2$. The bias itself is not squared in the variance bound; instead, its derivative affects the bound through $1+b'(\theta)$.

Conditions and a Short Derivation

For a sample with joint probability mass function or density $f_\theta(x)$, define the score by $U_\theta(X)=\partial \log f_\theta(X)/\partial\theta$. The Fisher information is $I(\theta)=E_\theta[U_\theta(X)^2]$. The formula requires a differentiable model and expectation $g(\theta)$, finite variance for $T$, and finite, positive Fisher information. It also requires regularity sufficient to differentiate the expectation under the integral or sum. In particular, the support should not change with $\theta$, and the derivative and expectation must be interchangeable. If boundary terms arise or these conditions fail, this derivation does not establish the bound.

Differentiate the expectation.
Under the stated regularity conditions, $g'(\theta)=\int T(x)\,\partial f_\theta(x)/\partial\theta\,dx$, with a sum replacing the integral for discrete data.

Use the score.
Since $\partial f_\theta(x)/\partial\theta=f_\theta(x)U_\theta(x)$, this gives $g'(\theta)=E_\theta[TU_\theta]$. The same regularity conditions imply $E_\theta[U_\theta]=0$, so $g'(\theta)=\operatorname{Cov}_\theta(T,U_\theta)$.

Apply Cauchy–Schwarz.
$\operatorname{Cov}_\theta(T,U_\theta)^2\leq\operatorname{Var}_\theta(T)\operatorname{Var}_\theta(U_\theta)$, and $\operatorname{Var}_\theta(U_\theta)=I(\theta)$. Therefore $[g'(\theta)]^2\leq\operatorname{Var}_\theta(T)I(\theta)$, which yields the bound.

The derivation explains why an unbiased estimator has the familiar bound $1/I(\theta)$: in that case $g(\theta)=\theta$, so $g'(\theta)=1$. For a biased estimator, replace that derivative of one with the derivative of its actual expectation. If $g'(\theta)=0$ at a particular parameter value, the bound there is zero and provides no positive variance guarantee.

Applying the Bound

For independent observations, Fisher information adds when the observations satisfy the regularity conditions. The examples below state the model and parameters explicitly, calculate the estimator’s expectation and derivative, and compare its variance with the resulting bound. Each also checks whether equality holds.

Worked Example: A Shifted Estimator from a Poisson Sample

Let $X_1,\ldots,X_4$ be independent Poisson random variables with parameter $\lambda>0$, so each has mean and variance $\lambda$. Consider the estimator $T=\bar X+2$. Find the biased-estimator Cramér–Rao bound and compare it with the actual variance at $\lambda=3$.

Method. Compute $g(\lambda)=E_\lambda[T]$, differentiate it, and use the Fisher information for the four observations. The constant shift makes the estimator biased but does not change its variance.

Expectation and bias. Since $E_\lambda[\bar X]=\lambda$, $g(\lambda)=\lambda+2$. Thus $b(\lambda)=g(\lambda)-\lambda=2$, and $g'(\lambda)=1$.

Information and bound. A Poisson observation has Fisher information $1/\lambda$, so independence gives $I_4(\lambda)=4/\lambda$. At $\lambda=3$, $I_4(3)=4/3$. Therefore

\operatorname{Var}_3(T)\geq \frac{[g'(3)]^2}{I_4(3)} =\frac{1}{4/3}=\frac{3}{4}=0.75.

Actual variance and check. The constant 2 has zero variance, and $\operatorname{Var}_3(\bar X)=3/4=0.75$. The actual variance equals the bound. The estimator has bias 2, so its mean squared error is $0.75+2^2=4.75$, not $0.75$. The common mistake is to interpret the variance bound as a bound on mean squared error.

Worked Example: An Affine Estimator of a Normal Mean

Let $X_1,\ldots,X_4$ be independent with $X_i\sim N(\mu,9)$; the variance 9 is known, and $\mu$ is unknown. Consider $T=0.8\bar X+2$. At $\mu=10$, calculate its expectation, bias, Cramér–Rao variance bound, and actual variance.

Method. Use the expectation of the affine estimator to find $g'(\mu)$, then use the information for independent normal observations with known variance.

Expectation and bias. Since $E_\mu[\bar X]=\mu$, $g(\mu)=0.8\mu+2$. At $\mu=10$, $E[T]=0.8(10)+2=10$, so the bias at this particular value is zero. In general, however, $b(\mu)=2-0.2\mu$, so $T$ is not an unbiased estimator for all $\mu$. The derivative is $g'(\mu)=0.8$.

Information and bound. For four independent $N(\mu,9)$ observations, $I_4(\mu)=4/9$. Hence, at $\mu=10$,

\operatorname{Var}_{10}(T)\geq \frac{(0.8)^2}{4/9} =\frac{0.64(9)}{4}=1.44.

Actual variance and check. The sample mean has variance $9/4=2.25$. Multiplying it by 0.8 gives $\operatorname{Var}_{10}(T)=(0.8)^2(2.25)=0.64(2.25)=1.44$. The actual variance equals the bound. Although the estimator happens to have zero bias at $\mu=10$, its expectation is not $\mu$ for all parameter values. The bound uses $g'(\mu)$, not a conclusion drawn from the bias at just one parameter value.

Worked Example: A Strict Bound for a Squared Binomial Proportion

Let $X\sim\operatorname{Binomial}(2,p)$, with $0<p<1$, and estimate $p$ using $T=(X/2)^2$. Calculate the bound and actual variance at $p=1/4$, and determine whether equality holds.

Method. Enumerate the three possible values of $X$ to calculate $E_p[T]$ and $E_p[T^2]$; then use the binomial Fisher information.

Expectation and derivative. The probabilities for $X=0,1,2$ are $(1-p)^2$, $2p(1-p)$, and $p^2$. The corresponding values of $T$ are $0,1/4,1$, so

g(p)=E_p[T]=0(1-p)^2+\frac14\,2p(1-p)+p^2 =\frac{p}{2}+\frac{p^2}{2}, \qquad g'(p)=\frac12+p.

At $p=1/4$, $g(1/4)=5/32$ and $g'(1/4)=3/4$. The estimator’s bias there is $5/32-1/4=-3/32$.

Information and bound. For two independent Bernoulli trials, $I_2(p)=2/[p(1-p)]$. Thus $I_2(1/4)=2/(3/16)=32/3$, giving

\operatorname{Var}_{1/4}(T)\geq \frac{(3/4)^2}{32/3} =\frac{27}{512}.

Actual variance and check. First, $E_{1/4}[T^2]=\frac{1}{16}(2p(1-p))+p^2 =\frac{3}{128}+\frac{1}{16}=\frac{11}{128}$. Also $g(1/4)^2=(5/32)^2=25/1024$. Therefore $\operatorname{Var}_{1/4}(T)=11/128-25/1024=88/1024-25/1024=63/1024$. The bound is $27/512=54/1024$, so the actual variance exceeds it by $9/1024$. Equality does not hold. The common mistake is to use $1/I_2(p)$, the unbiased-estimator bound, instead of $[g'(p)]^2/I_2(p)$.

Equality, Interpretation, and Exam Traps

The Cauchy–Schwarz inequality in the derivation is an equality precisely when $T-E_\theta[T]$ is proportional to the score $U_\theta$, with probability one, at the parameter value considered. This condition is sufficient for equality in the derived bound, but equality at a single parameter value does not establish that an estimator is unbiased or efficient throughout a parameter space.

The derivative $g'(\theta)$ describes how the estimator’s expectation responds to a change in the true parameter. A constant bias has derivative zero, so it does not alter $g'(\theta)$; a bias that changes with $\theta$ can increase, reduce, or even make the bound zero at a particular value. This is why knowing only the bias at one point is not enough to calculate the bound.

Wrong: use $1/I(\theta)$ for every estimator. Correction: use $[g'(\theta)]^2/I(\theta)$; $1/I(\theta)$ is the special case $g'(\theta)=1$.
Wrong: treat the variance bound as a lower bound on mean squared error. Correction: add the squared bias to the actual variance to obtain mean squared error.
Wrong: differentiate the bias and stop at $b'(\theta)$. Correction: $g'(\theta)=1+b'(\theta)$, so the bound uses $[1+b'(\theta)]^2$.
Wrong: apply the result when the regularity conditions fail. Correction: verify the model’s support and differentiation conditions before using the score identity.

Key takeaway: For a biased estimator $T$, first find $g(\theta)=E_\theta[T]$, then differentiate it. Under the regularity conditions, the variance is at least $[g'(\theta)]^2/I(\theta)$, or equivalently $[1+b'(\theta)]^2/I(\theta)$.

Check Your Understanding

For each problem, calculate the biased-estimator Cramér–Rao variance bound at the stated parameter value. Compare with the actual variance when requested.

Let $X_1,\ldots,X_8$ be independent Bernoulli observations with success probability $p$, and let $T=0.75\bar X+0.2$. At $p=0.4$, find $g'(p)$, $I_8(p)$, the bound, and the actual variance. (Answer: $g'=0.75$, $I_8(0.4)=100/3$, bound $=0.016875$, and actual variance $=0.016875$.)
Let $X_1,\ldots,X_5$ be independent Poisson observations with parameter $\lambda$, and let $T=2\bar X-1$. At $\lambda=2$, calculate the bound and compare it with the actual variance. (Answer: $g'=2$, $I_5(2)=5/2$, bound $=1.6$, and actual variance $=1.6$.)
Let $X_1,\ldots,X_6$ be independent $N(\mu,4)$ observations, with known variance 4, and let $T=2\bar X-3$. At $\mu=5$, find the bound and the actual variance. (Answer: $g'=2$, $I_6(\mu)=3/2$, bound $=8/3$, and actual variance $=8/3$.)
Let $X\sim\operatorname{Binomial}(2,p)$ and $T=(X/2)^2$. At $p=1/4$, calculate the bound and actual variance, and state whether equality holds. (Answer: bound $=27/512$, actual variance $=63/1024$, so the bound is strict.)

What You'll Learn