Why a Variance Lower Bound Is Useful
An estimator can be unbiased and still vary substantially from sample to sample. The Cramér-Rao lower bound gives a benchmark: under suitable regularity conditions, it places a lower limit on the variance any unbiased estimator of a model parameter can achieve. The key idea is to compare the estimator with the score, which records how sensitive the likelihood is to a change in the parameter.
The bound is not a recipe for finding an estimator, and it does not say that every unbiased estimator has variance equal to the bound. It is a limit that can be used to assess how much sampling variation is unavoidable in a specified model. The derivation below is aligned with the SOA VEE Mathematical Statistics syllabus.
Conditions Behind the Derivation
The result requires more than an estimator being unbiased. We use a scalar parameter, a differentiable likelihood, finite Fisher information, and an estimator with finite variance. We also require regularity conditions that permit differentiating the expectation by differentiating the density or mass function inside the integral or sum. A common sufficient setting has support that does not depend on \(\theta\), with the relevant derivatives and integrals well behaved. If support changes with the parameter or differentiation cannot be interchanged with integration, the argument below may fail; the bound should not be applied automatically.
Under these conditions, the score has mean zero. Indeed, since the density or mass function integrates or sums to one, \(\int f_\theta(x)\,dx=1\), and differentiating gives \(\int \frac{\partial}{\partial\theta}f_\theta(x)\,dx=0\). Using \(\frac{\partial}{\partial\theta}f_\theta(x)=f_\theta(x)U_\theta(x)\), this becomes \(E_\theta[U_\theta(X)]=0\). For a discrete model the same argument uses a sum instead of an integral.
For independent observations, the full-sample score is the sum of the individual scores. When the observations satisfy the same regularity conditions, each score has mean zero, so the cross-products between distinct scores have expectation zero. Thus the full-sample Fisher information is the sum of the individual information values. The bound uses this total information, not just the information from one observation.
Deriving the Cramér-Rao Lower Bound
The central step is to find the covariance between an unbiased estimator and the score. An unbiased estimator has expectation \(\theta\), so differentiating its expectation gives a particularly simple result. The estimator is a function of the data, not of the unknown parameter, and is held fixed when differentiating.
Write \(E_\theta[T(X)]=\int T(x)f_\theta(x)\,dx=\theta\). By the regularity condition that permits differentiation inside the integral, \(\frac{d}{d\theta}E_\theta[T(X)]=\int T(x)\frac{\partial}{\partial\theta}f_\theta(x)\,dx=1\).
Since \(\frac{\partial}{\partial\theta}f_\theta(x)=f_\theta(x)U_\theta(x)\), the derivative in Step 1 is \(E_\theta[T(X)U_\theta(X)]\). Therefore \(E_\theta[T(X)U_\theta(X)]=1\).
Because \(E_\theta[T(X)]=\theta\) and \(E_\theta[U_\theta(X)]=0\), \(\operatorname{Cov}_\theta(T(X),U_\theta(X))=E_\theta[T(X)U_\theta(X)]-E_\theta[T(X)]E_\theta[U_\theta(X)]=1\).
For random variables with finite variances, \(\operatorname{Cov}(T,U)^2\leq \operatorname{Var}(T)\operatorname{Var}(U)\). Here \(\operatorname{Cov}(T,U)=1\) and \(\operatorname{Var}(U)=E[U^2]=I_n(\theta)\).
Combining the last step’s quantities gives the Cramér-Rao lower bound for an unbiased estimator of the scalar parameter:
This derivation explains the role of Fisher information: a larger score variance permits a smaller lower bound on estimator variance. The argument depends on the covariance identity being exactly one, which in turn depends on unbiasedness for \(\theta\) and the stated regularity conditions. It does not establish this bound for a biased estimator.
Worked Examples
Worked Example: A Single Device Indicator From a Bernoulli Sample
A hypothetical quality audit records \(n=4\) independently selected devices. Let \(X_i=1\) if device \(i\) has a specified defect and \(X_i=0\) otherwise, with \(X_i\sim\operatorname{Bernoulli}(p)\), where \(0<p<1\). Consider \(T=X_1\) as an estimator of \(p\). At \(p=0.20\), calculate the Cramér-Rao lower bound and compare it with the variance of \(T\).
Method. Find the full-sample score and information, then compare the bound with the estimator’s direct variance. The observations are independent and the support \(\{0,1\}^4\) does not depend on \(p\), so the regularity conditions apply for \(0<p<1\).
For one observation, the log likelihood is \(X_i\log p+(1-X_i)\log(1-p)\). Its derivative is \(\frac{X_i}{p}-\frac{1-X_i}{1-p}=\frac{X_i-p}{p(1-p)}\). Therefore the full-sample score is \(U_p=\sum_{i=1}^4\frac{X_i-p}{p(1-p)}\). Each observation contributes information \(1/[p(1-p)]\), so the four-observation information is \(I_4(p)=4/[p(1-p)]\).
Substitute \(p=0.20\): \(p(1-p)=0.20(0.80)=0.16\), so \(I_4(0.20)=4/0.16=25\). The lower bound is \(1/25=0.04\).
The estimator is unbiased because \(E_p[T]=E_p[X_1]=p\). Its variance is \(\operatorname{Var}_p(T)=p(1-p)\), which at \(p=0.20\) is \(0.16\). Thus \(0.16\geq0.04\), as required. The bound is a lower limit for all unbiased estimators here; it does not claim that this particular estimator must attain it.
Check and common mistake. The full sample contains four observations, so its information is four times the one-observation information. Using only \(1/0.16=6.25\) as the sample information would give the incorrect bound \(0.16\). The correct sample information is \(25\), giving \(0.04\).
Worked Example: The Mean of Independent Poisson Counts
A hypothetical analyst observes counts in \(n=4\) independent, equal-length monitoring windows. Assume \(X_i\sim\operatorname{Poisson}(\lambda)\), with unknown rate \(\lambda>0\), and use \(T=\bar X\) to estimate \(\lambda\). At \(\lambda=3\), calculate the information, the lower bound, and the variance of \(T\).
Method. Differentiate the joint log likelihood to obtain the score, then use the Poisson mean and variance to calculate information. The count support is the fixed set of nonnegative integers, and the parameter is positive, so the required regularity conditions hold.
For one count, the log likelihood is \(X_i\log\lambda-\lambda-\log(X_i!)\), with score \(X_i/\lambda-1\). The full-sample score is \(U_\lambda=\sum_{i=1}^4(X_i/\lambda-1)\). Since \(E[X_i]=\lambda\) and \(\operatorname{Var}(X_i)=\lambda\), each score has mean zero and variance \(\lambda/\lambda^2=1/\lambda\). Independence gives \(I_4(\lambda)=4/\lambda\).
At \(\lambda=3\), \(I_4(3)=4/3\). The Cramér-Rao lower bound is \(1/(4/3)=3/4=0.75\).
The estimator is unbiased: \(E[\bar X]=\frac14\sum_{i=1}^4 E[X_i]=\frac14(4\lambda)=\lambda\). Its variance is \(\operatorname{Var}(\bar X)=\frac{1}{16}\sum_{i=1}^4\operatorname{Var}(X_i)=\frac{1}{16}(4\lambda)=\lambda/4\). At \(\lambda=3\), this is \(3/4=0.75\), which meets the bound.
A covariance check makes the derivation concrete. At \(\lambda=3\), write \(S=\sum X_i\), so \(T=S/4\) and \(U_\lambda=S/3-4\). Since \(\operatorname{Var}(S)=4(3)=12\), \(\operatorname{Cov}(T,U_\lambda)=\operatorname{Cov}(S/4,S/3)=12/12=1\). This matches the identity used in the proof.
Check and common mistake. The calculation uses the information from all four windows. Using only the one-window information \(1/3\) would produce the incorrect bound \(3\), larger than the actual variance \(0.75\). That contradiction is a signal that the sample information has been miscounted.
Worked Example: An Unbiased but Noisy Estimator of a Normal Mean
Suppose \(X_1,X_2,X_3\) are independent observations from \(N(\mu,4)\), where the variance \(4\) is known and the unknown parameter is the mean \(\mu\). Consider the estimator \(T=\bar X+\frac12(X_1-X_2)\). Find the lower bound for an unbiased estimator of \(\mu\), verify that \(T\) is unbiased, and compare its variance with the bound.
Method. Calculate the sample information from the normal likelihood, then calculate the variance of the stated estimator using covariance. The support is the real line for all \(\mu\), and the normal density is differentiable, so the regularity conditions apply.
For one observation, the log density has derivative \((X_i-\mu)/4\) with respect to \(\mu\). The full score is \(U_\mu=\sum_{i=1}^3(X_i-\mu)/4\). Each observation contributes information \(1/4\), so \(I_3(\mu)=3/4\), and the lower bound is \(1/(3/4)=4/3\).
The contrast \(X_1-X_2\) has expectation \(\mu-\mu=0\), while \(E[\bar X]=\mu\). Hence \(E[T]=\mu+\frac12(0)=\mu\): the estimator is unbiased.
For the variance, \(\operatorname{Var}(\bar X)=4/3\), and \(\operatorname{Var}(X_1-X_2)=4+4=8\), so \(\operatorname{Var}(\frac12(X_1-X_2))=\frac14(8)=2\). The covariance term is zero: \(\operatorname{Cov}(\bar X,X_1-X_2)=\frac13[\operatorname{Var}(X_1)-\operatorname{Var}(X_2)]=\frac13(4-4)=0\). Therefore \(\operatorname{Var}(T)=4/3+2+2(0)=10/3\), which is greater than the lower bound \(4/3\).
Check and common mistake. The added contrast has mean zero, so it does not change the estimator’s expectation, but it does add variance. Treating a zero-mean term as if it also had zero variance would incorrectly give \(4/3\); the actual variance is \(10/3\).
Common Exam Traps and Interpretation
The most frequent errors are usually about the conditions or the quantity being compared, rather than the final inequality itself.
- Using one-observation information for a sample. The bound uses the score and information for the entire observed sample. For independent observations, add the individual information values.
- Forgetting the unbiasedness requirement. The covariance identity \(\operatorname{Cov}(T,U)=1\) was derived from \(E_\theta[T]=\theta\). If the estimator is biased, this derivation does not establish the displayed bound for it.
- Assuming the bound is the estimator’s variance. The conclusion is an inequality. An estimator can have variance above the bound, as the normal example demonstrates.
- Applying the result when regularity fails. If the support depends on the parameter or differentiation cannot be passed through the expectation, the score identities used in the proof may not hold.
- Confusing information with its reciprocal. Fisher information is \(E[U^2]\); the variance lower bound is its reciprocal. For a positive information value, increasing information decreases the lower bound.
The practical value of the derivation is that it turns a general inequality into a model-based benchmark. First verify that the parameter, estimator, and regularity assumptions match the result. Then compute the full-sample information and compare the estimator’s variance with its reciprocal. This gives a disciplined check on claims about precision without implying that every unbiased estimator is equally precise.
Check Your Understanding
For each problem, identify the model and estimator before calculating. Assume the stated models satisfy the regularity conditions.
- Six independent Bernoulli observations have unknown defect probability \(p\). Find the Fisher information in the sample and the Cramér-Rao lower bound for an unbiased estimator of \(p\). Evaluate the bound at \(p=0.25\). (Answer: \(I_6(p)=6/[p(1-p)]\); the bound is \(p(1-p)/6\), or \(0.03125\) at \(p=0.25\).)
- Five independent Poisson counts have unknown mean \(\lambda=4\). For the sample mean, calculate the sample information, the bound, and the estimator’s variance. (Answer: \(I_5(4)=5/4\); the bound and sample-mean variance are both \(4/5=0.8\).)
- Eight independent observations are from \(N(\mu,16)\), with known variance \(16\) and unknown mean \(\mu\). Find the information and lower bound for an unbiased estimator of \(\mu\). (Answer: \(I_8(\mu)=8/16=1/2\), so the bound is \(2\).)
- Three independent Bernoulli observations have \(p=0.40\). For \(T=X_1\), calculate its variance and the lower bound for unbiased estimation of \(p\) from all three observations. (Answer: \(\operatorname{Var}(T)=0.40(0.60)=0.24\); the bound is \(0.40(0.60)/3=0.08\), and \(0.24\geq0.08\).)
- Four independent Poisson counts have unknown mean \(\lambda\), but an analyst proposes \(T=2\bar X\) as an estimator of \(\lambda\). Is the unbiased-estimator bound derived here justified for this \(T\)? State its expectation. (Answer: No; \(E[T]=2\lambda\), so it is biased for \(\lambda\) and the unbiased-estimator derivation does not apply.)