Applying the Cramér-Rao Bound in One-Parameter Models

Using the Bound in a Statistical Model

The Cramér–Rao lower bound turns a model’s Fisher information into a benchmark for the variance of an unbiased estimator. In practice, applying it requires three decisions: identify the unknown parameter, calculate the information in the observed sample, and confirm that the estimator is unbiased for that parameter. The bound is not a variance formula for every estimator; its standard form here applies to unbiased estimators under regularity conditions.

We apply the bound to three models commonly used in statistical work: Bernoulli observations, Poisson counts, and normal observations with known variance. All examples assume independent observations from the stated model and use the information in the full sample, unless specified otherwise.

Definition: Let $X_1,\ldots,X_n$ be independent observations from a regular one-parameter model with parameter $\theta$, and let $T$ be an estimator with finite variance satisfying $E_\theta[T]=\theta$. If $I_n(\theta)$ is the Fisher information in the full sample, the Cramér–Rao lower bound states $$ \operatorname{Var}_\theta(T)\geq \frac{1}{I_n(\theta)}. $$ The bound concerns estimators unbiased for $\theta$. It does not, in this form, give a lower bound for biased estimators or for estimators of a different function of the parameter.

For independent observations, information adds: $I_n(\theta)=nI_1(\theta)$, where $I_1(\theta)$ is the information in one observation. The model and its regularity conditions matter. The usual calculation assumes an interior parameter value, a differentiable likelihood, and conditions that permit differentiation under the expectation. In the models below, the parameter values are restricted to ranges where these conditions hold.

Why Information Gives a Variance Bound

A brief justification shows why the information appears in the denominator. Write the sample log-likelihood as $\ell(\theta)$, and define the score $U=\partial\ell(\theta)/\partial\theta$. Under the regularity conditions, $E_\theta[U]=0$ and $E_\theta[U^2]=I_n(\theta)$. If $T$ is unbiased for $\theta$, differentiating $E_\theta[T]=\theta$ with respect to $\theta$ gives $E_\theta[TU]=1$. Therefore $\operatorname{Cov}_\theta(T,U)=1$, since $E_\theta[U]=0$. The Cauchy–Schwarz inequality then implies

1=\operatorname{Cov}_\theta(T,U)^2 \leq \operatorname{Var}_\theta(T)\operatorname{Var}_\theta(U) =\operatorname{Var}_\theta(T)I_n(\theta), \qquad \operatorname{Var}_\theta(T)\geq \frac{1}{I_n(\theta)}.

The result says that, among unbiased estimators in the stated model, variance cannot be below the reciprocal of the information in the data. It does not say that every estimator has variance equal to that value.

Bernoulli Observations

Suppose $X_1,\ldots,X_n$ are independent Bernoulli random variables with success probability $p$, where $0<p<1$. Each observation is 1 for a success and 0 otherwise. For one observation $X$, the log-likelihood is $X\log p+(1-X)\log(1-p)$. Its score is $X/p-(1-X)/(1-p)$, and its Fisher information is $1/[p(1-p)]$. Independence gives the sample information and the bound below.

I_n(p)=\frac{n}{p(1-p)}, \qquad \operatorname{Var}_p(T)\geq \frac{p(1-p)}{n} \quad\text{for an unbiased estimator }T\text{ of }p.

The bound depends on the unknown $p$. When a problem supplies a value of $p$, use it to evaluate the bound. Do not replace $p$ with an observed sample proportion unless the task specifically asks for a plug-in estimate; the theoretical bound at a stated parameter value uses that parameter value.

Worked Example: Bernoulli Data and an Estimator Using Only Part of the Sample

A hypothetical quality audit records $n=30$ independent Bernoulli outcomes, with success probability $p=0.40$. Consider the estimator $T$ that is the average of only the first 10 observations. Calculate the Cramér–Rao bound based on all 30 observations and compare it with $\operatorname{Var}_p(T)$.

Method and model. The full sample consists of 30 independent Bernoulli observations with $p=0.40$. The estimator is $T=(X_1+\cdots+X_{10})/10$. Since each observation has expectation $p$, $E_p[T]=10p/10=p$; thus $T$ is unbiased for $p$.

Full-sample information. First, $p(1-p)=0.40(0.60)=0.24$. Therefore,

I_{30}(p)=\frac{30}{0.24}=125, \qquad \frac{1}{I_{30}(p)}=\frac{1}{125}=0.008.

Estimator variance. The first 10 independent observations each have variance $p(1-p)=0.24$. Hence

\operatorname{Var}_p(T) =\frac{10(0.24)}{10^2} =\frac{2.4}{100} =0.024.

The estimator’s variance, $0.024$, exceeds the full-sample bound, $0.008$. The bound uses the information in all 30 available observations even though this particular estimator discards 20 of them. The common mistake is to calculate the bound using only the 10 observations used by $T$; that would answer a different question about information in a reduced sample. Check: $0.024\geq0.008$, as required.

Poisson Observations

Suppose $X_1,\ldots,X_n$ are independent Poisson random variables with common rate $\lambda>0$. For one observation, the log-likelihood, apart from no parameter-dependent terms being omitted, is $X\log\lambda-\lambda-\log(X!)$. Its score is $X/\lambda-1$, and the information in one observation is $1/\lambda$. Thus the full-sample information is $n/\lambda$.

I_n(\lambda)=\frac{n}{\lambda}, \qquad \operatorname{Var}_\lambda(T)\geq \frac{\lambda}{n} \quad\text{for an unbiased estimator }T\text{ of }\lambda.

The parameter $\lambda$ is the expected count per observation, not the expected total count across the sample. The sum of $n$ observations has expectation $n\lambda$; the sample mean has expectation $\lambda$.

Worked Example: Comparing a Poisson Estimator with the Bound

A hypothetical monitoring system records $n=8$ independent daily counts, each with a Poisson distribution of rate $\lambda=3$. Consider $T=\bar{X}+\tfrac12(X_1-X_2)$, where $\bar{X}=(X_1+\cdots+X_8)/8$. Find the Cramér–Rao bound and the variance of $T$.

Method and unbiasedness. The sample mean satisfies $E_\lambda[\bar{X}]=\lambda$. Also, $E_\lambda[X_1-X_2]=\lambda-\lambda=0$. Consequently $E_\lambda[T]=\lambda$, so the estimator is unbiased.

Information and bound. Using the full sample size and the specified rate,

I_8(\lambda)=\frac{8}{3}, \qquad \frac{1}{I_8(\lambda)}=\frac{3}{8}=0.375.

Variance of the sample mean. Each count has variance $\lambda=3$, and the counts are independent. Therefore

\operatorname{Var}_\lambda(\bar{X})=\frac{3}{8}=0.375.

Variance of the adjustment and its covariance with the mean. Since $X_1$ and $X_2$ are independent, $\operatorname{Var}(X_1-X_2)=3+3=6$, so $\operatorname{Var}[\tfrac12(X_1-X_2)]=6/4=1.5$. Also,

\operatorname{Cov}\!\left(\bar{X},\tfrac12(X_1-X_2)\right) =\frac{1}{16}\operatorname{Cov}(X_1+\cdots+X_8,X_1-X_2) =\frac{1}{16}(3-3)=0.

It follows that $\operatorname{Var}_\lambda(T)=0.375+1.5+2(0)=1.875$. This exceeds the bound of $0.375$. The common mistake is to treat the adjustment as if it were uncorrelated with $\bar{X}$ without checking; here the covariance is zero because the contributions from $X_1$ and $X_2$ cancel. Check: $1.875\geq0.375$.

Normal Observations with Known Variance

Suppose $X_1,\ldots,X_n$ are independent and identically distributed as $N(\mu,\sigma^2)$, with unknown mean $\mu$ and known variance $\sigma^2>0$. The parameter is $\mu$; $\sigma^2$ is held fixed, not estimated. The score from one observation is $(X-\mu)/\sigma^2$, so its variance, and hence its Fisher information, is $1/\sigma^2$. The sample information and bound are

I_n(\mu)=\frac{n}{\sigma^2}, \qquad \operatorname{Var}_\mu(T)\geq\frac{\sigma^2}{n} \quad\text{for an unbiased estimator }T\text{ of }\mu.

This formula requires that the variance be known and constant as $\mu$ varies. If both normal parameters are unknown, or if the target is the variance rather than the mean, this one-parameter calculation is not the appropriate bound.

Worked Example: A Normal-Mean Estimator with an Added Contrast

A hypothetical instrument produces $n=4$ independent readings from $N(\mu,9)$, where $\mu$ is unknown and the variance $9$ is known. Consider $T=\bar{X}+\tfrac14(X_1-X_2)$. Calculate the bound and the estimator’s variance.

Method and unbiasedness. The average has expectation $\mu$, while $E_\mu[X_1-X_2]=\mu-\mu=0$. Thus $E_\mu[T]=\mu$, and $T$ is unbiased.

Information and bound. Here $n=4$ and $\sigma^2=9$, so

I_4(\mu)=\frac{4}{9}, \qquad \frac{1}{I_4(\mu)}=\frac{9}{4}=2.25.

Variance calculation. The sample mean has variance $9/4=2.25$. Independence gives $\operatorname{Var}(X_1-X_2)=9+9=18$, so the added contrast has variance $18/16=1.125$. Its covariance with the mean is

\operatorname{Cov}\!\left(\bar{X},\tfrac14(X_1-X_2)\right) =\frac{1}{16}\operatorname{Cov}(X_1+X_2+X_3+X_4,X_1-X_2) =\frac{1}{16}(9-9)=0.

Therefore $\operatorname{Var}_\mu(T)=2.25+1.125+2(0)=3.375$, which is above the bound $2.25$. The common mistake is to use the standard deviation $3$ in the information formula in place of the variance $9$. The information is $n/\sigma^2=4/9$, not $n/3$. Check: $3.375\geq2.25$.

Choosing and Interpreting the Bound

In each model, the calculation follows the same sequence. Identify the one unknown parameter and its model, use the full sample size to obtain Fisher information, take its reciprocal, then compare that value with the variance of an estimator only if the estimator is unbiased for the parameter. A bound is a benchmark, not evidence by itself that an estimator is good in every respect: it does not assess bias, and the formulas above do not address estimation of a different parameter function.

State the model.
For example, specify Bernoulli with parameter $p$, Poisson with rate $\lambda$, or normal with unknown $\mu$ and known $\sigma^2$.

Calculate full-sample information.
Use $n/[p(1-p)]$, $n/\lambda$, or $n/\sigma^2$, respectively.

Take the reciprocal and check applicability.
Confirm that the estimator is unbiased for the stated parameter and that the model conditions for the bound hold.

A useful final check is dimensional and numerical. The bound must be nonnegative and have the units of a variance. It must not exceed the variance of an unbiased estimator calculated under the same model and parameter value. If a comparison fails, recheck the sample size, the parameter used in the information formula, and whether a variance or standard deviation was substituted.

Key takeaway: For an unbiased estimator of the one unknown parameter, the standard Cramér–Rao bound is the reciprocal of the Fisher information in the full sample. In Bernoulli, Poisson, and normal mean models with known variance, that bound is $p(1-p)/n$, $\lambda/n$, and $\sigma^2/n$, respectively.

Check Your Understanding

For each problem, identify the applicable one-parameter model and use the stated full sample size.

There are 25 independent Bernoulli observations with success probability $p=0.40$. What is the Cramér–Rao bound for an unbiased estimator of $p$? (Answer: $0.0096$)
There are 50 independent Bernoulli observations with $p=0.20$. An estimator is the mean of only the first 10 observations. Find the full-sample bound and the variance of this estimator. (Answer: Bound $=0.0032$; estimator variance $=0.016$)
Nine independent Poisson counts have rate $\lambda=6$. Calculate the Cramér–Rao bound for an unbiased estimator of $\lambda$. (Answer: $6/9=0.6667$)
Twelve independent observations have distribution $N(\mu,25)$, where the variance is known and $\mu$ is unknown. Find the bound for an unbiased estimator of $\mu$. (Answer: $25/12=2.0833$)
Six independent Poisson counts have rate $\lambda=3$. For $T=\bar{X}+\tfrac13(X_1-X_2)$, find its expectation, variance, and the full-sample Cramér–Rao bound. (Answer: $E[T]=3$; $\operatorname{Var}(T)=7/6=1.1667$; bound $=1/2=0.5000$)

What You'll Learn