Tutorials › VEE Mathematical Statistics › Equality Conditions and Efficient Estimators

Home  /  Tutorials  /  VEE Mathematical Statistics / Bias, Variance, and Efficiency

VEE Mathematical Statistics · Bias, Variance, and Efficiency · Tutorial 496 of 880

Equality Conditions and Efficient Estimators

Use the score function and the equality case of Cauchy–Schwarz to identify when an unbiased estimator attains the Cramér–Rao bound.

Intermediate 11 min read

What You'll Learn

  • State the regularity conditions needed for the equality result.
  • Derive the Cramér–Rao equality condition from the score identity.
  • Define an efficient unbiased estimator in a one-parameter model.
  • Verify equality in Bernoulli and normal sampling models.
  • Diagnose an unbiased estimator whose variance exceeds the bound.
  • Avoid confusing attainment of the bound with unbiasedness alone.

Why the Equality Condition Matters

The Cramér–Rao lower bound gives a variance that no unbiased estimator can beat under suitable regularity conditions. But the bound is not automatically attainable: it is a benchmark, and an estimator may have variance strictly above it. The equality condition tells us exactly when the benchmark is reached.

The key idea is that an estimator attains the bound only when its deviation from the parameter is proportional to the score. This gives a practical test: calculate the score, compare it with the centered estimator, and check whether the required proportionality holds with probability one.

Definition: Let \(X\) have a regular one-parameter model with parameter \(\theta\), and let \(T=T(X)\) be an unbiased estimator of \(\theta\), so \(E_\theta[T]=\theta\). Write \(U_\theta(X)=\frac{\partial}{\partial\theta}\log f_\theta(X)\) for the score and \(I(\theta)=E_\theta[U_\theta^2]\) for the Fisher information. If \(0<I(\theta)<\infty\), the Cramér–Rao bound is \(\operatorname{Var}_\theta(T)\geq 1/I(\theta)\). The estimator \(T\) is efficient at \(\theta\) if it is unbiased and \(\operatorname{Var}_\theta(T)=1/I(\theta)\).

Here \(f_\theta\) denotes the probability mass function or density, as appropriate. For a sample of independent observations, the score is the derivative of the log of the joint mass function or density, and the Fisher information is computed for that whole sample. Efficiency is a property of an estimator in a specified model, not a label that follows merely from unbiasedness.

Deriving the Equality Condition

Assume the usual regularity conditions for the one-parameter Cramér–Rao result: the support does not depend on \(\theta\), the model is differentiable in \(\theta\), differentiation can be interchanged with integration or summation, and the score has finite positive variance. Also assume that \(T\) has finite variance and is unbiased. These conditions give \(E_\theta[U_\theta]=0\).

Differentiate the unbiasedness identity \(E_\theta[T]=\theta\). Since \(T\) is a function of the data and not of \(\theta\), the regularity conditions allow the derivative to pass through the expectation:

$$ \frac{d}{d\theta}E_\theta[T] =E_\theta[TU_\theta] =1. $$

Because \(E_\theta[U_\theta]=0\) and \(E_\theta[T]=\theta\), this also says that the covariance between \(T\) and the score is one. Applying the Cauchy–Schwarz inequality to the centered estimator \(T-\theta\) and \(U_\theta\) gives the bound and its equality condition.

1
Use the score identity.
\(E_\theta[(T-\theta)U_\theta]=E_\theta[TU_\theta]-\theta E_\theta[U_\theta]=1\).
2
Apply Cauchy–Schwarz.
\(1^2\leq E_\theta[(T-\theta)^2]E_\theta[U_\theta^2]=\operatorname{Var}_\theta(T)I(\theta)\), so \(\operatorname{Var}_\theta(T)\geq 1/I(\theta)\).
3
Use the equality case.
Equality in Cauchy–Schwarz holds exactly when the two centered random variables are proportional with probability one. The score identity fixes the proportionality constant.

For clarity, suppose equality holds and write \(T-\theta=cU_\theta\) with probability one. Taking the expectation of the product with \(U_\theta\) gives \(1=cI(\theta)\), so \(c=1/I(\theta)\). Conversely, if this proportionality holds, its variance is exactly the bound. Therefore:

$$ \operatorname{Var}_\theta(T)=\frac{1}{I(\theta)} \quad\Longleftrightarrow\quad T-\theta=\frac{U_\theta}{I(\theta)} \ \text{with probability one}. $$

This is a model-specific condition. For an estimator to be efficient throughout a parameter space, the identity must hold for every parameter value in that space. An identity that holds only at one value establishes efficiency only at that value.

Worked Examples

Worked Example: The Sample Proportion in a Bernoulli Model

Five independent hypothetical device inspections each record whether a component passes a stress test. Let \(X_i=1\) for a pass and \(X_i=0\) otherwise, with \(X_i\sim\operatorname{Bernoulli}(p)\), where \(0<p<1\). Determine whether the sample proportion \(T=\bar X\) is efficient for estimating \(p\). For the observed outcomes \(1,0,1,1,0\), also check the equality condition at \(p=0.4\).

Method. Calculate the sample Fisher information and compare the estimator's variance with its Cramér–Rao bound. For one Bernoulli observation, the log likelihood is \(x\log p+(1-x)\log(1-p)\), so its score is \((x-p)/(p(1-p))\). For five independent observations, the score is

$$ U_p=\frac{\sum_{i=1}^5 X_i-5p}{p(1-p)},\qquad I(p)=\frac{5}{p(1-p)}. $$

The estimator is unbiased, since \(E_p[\bar X]=p\), and its variance is \(\operatorname{Var}_p(\bar X)=p(1-p)/5\). The bound is

$$ \frac{1}{I(p)}=\frac{p(1-p)}{5} =\operatorname{Var}_p(\bar X). $$

Thus \(\bar X\) is efficient. The equality condition also holds directly:

$$ \frac{U_p}{I(p)} =\frac{\sum_{i=1}^5X_i-5p}{5} =\bar X-p. $$

For the specified observations, \(\sum X_i=3\), so \(\bar X=3/5=0.6\). At \(p=0.4\), the score is \((3-2)/0.24=25/6\), and \(I(0.4)=5/0.24=125/6\). Hence \(U_p/I(p)=1/5=0.2=\bar X-p\), as required. The variance and bound at \(p=0.4\) are both \(0.4(0.6)/5=0.048\). The common mistake this avoids is using the variance of one Bernoulli observation, \(p(1-p)\), instead of the variance of the five-observation sample proportion.

Worked Example: The Sample Mean with Known Normal Variance

Suppose three independent hypothetical sensor errors \(X_1,X_2,X_3\) follow \(N(\mu,4)\), where the variance \(4\) is known and \(\mu\) is unknown. For observations \(8,10,12\), determine whether \(\bar X\) is efficient for estimating \(\mu\), and verify the equality condition at \(\mu=9\).

Method. The log density of one observation, up to a term not involving \(\mu\), is \(-(x-\mu)^2/8\), giving score \((x-\mu)/4\). Thus the sample score and information are

$$ U_\mu=\frac{\sum_{i=1}^3X_i-3\mu}{4}, \qquad I(\mu)=\frac{3}{4}. $$

The sample mean is unbiased, and independence gives \(\operatorname{Var}(\bar X)=4/3\). The Cramér–Rao bound is \(1/I(\mu)=4/3\), so the estimator is efficient. In fact, for every sample,

$$ \frac{U_\mu}{I(\mu)} =\frac{\sum_{i=1}^3X_i-3\mu}{3} =\bar X-\mu. $$

For the given data, \(\bar X=(8+10+12)/3=10\). At \(\mu=9\), the score is \((30-27)/4=3/4\), and \(U_\mu/I(\mu)=(3/4)/(3/4)=1=\bar X-\mu\). The variance and bound both equal \(4/3\). The common mistake this avoids is treating the known variance \(4\) as a standard deviation; the model is \(N(\mu,4)\), so the standard deviation is \(2\), while the information uses the variance \(4\).

Worked Example: An Unbiased Estimator That Does Not Attain the Bound

Two independent hypothetical quality checks have outcomes \(X_1,X_2\sim\operatorname{Bernoulli}(p)\), with \(0<p<1\). Consider \(T=X_1\) as an estimator of \(p\). Determine whether it is efficient and quantify how its variance compares with the bound.

Method. The estimator is unbiased because \(E_p[X_1]=p\), and its variance is \(p(1-p)\). For the two-observation sample, the information is \(I(p)=2/[p(1-p)]\), so the bound is \(p(1-p)/2\). The variance of \(T\) is twice that bound, not equal to it.

The equality condition shows precisely why. The sample score divided by the information is

$$ \frac{U_p}{I(p)} =\frac{X_1+X_2-2p}{2}, $$

whereas \(T-p=X_1-p\). Their difference is

$$ (T-p)-\frac{U_p}{I(p)} =\frac{X_1-X_2}{2}. $$

This difference is nonzero whenever the checks disagree. That event has probability \(P(X_1\ne X_2)=2p(1-p)>0\), so the equality condition fails. At \(p=0.25\), \(\operatorname{Var}(T)=0.25(0.75)=0.1875\), while \(I(p)=2/0.1875=32/3\) and the bound is \(3/32=0.09375\). Thus \(T\) is unbiased but not efficient. The common mistake this avoids is concluding that an unbiased estimator must attain the Cramér–Rao bound.

Interpreting Efficiency and Avoiding Traps

Efficiency is defined relative to a specified model and its Fisher information. It means that an unbiased estimator has variance equal to the lower bound under the stated regularity conditions. It does not, by itself, establish that an estimator is best under every possible criterion, nor does it apply when the estimator is biased under the definition used here.

A frequent exam error is to check only that the variance matches a number that looks like a lower bound. A complete argument should identify the model, calculate information for the full sample, verify unbiasedness, and compare the variance with \(1/I(\theta)\). When useful, the score identity provides an additional direct check: \(T-\theta=U_\theta/I(\theta)\) must hold with probability one, not merely for one observed data set.

The regularity assumptions matter as well. If a model has parameter-dependent support or fails the differentiation conditions, the standard score identity or Cramér–Rao result may not apply in its usual form. One should not infer efficiency from a bound without first confirming that the model falls within the regular setting.

Key takeaway: Under the regular one-parameter conditions, an unbiased estimator \(T\) attains the Cramér–Rao bound exactly when its centered value is the score divided by the Fisher information: \(T-\theta=U_\theta/I(\theta)\) with probability one.

Check Your Understanding

For each problem, check unbiasedness and compare the estimator's variance with the appropriate Cramér–Rao bound.

  1. Let \(X_1,\ldots,X_4\) be independent \(\operatorname{Bernoulli}(p)\) observations, with \(0<p<1\). Is \(\bar X\) efficient for estimating \(p\)? State the bound. (Answer: Yes; \(1/I(p)=p(1-p)/4\).)
  2. Let \(X_1,X_2,X_3\) be independent Poisson random variables with parameter \(\lambda>0\). Is \(\bar X\) efficient for estimating \(\lambda\), and what is the bound? (Answer: Yes; \(I(\lambda)=3/\lambda\), so the bound is \(\lambda/3\).)
  3. Let \(X_1,X_2\) be independent \(N(\mu,9)\) observations, where the variance is known. Is \((X_1+X_2)/2\) efficient for estimating \(\mu\)? (Answer: Yes; its variance and the bound are both \(9/2\).)
  4. For three independent \(\operatorname{Bernoulli}(p)\) observations, use \(T=X_1\) to estimate \(p\). Is it efficient, and what is its variance as a multiple of the bound? (Answer: No; \(\operatorname{Var}(T)=p(1-p)\), three times the bound \(p(1-p)/3\).)
  5. Let \(X_1,X_2\) be independent \(N(\mu,1)\) observations. Consider \(T=(3X_1-X_2)/2\). Is it efficient for estimating \(\mu\)? (Answer: No; it is unbiased with variance \(10/4=2.5\), while the bound is \(1/2\).)

Previous

Next

Back to Tutorials
Clinical Biostats