Worked Efficiency Audit For Exponential-Rate Estimators

Set-Up: Estimating an Exponential Rate

Suppose \(X_1,\ldots,X_n\) are independent and identically distributed exponential random variables with rate \(\lambda>0\). Their density is \(f(x;\lambda)=\lambda e^{-\lambda x}\) for \(x\geq 0\), and zero otherwise. The mean is \(1/\lambda\), so larger rates correspond to shorter average waiting times. We will compare the maximum-likelihood estimator, the reciprocal sample mean, and a simple unbiased correction. A key point is that the first two names describe exactly the same estimator in this model.

The likelihood and log-likelihood, for observed positive values, are

Differentiating gives \(\ell'(\lambda)=n/\lambda-S\). The log-likelihood increases when \(\lambda<n/S\) and decreases when \(\lambda>n/S\), since \(\ell''(\lambda)=-n/\lambda^2<0\). Thus its unique maximum is at \(n/S=1/\bar X\). The reciprocal sample mean is not a second, distinct estimator: it is the MLE itself.

Regularity and the Information Bound

For one observation, the score is \(U(\lambda;X)=\partial\log f(X;\lambda)/\partial\lambda=1/\lambda-X\). It has mean zero and variance \(1/\lambda^2\), because \(E[X]=1/\lambda\) and \(\operatorname{Var}(X)=1/\lambda^2\). The expected negative second derivative of the log density is also \(1/\lambda^2\). Thus the Fisher information in one observation is \(I_1(\lambda)=1/\lambda^2\), and for \(n\) independent observations it is \(I_n(\lambda)=n/\lambda^2\).

The model is regular for \(\lambda>0\): the parameter lies in an open interval, the support \(x\geq0\) does not depend on \(\lambda\), the log density is twice differentiable in \(\lambda\), and the score has finite variance. For the Cramér–Rao bound below, an estimator \(T\) must also have finite variance, satisfy \(E_\lambda[T]=\lambda\), and permit differentiation of its expectation under the integral. These are the conditions that yield the score-covariance identity.

To see the bound, differentiate \(E_\lambda[T]=\lambda\) with respect to \(\lambda\). The stipulated interchange of differentiation and integration gives \(E_\lambda[TU_n]=1\). Also \(E_\lambda[U_n]=0\), so \(\operatorname{Cov}_\lambda(T,U_n)=1\). The Cauchy–Schwarz inequality then gives

Rearranging proves the bound. Equality in Cauchy–Schwarz requires \(T-E_\lambda[T]\) to be a constant multiple of \(U_n\); the covariance identity fixes that constant as \(1/I_n(\lambda)\). This also gives the stated equality condition. \(\square\)

Exact Biases and Variances

The sum \(S\) has a gamma distribution with shape \(n\) and rate \(\lambda\), with density \(g(s)=\lambda^n s^{n-1}e^{-\lambda s}/\Gamma(n)\) for \(s>0\). For \(n>1\), integrating this density gives \(E[1/S]=\lambda/(n-1)\). For \(n>2\), it also gives \(E[1/S^2]=\lambda^2/((n-1)(n-2))\). For example, the second identity follows by integrating \(s^{-2}g(s)\) and using \(\Gamma(n)=(n-1)(n-2)\Gamma(n-2)\). These conditions matter: the second inverse moment, and hence the variances below, is finite only for \(n>2\).

The expectations follow by multiplying \(E[1/S]\) by the respective numerators. For the MLE variance, subtract the squared mean from its second moment:

For the corrected estimator, the corresponding calculation is

The unbiased estimator does not attain the finite-sample information bound: \(\lambda^2/(n-2)>\lambda^2/n\) for \(n>2\). It also fails the equality condition. The score is linear in \(S\), whereas \((n-1)/S\) is reciprocal in \(S\), so they cannot satisfy the required affine relationship almost surely.

First-Order Asymptotic Comparison

The variance formulas describe finite samples, but their leading terms agree. To verify the limiting distributions, note that \(E[X_i]=1/\lambda\) and \(\operatorname{Var}(X_i)=1/\lambda^2\). The central limit theorem gives \(\sqrt n(\bar X-1/\lambda)\xrightarrow{d}N(0,1/\lambda^2)\). The function \(h(x)=1/x\) is differentiable at \(1/\lambda>0\), with derivative \(-\lambda^2\). The delta method therefore yields

This establishes the MLE's asymptotic variance \(\lambda^2/n\) to first order, equal to \(1/I_n(\lambda)\). The unbiased correction is \((n-1)/n\) times the MLE. Since \((n-1)/n\to1\), and the MLE converges in probability to \(\lambda\) by the law of large numbers and continuity of \(1/x\), the difference between the two estimators is negligible on the \(\sqrt n\) scale:

Slutsky's theorem gives the same limiting distribution for \(\tilde{\lambda}\). This agrees with the exact variance comparison: \(n\operatorname{Var}(\tilde{\lambda})=n\lambda^2/(n-2)\to\lambda^2\), and \(n\operatorname{Var}(\hat{\lambda}_{\mathrm{MLE}})=n^3\lambda^2/((n-1)^2(n-2))\to\lambda^2\). The MLE's bias, \(\lambda/(n-1)\), is also of smaller order than \(n^{-1/2}\).

Interpretation and Common Pitfall

This audit separates three questions that are easy to conflate: whether an estimator is unbiased, whether it reaches a finite-sample information bound, and whether it has the efficient first-order asymptotic variance. Here, the MLE and reciprocal sample mean are identical and biased for finite \(n\); the adjusted estimator is unbiased but does not attain the finite-sample bound; and both have the same first-order asymptotic variance.

• Wrong reasoning: “The reciprocal sample mean and the MLE are two competing procedures, so their variances should be compared separately.” Correction: They are algebraically identical, since \(1/\bar X=n/S\).
• Wrong reasoning: “Because the adjusted estimator is unbiased, it must attain the Cramér–Rao bound.” Correction: Unbiasedness is required to apply that bound, not sufficient for equality. The equality condition must also hold, and it does not hold here.
• Wrong reasoning: “The unbiased bound directly evaluates the MLE.” Correction: The usual bound stated here is for unbiased estimators. The MLE is biased at finite \(n\), so its bias must not be ignored when interpreting a variance comparison.