Deriving the Parametric Asymptotic Information Bound

Why an Information Bound Appears

An estimator can vary from sample to sample, but a regular estimator must also respond correctly when the parameter changes slightly. In a smooth parametric model, the score records how the likelihood responds to such a change. The key link is a covariance: an estimator whose mean tracks the parameter has a non-negligible covariance with the score. Cauchy–Schwarz then limits how small its variance can be.

This tutorial derives that scalar bound in an independent, identically distributed model. The conclusion is about first-order asymptotic variance, not the exact variance of every estimator at every sample size. To make the derivation precise, we state the differentiability and moment conditions needed for the covariance argument.

Score and Fisher Information

Definition: Let $X_1,\ldots,X_n$ be independent observations with density or mass function $f(x;\theta)$, where the scalar parameter $\theta$ belongs to an open interval. The one-observation score is $$ U_\theta(X)=\frac{\partial}{\partial\theta}\log f(X;\theta), $$ and the Fisher information per observation is $$ I(\theta)=E_\theta[U_\theta(X)^2]. $$ The sample score is $S_n(\theta)=\sum_{i=1}^n U_\theta(X_i)$.

For the derivation below, assume the model has a common support that does not depend on $\theta$; the density or mass function is differentiable in $\theta$; differentiation can be passed under the integral or sum defining expectations; and $0<I(\theta)<\infty$. These conditions imply that the score has mean zero. Indeed, differentiating the total probability gives

E_\theta[U_\theta(X)] =\int \frac{\partial}{\partial\theta}f(x;\theta)\,dx =\frac{\partial}{\partial\theta}\int f(x;\theta)\,dx =0.

The integral notation also represents a sum for a discrete model. Independence therefore gives $E_\theta[S_n(\theta)]=0$ and $\operatorname{Var}_\theta(S_n(\theta))=nI(\theta)$: the cross-covariances between distinct observations’ scores vanish.

The Score Covariance Identity

Let $T_n=T_n(X_1,\ldots,X_n)$ be an estimator whose definition does not depend on the unknown $\theta$. Suppose $E_\theta[T_n]$ is finite and differentiation under the integral defining this expectation is valid. The joint density is $f_n(x_1,\ldots,x_n;\theta)=\prod_{i=1}^n f(x_i;\theta)$. Differentiating this product’s log gives

\frac{\partial}{\partial\theta}\log f_n(x_1,\ldots,x_n;\theta) =S_n(\theta).

Consequently,

\frac{\partial}{\partial\theta}E_\theta[T_n] =\int T_n(x)\frac{\partial}{\partial\theta}f_n(x;\theta)\,dx =E_\theta[T_nS_n(\theta)].

Since the sample score has mean zero, this derivative is also a covariance:

\frac{\partial}{\partial\theta}E_\theta[T_n] =\operatorname{Cov}_\theta(T_n,S_n(\theta)).

This identity says that the estimator’s mean response to changes in $\theta$ is encoded by its covariance with the score. It does not require $T_n$ to be exactly unbiased. It does require the stated differentiation and integrability conditions; if, for example, the support changes with $\theta$, additional boundary terms may invalidate this version of the identity.

Worked Example: Normal Mean With Known Variance

Suppose $X_i\sim N(\theta,\sigma^2)$, independently, where $\sigma^2$ is known and positive. For one observation,

\ell(\theta;x)=-\frac12\log(2\pi\sigma^2)-\frac{(x-\theta)^2}{2\sigma^2}, \qquad U_\theta(x)=\frac{x-\theta}{\sigma^2}.

Because $E_\theta[X-\theta]=0$ and $\operatorname{Var}_\theta(X)=\sigma^2$, the information is

I(\theta)=E_\theta\left[\left(\frac{X-\theta}{\sigma^2}\right)^2\right] =\frac{\sigma^2}{\sigma^4} =\frac{1}{\sigma^2}.

For the sample mean $\bar X$, $E_\theta[\bar X]=\theta$, so its derivative with respect to $\theta$ is $1$. Its covariance with the sample score is therefore $1$. Its variance is $\sigma^2/n$, and the bound derived below is $1/(nI(\theta))=\sigma^2/n$. The quantities agree in this model.

Finite-Sample Inequality and Asymptotic Bound

Theorem: Under the differentiability and information conditions above, suppose $T_n$ has finite variance and its mean is differentiable in $\theta$. Then $$ \operatorname{Var}_\theta(T_n) \geq \frac{\left\{\dfrac{\partial}{\partial\theta}E_\theta[T_n]\right\}^2}{nI(\theta)}. $$ In particular, if $\dfrac{\partial}{\partial\theta}E_\theta[T_n]\to 1$ at the parameter value of interest, then $$ \liminf_{n\to\infty} n\operatorname{Var}_\theta(T_n)\geq \frac{1}{I(\theta)}. $$

The first inequality follows directly from the score covariance identity and Cauchy–Schwarz. The second follows by multiplying the first by $n$ and taking the lower limit, since $I(\theta)$ is fixed, positive, and finite.

Use the covariance identity.
$\operatorname{Cov}_\theta(T_n,S_n)=\dfrac{\partial}{\partial\theta}E_\theta[T_n]$.

Apply Cauchy–Schwarz.
$\operatorname{Cov}_\theta(T_n,S_n)^2\leq \operatorname{Var}_\theta(T_n)\operatorname{Var}_\theta(S_n)$.

Substitute the score variance.
$\operatorname{Var}_\theta(S_n)=nI(\theta)$, so division by this positive quantity gives the claimed inequality.

Equivalently, define the scaled estimator error $Z_n=\sqrt n(T_n-\theta)$ and normalized score $\Delta_n=S_n(\theta)/\sqrt n$. Their variances and covariance satisfy

\operatorname{Var}_\theta(Z_n)=n\operatorname{Var}_\theta(T_n),\qquad \operatorname{Var}_\theta(\Delta_n)=I(\theta),\qquad \operatorname{Cov}_\theta(Z_n,\Delta_n) =\frac{\partial}{\partial\theta}E_\theta[T_n].

Thus, when the derivative of the estimator’s mean tends to $1$, the scaled error must retain asymptotic covariance $1$ with a score whose variance is $I(\theta)$. Cauchy–Schwarz then requires its variance to be at least $1/I(\theta)$.

The derivative condition has a local interpretation. Under a change from $\theta$ to $\theta+h/\sqrt n$, the mean of a well-calibrated estimator should track the target’s change of $h/\sqrt n$, to first order. For the direct argument above, the needed condition is specifically $\partial E_\theta[T_n]/\partial\theta\to1$. A local distributional regularity statement alone does not automatically give this moment condition: suitable integrability and differentiability are needed to pass from distributional behavior to mean response. If the scaled errors also have convergent second moments, the bound applies to the variance of their limiting distribution. Without such moment control, a limit in distribution by itself does not ensure convergence of variances.

Worked Example: Poisson Rate

Let $X_i\sim\operatorname{Poisson}(\theta)$, independently, with $\theta>0$. For one observation, $\ell(\theta;x)=x\log\theta-\theta-\log(x!)$, so

U_\theta(x)=\frac{x}{\theta}-1, \qquad I(\theta)=E_\theta\left[\left(\frac{X}{\theta}-1\right)^2\right] =\frac{\operatorname{Var}_\theta(X)}{\theta^2} =\frac{\theta}{\theta^2} =\frac1\theta.

The estimator $T_n=\bar X$ has mean $\theta$, hence mean derivative $1$. Its variance is $\theta/n$, while the bound is

\frac{1}{nI(\theta)}=\frac{\theta}{n}.

Thus its scaled variance $n\operatorname{Var}_\theta(\bar X)=\theta$ is exactly the reciprocal information $1/I(\theta)=\theta$.

Worked Example: A Bernoulli Estimator Using Half the Sample

Suppose $X_i\sim\operatorname{Bernoulli}(p)$, independently, where $0<p<1$. The score and information for one observation are

U_p(X)=\frac{X-p}{p(1-p)}, \qquad I(p)=\frac{1}{p(1-p)}.

Let $m=\lfloor n/2\rfloor$ and estimate $p$ using only the first $m$ observations: $T_n=m^{-1}\sum_{i=1}^m X_i$. Its expectation is $p$, so the mean derivative is $1$. The bound for $n$ observations is

\frac{1}{nI(p)}=\frac{p(1-p)}{n}.

But the estimator’s actual variance is $p(1-p)/m$, and therefore

n\operatorname{Var}_p(T_n) =\frac{n}{m}p(1-p) \longrightarrow 2p(1-p).

The asymptotic variance is twice the lower bound $p(1-p)$. The covariance calculation shows why the bound still applies: the full sample score is $S_n=\sum_{i=1}^n(X_i-p)/[p(1-p)]$. Independence gives

\operatorname{Cov}_p(T_n,S_n) =\frac{1}{mp(1-p)}\sum_{i=1}^m\operatorname{Var}_p(X_i) =1.

Cauchy–Schwarz with $\operatorname{Var}_p(S_n)=n/[p(1-p)]$ yields $\operatorname{Var}_p(T_n)\geq p(1-p)/n$, as required. Using only half the observations increases the estimator’s variance, but does not change the mean response that drives the lower bound.

Common Mistake and Practical Meaning

A common mistake is to claim that every estimator’s variance is at least $1/[nI(\theta)]$ without checking its mean response. The finite-sample inequality has the squared derivative $\{\partial E_\theta[T_n]/\partial\theta\}^2$ in its numerator. If the estimator’s mean does not respond to $\theta$, that derivative may be zero, and this argument gives no positive lower bound. The correction is to verify the score covariance identity’s conditions and establish the relevant derivative; for the asymptotic bound stated here, that derivative must tend to $1$.

The information bound is useful as a benchmark: it quantifies how much first-order variance remains unavoidable in a specified model. The result depends on the model’s information and on regular local behavior; it is not a guarantee that an arbitrary estimator has small variance, nor does it establish that a particular estimator attains the bound.

Check Your Understanding

Use the covariance argument and its assumptions to answer these questions.

Why does the sample score have variance $nI(\theta)$ under independent sampling?
Which differentiation step produces the covariance between $T_n$ and the sample score?
If $\partial E_\theta[T_n]/\partial\theta\to 0.8$, what lower bound does the argument give for $\liminf n\operatorname{Var}_\theta(T_n)$?
For the Poisson model in the worked example, what is the information bound when $\theta=4$ and $n=20$?
Why is convergence in distribution of scaled estimation errors alone insufficient to conclude that their variances converge?