Why an Information Bound Needs Regularity
An information bound is not a universal lower limit on the variance of every estimator. It is a conclusion drawn under particular assumptions about how a model changes with its parameter. When those assumptions fail, a formal calculation of Fisher information may still be possible, but the usual interpretation as a first-order efficiency bound may not be.
The scalar Cramér–Rao inequality is the basic finite-sample illustration. Its familiar derivation requires a score with mean zero and an identity linking the score to the derivative of an estimator's expectation. For asymptotic efficiency bounds, further regularity is needed: in particular, the parameter should be locally identifiable and interior to a suitably smooth model, and the estimator should have regular first-order behavior. This tutorial examines three ways those requirements can break: a boundary, nonidentification, and parameter-dependent support.
The Regular Information Inequality
Here is a precise scalar version of the Cramér–Rao inequality. Suppose \(\theta\) ranges over an open interval, the model has densities \(f_\theta\) with respect to a common measure and parameter-independent support, and differentiation can be passed under the integral in the identities below. Assume the score exists, has finite second moment, and satisfies \(E_\theta[U_\theta(X)]=0\). If an estimator \(T\) has finite variance and its expectation \(m(\theta)=E_\theta[T]\) is differentiable with \(m'(\theta)=E_\theta[TU_\theta(X)]\), then, whenever \(I(\theta)>0\),
For an unbiased estimator, \(m(\theta)=\theta\), so the numerator is one. Under independent sampling of size \(n\), the information is \(nI(\theta)\), giving the familiar lower bound \(1/[nI(\theta)]\). The conditions are substantive: without the score identity or the derivative identity, the covariance argument does not go through.
The asymptotic information bound for regular estimators has a related role: it describes the best first-order variance available in a locally smooth, identifiable model. It is not a guarantee that every MLE is asymptotically normal, nor that every estimator has variance at least the bound at every individual parameter value. At a boundary, under nonidentification, or when the support changes with the parameter, the regular argument can fail at its foundations.
Worked Example: A Normal Mean at the Boundary
Worked Example: A Constrained Normal-Mean MLE
Let \(X_1,\ldots,X_n\) be independent \(N(\theta,\sigma^2)\) observations, where \(\sigma^2\) is known and the parameter space is \(\theta\geq 0\). The unrestricted sample mean is \(\bar X\), but maximizing the likelihood over the constrained parameter space gives \(\hat\theta=\max(0,\bar X)\). Consider the true value \(\theta=0\).
At this boundary point, \(\sqrt n\bar X/\sigma\) has the standard normal distribution for every \(n\). Thus \[ \frac{\sqrt n\,\hat\theta}{\sigma} =\max\left(0,\frac{\sqrt n\,\bar X}{\sigma}\right) \ \xrightarrow{d}\ \max(0,Z),\qquad Z\sim N(0,1). \] The limit has probability \(1/2\) at zero, so it is not a normal distribution. The usual interior-point conclusion that the MLE is asymptotically normal with variance \(\sigma^2/n\) does not describe this boundary behavior.
The boundary estimator is also biased at \(\theta=0\). Since \(E[\max(0,Z)]=1/\sqrt{2\pi}\) and \(E[\max(0,Z)^2]=1/2\), \[ E_0[\hat\theta]=\frac{\sigma}{\sqrt{2\pi n}},\qquad \operatorname{Var}_0(\hat\theta) =\frac{\sigma^2}{n}\left(\frac12-\frac{1}{2\pi}\right). \] The unrestricted \(\bar X\) has variance \(\sigma^2/n\), but it is not constrained to be nonnegative. Neither comparison turns the regular information bound into a universal pointwise variance rule: the constrained MLE is biased and has a nonnormal limit at the boundary.
The likelihood has the ordinary normal score and per-observation information \(1/\sigma^2\). What fails is not the algebraic calculation of that information; the true parameter is not an interior point of the allowed parameter space, so the usual two-sided local regularity argument for the MLE does not apply.
Worked Example: Nonidentification Makes a Parameter Unrecoverable
Worked Example: A Normal Model Depending on the Square of the Parameter
Suppose \(X_1,\ldots,X_n\) are independent \(N(\theta^2,1)\) observations with \(\theta\in\mathbb R\). The distribution depends on \(\theta\) only through \(\theta^2\). In particular, \(\theta=1\) and \(\theta=-1\) produce exactly the same distribution for every sample size. Thus the sign of \(\theta\) is not identifiable.
For one observation, the log density, apart from a constant, is \(-\tfrac12(x-\theta^2)^2\). Differentiating gives the score \(U_\theta(x)=2\theta(x-\theta^2)\), and therefore \[ I(\theta)=E_\theta[4\theta^2(X-\theta^2)^2]=4\theta^2. \] At \(\theta=0\), the information is zero. A putative scalar bound involving \(1/[nI(0)]\) is not finite; this is a warning that the regular information calculation cannot supply a usable bound at that point.
More strongly, no estimator can consistently estimate the signed parameter at both \(1\) and \(-1\). If \(T_n\) were consistent at both, then under the same sample distribution it would have to satisfy both \(T_n\xrightarrow{p}1\) and \(T_n\xrightarrow{p}-1\). For example, the disjoint events \(\{|T_n-1|<1/3\}\) and \(\{|T_n+1|<1/3\}\) would each have probability tending to one under that same distribution, which is impossible. The obstruction is identification, not merely an inefficient choice of estimator.
A singular information matrix in a multiparameter model can likewise signal that some parameter direction is not locally distinguished by the distribution. The matrix cannot then be inverted to produce the ordinary information bound for all components. Before interpreting a bound, check that the target is identifiable and that the information for the relevant direction is positive.
Worked Example: A Uniform Model with Changing Support
Worked Example: The Maximum from a Uniform Sample
Let \(X_1,\ldots,X_n\) be independent Uniform\((0,\theta)\) observations with \(\theta>0\), and write \(M_n=\max_i X_i\). The density is \(f_\theta(x)=1/\theta\) for \(0<x<\theta\) and zero otherwise. The support itself depends on \(\theta\).
For \(0\leq x\leq\theta\), \(P_\theta(M_n\leq x)=(x/\theta)^n\). Consequently, for fixed \(y\geq0\) and \(n\) large enough that \(y<n\theta\), \[ P_\theta\{n(\theta-M_n)>y\} =P_\theta\{M_n<\theta-y/n\} =\left(1-\frac{y}{n\theta}\right)^n \longrightarrow e^{-y/\theta}. \] Thus \(n(\theta-M_n)\xrightarrow{d}Y\), where \(Y\) is exponential with rate \(1/\theta\). The maximum approaches \(\theta\) at rate \(n\), not the usual \(\sqrt n\) rate.
This rate can be checked directly from moments. The density of \(M_n\) is \(n x^{n-1}/\theta^n\) on \((0,\theta)\), so \[ E_\theta[M_n]=\frac{n\theta}{n+1},\qquad \operatorname{Var}_\theta(M_n) =\frac{n\theta^2}{(n+1)^2(n+2)}. \] The maximum is biased downward, with bias \(-\theta/(n+1)\). The rescaled estimator \((n+1)M_n/n\) is unbiased for \(\theta\), and its variance is \[ \operatorname{Var}_\theta\left(\frac{n+1}{n}M_n\right) =\frac{\theta^2}{n(n+2)}. \]
A tempting but invalid calculation differentiates the log density inside the support, obtaining the score \(-1/\theta\), and calls its squared expectation \(1/\theta^2\). But this putative score has expectation \(-1/\theta\), not zero. The parameter-dependent endpoint contributes when differentiating the integral of the density; the usual score identity has failed. Applying the regular bound \(1/[nI(\theta)]=\theta^2/n\) would contradict the variance \(\theta^2/[n(n+2)]\) of the unbiased estimator just calculated. The contradiction diagnoses the invalid application: this is not a regular Fisher-information bound.
How to Diagnose a Failed Bound
The examples differ in their details, but each calls for the same audit before an information bound is used. A finite expression for a score or information does not substitute for checking the assumptions that connect it to estimator variance or a first-order limit.
Is the true value interior? A boundary can prevent the two-sided local perturbations used in regular asymptotic arguments.
Do different parameter values imply different distributions, at least locally for the target direction? If not, that component cannot be recovered from the data distribution.
Is the support common across parameter values, and does differentiation under the integral justify a zero-mean score? If not, the regular Cramér–Rao derivation may not apply.
Do not assume a root-\(n\) normal limit or compare variances as if it had been established. Derive the rate and limit appropriate to the model.
Common Mistake and Practical Significance
Wrong reasoning: “I differentiated the log density, calculated \(I(\theta)\), and got \(1/[nI(\theta)]\), so no estimator can have smaller variance.” This conclusion skips the assumptions that make the expression a bound. In the Uniform\((0,\theta)\) example, the proposed score does not even have mean zero; in the constrained normal model, the boundary MLE has a nonnormal limit; in the squared-mean model, the signed parameter is not identifiable.
Correction: Treat the information bound as conditional, not automatic. Verify the score and differentiation identities, parameter-space conditions, and identification before interpreting the result. When regularity fails, investigate the model directly: determine whether the parameter is estimable, derive the estimator's actual distribution or rate where possible, and state clearly which regular conclusion is unavailable.
This diagnosis matters in practice because a numerical “efficiency” comparison can be misleading if its benchmark was derived outside its domain of validity. A failed bound does not automatically mean that an estimator is better than a regular efficient estimator; it may mean that the estimator has bias, a nonnormal limit, a different convergence rate, or that the target itself cannot be identified.
Check Your Understanding
Use the model assumptions, not just the displayed information formula, to answer each question.
- Which two identities are used in the Cauchy–Schwarz derivation of the scalar Cramér–Rao inequality?
- Why does the constrained normal-mean MLE at \(\theta=0\) fail to have the usual normal first-order limit?
- In the model \(X\sim N(\theta^2,1)\), why can no estimator consistently recover the sign of \(\theta\) over both \(1\) and \(-1\)?
- What is the limiting distribution of \(n(\theta-M_n)\) in the Uniform\((0,\theta)\) example?
- What specific score property fails in the naive information calculation for the Uniform\((0,\theta)\) model?