Reparameterization And First-Order Efficiency

Why Parameter Coordinates Matter

An asymptotic variance bound describes precision in the coordinate used to express the parameter. If a parameter is measured in one unit and then expressed through a smooth transformation, its numerical variance generally changes. That change does not, by itself, mean that the estimator has become less efficient: the bound must be transformed too. The key distinction is between the numerical size of a variance, which depends on the coordinate, and efficiency relative to the appropriate bound, which is invariant under a smooth one-to-one reparameterization.

We use first-order asymptotic covariance to mean the covariance matrix \(V\) in a limit of the form \(\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N(0,V)\). When the regular parametric information bound applies, \(I(\theta)^{-1}\) is the corresponding bound for one observation, so the covariance scale for \(\hat{\theta}\) is \(I(\theta)^{-1}/n\). This is a statement about the limiting distribution; it does not, without additional moment conditions, assert convergence of the estimators’ finite-sample covariances.

Definition: Let \(\theta\) be an interior point of an open parameter space in \(\mathbb{R}^p\), and let \(\eta=h(\theta)\), where \(h\) is a one-to-one continuously differentiable transformation with continuously differentiable inverse and nonsingular Jacobian. Write \(J(\theta)=\partial h(\theta)/\partial\theta^T\). If \(\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N(0,V_\theta)\), then the first-order covariance for \(\hat{\eta}=h(\hat{\theta})\) is \(V_\eta=J(\theta)V_\theta J(\theta)^T\). An estimator is first-order efficient when its asymptotic covariance equals the applicable information bound in its parameterization.

The Covariance Bound Under a Transformation

The covariance transformation follows from the multivariate delta method. The essential point is that small estimation errors are multiplied, to first order, by the Jacobian. Here is the argument, including why the Taylor remainder disappears at the root-\(n\) scale.

Theorem: Suppose \(h\) is differentiable at \(\theta\), \(\hat{\theta}\xrightarrow{p}\theta\), and \(\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N(0,V_\theta)\). Then \(\sqrt{n}\{h(\hat{\theta})-h(\theta)\}\xrightarrow{d}N(0,J(\theta)V_\theta J(\theta)^T)\).

Proof. Differentiability at \(\theta\) gives the expansion \(h(\hat{\theta})-h(\theta)=J(\theta)(\hat{\theta}-\theta)+r(\hat{\theta})\), where \(\|r(\hat{\theta})\|/\|\hat{\theta}-\theta\|\xrightarrow{p}0\) whenever \(\hat{\theta}\ne\theta\); set the remainder to zero when they are equal. The assumed convergence in distribution implies that \(\sqrt{n}(\hat{\theta}-\theta)=O_p(1)\), and hence \(\hat{\theta}-\theta=O_p(n^{-1/2})\). It follows that \(\sqrt{n}r(\hat{\theta})=o_p(1)\). Therefore

\sqrt{n}\{h(\hat{\theta})-h(\theta)\} =J(\theta)\sqrt{n}(\hat{\theta}-\theta)+o_p(1).

The continuous mapping theorem gives a normal limit for the first term, with covariance \(J(\theta)V_\theta J(\theta)^T\); adding a term that converges to zero in probability does not change that limit. \(\square\)

The same Jacobian transforms the information bound. To see this, let \(s_\theta(X)=\partial\ell(\theta;X)/\partial\theta\) be the score for one observation, and let \(A(\eta)=\partial\theta/\partial\eta^T=J(\theta)^{-1}\). The chain rule gives \(s_\eta(X)=A(\eta)^Ts_\theta(X)\). Under the regularity conditions that define Fisher information and permit this score calculation, taking the expected outer product yields the information in the new coordinates.

Theorem: For a regular, finite-dimensional model with nonsingular Fisher information \(I_\theta(\theta)=E[s_\theta(X)s_\theta(X)^T]\), a smooth one-to-one reparameterization \(\eta=h(\theta)\) has \(I_\eta(\eta)=A(\eta)^T I_\theta(\theta)A(\eta)\). Consequently, \(I_\eta(\eta)^{-1}=J(\theta)I_\theta(\theta)^{-1}J(\theta)^T\).

Indeed, since \(A\) is nonrandom at a fixed parameter value,

I_\eta =E[s_\eta s_\eta^T] =A^TE[s_\theta s_\theta^T]A =A^TI_\theta A.

Because \(A=J^{-1}\), the inverse of \(A^TI_\theta A\) is \(J I_\theta^{-1}J^T\): multiplying the two matrices in either order gives the identity. Thus the bound transforms by exactly the same Jacobian rule as the estimator’s asymptotic covariance. \(\square\)

These calculations use a regular model and an interior parameter value where the information matrix is nonsingular. The equality of transformation rules is what supports the invariance claim; a singular Jacobian or a parameter value where regularity fails cannot simply be handled by applying these formulas mechanically.

Efficiency Is Invariant; Variance Is Not

Suppose an estimator in \(\theta\)-coordinates has asymptotic covariance \(V_\theta\), and the applicable bound is \(B_\theta=I_\theta^{-1}\). If \(V_\theta-B_\theta\) is positive semidefinite, then its transformed difference is

V_\eta-B_\eta =J(\theta)(V_\theta-B_\theta)J(\theta)^T.

For any vector \(a\), the quadratic form of the right-hand side is \((J^Ta)^T(V_\theta-B_\theta)(J^Ta)\), which is nonnegative. Thus the ordering of covariance and bound is preserved. If \(J\) is nonsingular, the reverse implication also holds by applying the same argument with \(J^{-1}\). In particular, \(V_\theta=B_\theta\) if and only if \(V_\eta=B_\eta\): first-order efficiency is invariant under the stated transformation, although the entries of both matrices may change.

Worked Example: An Exponential Rate and Its Logarithm

Let \(X_1,\ldots,X_n\) be independent exponential observations with rate \(\lambda>0\), so the density is \(f(x;\lambda)=\lambda e^{-\lambda x}\) for \(x>0\). The log-likelihood is \(\ell(\lambda)=n\log\lambda-\lambda\sum_{i=1}^n x_i\), and the one-observation score is \(1/\lambda-X\). Since \(E[X]=1/\lambda\) and \(\operatorname{Var}(X)=1/\lambda^2\), its Fisher information is \(I_\lambda=1/\lambda^2\). The information bound for the first-order covariance coefficient of estimating \(\lambda\) is therefore \(\lambda^2\).

The maximum likelihood estimator is \(\hat{\lambda}=1/\bar{X}\). It is consistent, and the usual likelihood regularity conditions hold for \(\lambda>0\); its first-order covariance coefficient is \(\lambda^2\), so it attains the bound. Now use the coordinate \(\eta=\log\lambda\), whose Jacobian is \(d\eta/d\lambda=1/\lambda\). The transformed bound is

B_\eta =\left(\frac{1}{\lambda}\right)^2\lambda^2 =1.

At \(\lambda=4\), the rate-coordinate bound is \(4^2=16\), while the log-rate-coordinate bound is \((1/4)^2(16)=1\). The transformed estimator is \(\hat{\eta}=\log\hat{\lambda}\); by the delta method its first-order covariance coefficient is also \(1\). The two numerical bounds differ because the coordinates differ, while the estimator attains the bound in both.

Worked Example: The Bernoulli Probability and Logit

Let \(X_i\) be independent Bernoulli observations with success probability \(p\in(0,1)\). The score for one observation is \((X-p)/[p(1-p)]\), so the Fisher information is \(I_p=1/[p(1-p)]\). The bound for estimating \(p\) is \(p(1-p)\). The maximum likelihood estimator \(\hat p=\bar X\) has limiting covariance coefficient \(p(1-p)\), and thus attains that bound.

Set \(\eta=\log\{p/(1-p)\}\). Its derivative is \(d\eta/dp=1/[p(1-p)]\), so the logit-coordinate bound is

B_\eta =\left\{\frac{1}{p(1-p)}\right\}^2p(1-p) =\frac{1}{p(1-p)}.

At \(p=1/4\), \(p(1-p)=3/16\). The bound for \(p\) is \(3/16\), while the bound for its logit is \(16/3\). The transformed estimator \(\log\{\hat p/(1-\hat p)\}\) has first-order covariance coefficient \(16/3\), by the delta method, and is efficient at this interior parameter value. Although the estimator can be undefined in finite samples when \(\hat p\) equals 0 or 1, the probability of either event tends to zero for fixed \(p\in(0,1)\); this does not alter its first-order limit.

Worked Example: Changing the Scale of a Normal Mean

Suppose \(X_i\sim N(\mu,4)\) independently, with the variance known. The information for \(\mu\) from one observation is \(1/4\), so the bound for estimating \(\mu\) is \(4\). The sample mean \(\bar X\) has limiting covariance coefficient \(4\), attaining the bound.

Now express the parameter as \(\eta=3\mu+2\). The Jacobian is \(3\), and the bound becomes \(3^2(4)=36\). The estimator \(3\bar X+2\) has limiting covariance coefficient \(9(4)=36\), so it remains efficient. The addition of 2 changes the location of the parameter but not its variance; the multiplication by 3 scales its variance by \(3^2\). Comparing the numerical variance 4 in one coordinate with 36 in the other, without transforming the bound, would mistake a change of scale for a change in efficiency.

When the Invariance Argument Does Not Apply

The result is about smooth, locally invertible changes of coordinates and first-order behavior at regular parameter values. Several tempting extensions require care.

A numerical variance is not an efficiency measure on its own. A change of units or parameter scale changes the variance. Compare an estimator with the bound expressed in the same coordinate.
One-to-one matters. A transformation that loses information about the original coordinate, or has a singular Jacobian, does not meet the assumptions used above. The inverse-Jacobian argument for equivalence of the covariance ordering then fails.
Smoothness matters at the parameter value. The delta-method derivation uses a first-order Taylor expansion. At a boundary or a point where the transformation is not differentiable, the limiting behavior may not be normal and the usual bound transformation need not apply.
First-order efficiency is not an assertion about every finite sample. The transformations above compare limiting covariance coefficients and information bounds. They do not say the transformed estimator has the smallest finite-sample variance among all estimators.

The practical procedure is to specify the new coordinate, compute its Jacobian, transform both the estimator’s asymptotic covariance and the bound, and then compare the two. For a smooth one-to-one reparameterization under regularity, the equality or inequality that determines first-order efficiency is preserved. The variance’s numerical value is coordinate-dependent; the efficiency conclusion is not.

Check Your Understanding

Work through these questions using the transformation rule and its conditions.

If an estimator of \(\theta\) has first-order covariance \(V_\theta\), what is the first-order covariance of \(h(\hat{\theta})\) when \(h\) is differentiable at \(\theta\)?
For \(\eta=\log\lambda\), why is the exponential-rate information bound transformed by multiplying by \(1/\lambda^2\)?
In the Bernoulli example at \(p=1/4\), calculate the logit-coordinate bound from the probability-coordinate bound and the derivative of the logit.
Explain why an efficient estimator remains efficient under a smooth one-to-one reparameterization, even when its numerical variance changes.
Name two conditions in this tutorial that can fail and prevent the standard invariance argument from applying directly.

Reparameterization And First-Order Efficiency

What You'll Learn