Asymptotic Variance of a Transformed Maximum Likelihood Estimator

Why Reparameterization Changes the Variance

An estimator’s uncertainty depends on the scale on which it is reported. A small change in a probability is not numerically the same as the corresponding change in its log-odds; likewise, a change in a variance is not the same size as a change in its reciprocal. For a smooth transformation, the scalar delta method gives a precise first-order conversion. For a maximum likelihood estimator, the same conversion can also be read directly from how Fisher information changes under reparameterization.

The key point is to keep track of which derivative is being used. If the new parameter is $\eta=g(\theta)$, the transformed estimator’s variance uses $g'(\theta)$. The Fisher information in the new parameter uses the derivative of the inverse map, $d\theta/d\eta$. These are reciprocals, and the two calculations agree.

Definition: Let $X_1,\ldots,X_n$ be independent and identically distributed from a regular scalar model with density or mass function $f(x;\theta)$, where $\theta$ belongs to an open interval. The Fisher information per observation is $$ I(\theta)=E_\theta\left[\left(\frac{\partial}{\partial\theta}\log f(X;\theta)\right)^2\right]. $$ Suppose $I(\theta)$ is finite and positive, and the MLE satisfies $$ \sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N\left(0,\frac{1}{I(\theta)}\right). $$ For a one-to-one differentiable map $\eta=g(\theta)$, with differentiable inverse and $g'(\theta)\ne 0$ at the true value, the transformed parameter is $\eta$, and the corresponding transformed estimator is $g(\hat{\theta})$.

The stated MLE limit is the usual first-order asymptotic normality result under regularity conditions: in particular, the model is differentiable in the parameter, the support does not change with the parameter, the score has mean zero and finite positive variance, and the MLE is consistent and asymptotically normal. The regularity audit for MLE asymptotics gives a broader account of such conditions. Here they are assumed to hold on a neighborhood of the true parameter.

Two Ways to Obtain the Transformed Variance

First consider the estimator. The scalar delta method says that if $\sqrt n(\hat{\theta}-\theta)\xrightarrow{d}N(0,V)$ and $g$ is differentiable at $\theta$, then $\sqrt n\{g(\hat{\theta})-g(\theta)\}\xrightarrow{d}N(0,[g'(\theta)]^2V)$. Taking $V=1/I(\theta)$ gives the transformed MLE’s asymptotic variance on the $\sqrt n$ scale. Thus the usual first-order approximation to the variance of the estimator itself has an additional factor $1/n$.

Theorem: Under the conditions above, the transformed MLE satisfies $$ \sqrt{n}\{g(\hat{\theta})-g(\theta)\} \xrightarrow{d} N\left(0,\frac{[g'(\theta)]^2}{I(\theta)}\right). $$ Accordingly, its first-order asymptotic variance is $$ \frac{[g'(\theta)]^2}{nI(\theta)}. $$ This is a large-sample variance approximation; it need not equal the exact finite-sample variance of $g(\hat{\theta})$.

To prove the result, apply the scalar delta method to the assumed limit for $\hat{\theta}$, using $g'(\theta)$ as the derivative. Multiplication of a centered normal limit by $g'(\theta)$ multiplies its variance by $[g'(\theta)]^2$. The factor $1/n$ in the estimator-scale approximation follows because the theorem describes $\sqrt n\{g(\hat{\theta})-g(\theta)\}$. $\square$

Now derive the information transformation. Write $\theta=h(\eta)=g^{-1}(\eta)$. For one observation, the reparameterized density is $f(x;h(\eta))$. Differentiating its log density with respect to $\eta$ gives its score:

\frac{\partial}{\partial\eta}\log f(X;h(\eta)) = \frac{\partial}{\partial\theta}\log f(X;\theta) \frac{d\theta}{d\eta}.

This is the chain rule for the score. Squaring it and taking expectation under the same data distribution gives the information transformation.

Theorem: If the score is square-integrable and the inverse parameter map is differentiable, then the Fisher information per observation in the $\eta$ parameterization is $$ I_\eta(\eta)=I_\theta(\theta)\left(\frac{d\theta}{d\eta}\right)^2 =\frac{I_\theta(\theta)}{[g'(\theta)]^2}. $$

Indeed, the squared score in the new parameter is the squared score in the original parameter multiplied by $(d\theta/d\eta)^2$, which is constant with respect to the observation. Taking its expectation yields $I_\eta(\eta)=I_\theta(\theta)(d\theta/d\eta)^2$. Since $d\theta/d\eta=1/g'(\theta)$, the second equality follows. $\square$

The reciprocal of the transformed information is therefore $1/I_\eta(\eta)=[g'(\theta)]^2/I_\theta(\theta)$, exactly the variance in the transformed MLE’s $\sqrt n$-limit. This agreement is a useful check on both calculations.

Worked Examples

Worked Example: Log-Odds of a Bernoulli Probability

Let $X_1,\ldots,X_n$ be Bernoulli observations with success probability $p$, where $0<p<1$. Transform the parameter to the log-odds $\eta=g(p)=\log\{p/(1-p)\}$, and find the first-order asymptotic variance of the transformed MLE at $p=0.4$ for $n=100$.

For one observation, the score for $p$ is $X/p-(1-X)/(1-p)$. Its Fisher information is $I_p(p)=1/\{p(1-p)\}$. The derivative of the log-odds map is $g'(p)=1/\{p(1-p)\}$. Consequently,

\operatorname{avar}(\hat{\eta}) = \frac{[g'(p)]^2}{nI_p(p)} = \frac{1}{np(1-p)}.

At $p=0.4$, $p(1-p)=0.4(0.6)=0.24$, so the approximation is $1/(100\cdot0.24)=1/24\approx0.04167$. The corresponding first-order standard deviation is approximately $\sqrt{1/24}\approx0.2041$.

The same calculation in the $\eta$ parameter gives $dp/d\eta=p(1-p)$, hence $I_\eta=I_p\{dp/d\eta\}^2=p(1-p)$. Its reciprocal divided by $n$ is again $1/\{np(1-p)\}$. For a finite Bernoulli sample consisting entirely of successes or entirely of failures, the log-odds MLE is not finite. At any fixed $0<p<1$, the probability of those events is $p^n+(1-p)^n$, which tends to zero, so they do not change this interior first-order limit.

Worked Example: Exponentiating a Normal Mean

Suppose $X_1,\ldots,X_n$ are independent $N(\mu,\sigma^2)$ observations, with known $\sigma^2$ and unrestricted $\mu$. Let $\eta=e^\mu$. Find the transformed MLE’s asymptotic variance when $\mu=\log 2$, $\sigma^2=9$, and $n=100$.

The MLE of $\mu$ is $\hat{\mu}=\bar X$, and the Fisher information per observation is $I_\mu=1/\sigma^2$. Since $g'(\mu)=e^\mu$, the transformed estimator is $\hat{\eta}=e^{\bar X}$, and

\operatorname{avar}(\hat{\eta}) = \frac{e^{2\mu}}{nI_\mu} = \frac{e^{2\mu}\sigma^2}{n}.

At $\mu=\log 2$, $e^{2\mu}=e^{\log 4}=4$. Therefore the variance approximation is $4(9)/100=0.36$, and the corresponding first-order standard deviation is $0.6$. The target value is $\eta=e^\mu=2$. The variance is on the $\eta$ scale, not the $\mu$ scale.

Worked Example: Reparameterizing a Normal Variance as Precision

Let $X_1,\ldots,X_n$ be independent $N(0,v)$ observations, with known mean zero and unknown variance $v>0$. Use precision $\eta=1/v$ as the parameter. Find the first-order asymptotic variance of its MLE at $v=4$ for $n=50$.

The MLE of $v$ is $\hat v=n^{-1}\sum_{i=1}^n X_i^2$, so the transformed MLE is $\hat\eta=1/\hat v$. For one observation, the score for $v$ is $-1/(2v)+X^2/(2v^2)$, and the Fisher information is $I_v=1/(2v^2)$. The derivative of $g(v)=1/v$ is $-1/v^2$. Thus

\operatorname{avar}(\hat{\eta}) = \frac{(-1/v^2)^2}{n/(2v^2)} = \frac{2}{nv^2}.

At $v=4$ and $n=50$, this is $2/(50\cdot16)=1/400=0.0025$. As a check, $dv/d\eta=-1/\eta^2=-v^2$, so $I_\eta=I_v(dv/d\eta)^2=v^2/2$. Then $1/(nI_\eta)=2/(nv^2)=0.0025$, in agreement with the delta-method calculation.

MLE Equivariance and a Common Pitfall

The transformation of the estimator is not an extra modeling assumption. If $\hat{\theta}$ maximizes the likelihood over the parameter space and $\eta=g(\theta)$ is one-to-one, then $g(\hat{\theta})$ maximizes the same likelihood expressed in terms of $\eta$. To see this, write the reparameterized likelihood as $L_\eta(\eta)=L_\theta(g^{-1}(\eta))$. Every admissible $\eta$ corresponds to exactly one admissible $\theta$, so maximizing over $\eta$ selects the image of a maximizer over $\theta$. This is the invariance of the MLE under one-to-one reparameterization. The statement presumes the maximizer exists; if there are multiple maximizers, the maximizing set is mapped to the maximizing set.

A common mistake is to use $d\theta/d\eta$ in the transformed estimator’s variance. That derivative belongs in the information calculation, because the new score differentiates with respect to $\eta$. For the estimator, the delta method differentiates the function being applied to $\hat{\theta}$, so the factor is $g'(\theta)$. Using the inverse derivative in the estimator calculation reverses the scale change.

Another mistake is to attach a Jacobian factor to the likelihood when changing the parameter. A parameter reparameterization does not transform the observed data; the likelihood is simply composed with the inverse map. Jacobians arise when transforming a random variable’s data scale, not when relabeling the model parameter.

Key takeaway: For $\eta=g(\theta)$, a regular scalar MLE has first-order transformed variance $[g'(\theta)]^2/\{nI_\theta(\theta)\}$. Equivalently, the information becomes $I_\eta=I_\theta/[g'(\theta)]^2$, so the inverse-information calculation gives the same result.

Check Your Understanding

Use the chain rule and the first-order asymptotic variance formula to answer these questions.

If $\eta=3\theta+1$, what factor multiplies the asymptotic variance of $\hat{\theta}$ when reporting $3\hat{\theta}+1$?
For $\eta=g(\theta)$, which derivative appears in the transformed score: $g'(\theta)$ or $d\theta/d\eta$? Explain why.
If $I_\theta(\theta)=5$ and $g'(\theta)=2$, what is $I_\eta(\eta)$, and what is the transformed MLE’s asymptotic variance for sample size $n$?
Why is the first-order asymptotic variance of a transformed estimator not necessarily its exact finite-sample variance?
Does changing from a parameter $\theta$ to a parameter $\eta=g(\theta)$ require a Jacobian factor in the likelihood for the observed data? Explain.

Asymptotic Variance of a Transformed Maximum Likelihood Estimator

What You'll Learn