Transforming Fisher Information and Standard Errors

Why Parameterization Changes the Scale of Inference

A statistical model can describe the same unknown feature using different parameters. For example, a positive parameter \(\theta\) might be reported on a logarithmic scale as \(\eta=\log\theta\). The likelihood contains the same information either way, but its numerical curvature and the standard error attached to the parameter depend on the scale. Knowing exactly how they change is essential when interpreting likelihood-based output or comparing tests reported in different parameterizations.

This tutorial considers a scalar parameter and a smooth, one-to-one transformation. The central relationships are local: information changes by the square of the derivative of the inverse transformation, while standard errors change by the magnitude of the derivative of the forward transformation. Wald statistics are consequently equivalent to first order under the null, but need not match in a finite sample.

Definitions and Transformation Rules

Definition: Let \(\theta\) be a scalar parameter and let \(\eta=g(\theta)\), where \(g\) is one-to-one and continuously differentiable with \(g'(\theta)\ne 0\) in the parameter region under consideration. Write \(\theta=h(\eta)=g^{-1}(\eta)\). The Fisher information \(I_\theta(\theta)\) is the expected squared score for \(\theta\), and \(I_\eta(\eta)\) is the corresponding information when the same model is parameterized by \(\eta\). For a sample of size \(n\), these denote the information in the full sample.

Assume the usual regularity conditions for the score identities: the parameter is an interior point, the support does not depend on the parameter, the likelihood is differentiable, and differentiation can be interchanged with integration so that the score has mean zero and Fisher information equals the variance of the score. For independent observations, information in the full sample is the sum of the individual information contributions.

Theorem: Under these conditions, the information in the two parameterizations satisfies \(I_\eta(\eta)=I_\theta(h(\eta))[h'(\eta)]^2\), equivalently \(I_\eta(g(\theta))=I_\theta(\theta)/[g'(\theta)]^2\).

To prove this, let \(\ell_\theta(\theta)\) denote the log-likelihood in the original parameterization and let \(U_\theta(\theta)=\partial\ell_\theta(\theta)/\partial\theta\) be its score. The log-likelihood in the new parameterization is \(\ell_\eta(\eta)=\ell_\theta(h(\eta))\). By the chain rule,

U_\eta(\eta) = U_\theta(h(\eta))h'(\eta).

The derivative \(h'(\eta)\) is nonrandom, so taking the expected square of both sides gives

I_\eta(\eta) = E[U_\eta(\eta)^2] = [h'(\eta)]^2 E[U_\theta(h(\eta))^2] = [h'(\eta)]^2 I_\theta(h(\eta)).

Since \(h'(g(\theta))=1/g'(\theta)\), this is also the stated formula in terms of \(g\). The information changes because a unit change in \(\eta\) corresponds locally to a different-sized change in \(\theta\). The square makes the information positive regardless of whether \(g\) is increasing or decreasing. \(\square\)

Standard Errors on the New Scale

Under the usual asymptotic normality conditions for a regular maximum likelihood estimator, \(\hat\theta\) is approximately normal with variance \(1/I_\theta(\theta)\). If \(\hat\eta=g(\hat\theta)\), the first-order Taylor approximation gives the corresponding variance on the transformed scale. The information formula gives the same answer.

Result: If \(\hat\theta\) is consistent and asymptotically normal with variance \(1/I_\theta(\theta)\), and \(g\) is differentiable with \(g'(\theta)\ne 0\), then \(\hat\eta=g(\hat\theta)\) is asymptotically normal with variance \(g'(\theta)^2/I_\theta(\theta)=1/I_\eta(\eta)\). Correspondingly, plug-in information standard errors satisfy \(\widehat{\operatorname{SE}}(\hat\eta)\approx |g'(\hat\theta)|\widehat{\operatorname{SE}}(\hat\theta)\).

For the variance statement, expand \(g(\hat\theta)\) at the true value: \(g(\hat\theta)-g(\theta)=g'(\theta)(\hat\theta-\theta)+o_p(n^{-1/2})\). Thus the leading variance is \(g'(\theta)^2/I_\theta(\theta)\). On the other hand, the information transformation gives \(1/I_\eta(\eta)=g'(\theta)^2/I_\theta(\theta)\), so both routes agree. Replacing the unknown parameter by a consistent estimate yields the plug-in standard-error relationship.

Here \(I_\theta\) is the information in the full sample; in regular models it is typically proportional to \(n\), so these standard errors are of order \(n^{-1/2}\). The relationship is first-order asymptotic, rather than a claim that the exact finite-sample distribution of \(g(\hat\theta)\) is normal.

Worked Example: A Bernoulli Probability and Its Logit

Suppose \(X_1,\ldots,X_n\) are independent Bernoulli observations with success probability \(p\), and set \(\eta=\log\{p/(1-p)\}\). The score for \(p\) is \(\sum_i(X_i-p)/[p(1-p)]\), so the information is \(I_p(p)=n/[p(1-p)]\). Since \(dp/d\eta=p(1-p)\), the transformed information is

I_\eta(\eta) = I_p(p)\left(\frac{dp}{d\eta}\right)^2 = \frac{n}{p(1-p)}[p(1-p)]^2 = np(1-p).

For \(n=100\) and \(\hat p=0.25\), the plug-in standard error for \(\hat p\) is \(\sqrt{0.25(0.75)/100}=\sqrt{0.001875}\approx0.04330\). On the logit scale it is \(1/\sqrt{100(0.25)(0.75)}=1/\sqrt{18.75}\approx0.23094\). The derivative \(d\eta/dp=1/[p(1-p)]\) at \(0.25\) is \(5.3333\), and \(5.3333(0.04330)\approx0.23094\), as required.

For a finite-sample comparison, suppose 25 of the 100 observations are successes and test \(p_0=0.20\). On the probability scale the Wald statistic is \((0.25-0.20)/0.04330\approx1.1547\). On the logit scale, \(\eta_0=\log(0.20/0.80)\), and \(\hat\eta-\eta_0=\log(0.25/0.75)-\log(0.20/0.80)=\log(4/3)\approx0.28768\). Its Wald statistic is \(0.28768/0.23094\approx1.2457\). The statistics are not identical in this sample.

Worked Example: A Normal Mean and Its Logarithm

Let \(X_1,\ldots,X_{25}\) be independent \(N(\mu,4)\) observations, with \(\mu>0\). The variance is known, so \(I_\mu=25/4=6.25\). Choose \(\eta=\log\mu\), for which \(d\mu/d\eta=\mu\). Therefore \(I_\eta=6.25\mu^2\).

Suppose the observed sample mean is \(\hat\mu=4.2\). The original-scale standard error is \(2/\sqrt{25}=0.4\). The transformed-scale standard error from information is \(1/\sqrt{6.25(4.2)^2}=1/10.5\approx0.09524\). Equivalently, the derivative of \(\log\mu\) at \(4.2\) is \(1/4.2\), and \((1/4.2)(0.4)\approx0.09524\).

Testing \(\mu_0=4\), the original-scale Wald statistic is \((4.2-4)/0.4=0.5\). On the logarithmic scale it is \(\log(4.2)-\log(4)\), divided by \(0.09524\), which is approximately \(0.5123\). The difference illustrates the effect of the nonlinear transformation on the finite-sample Wald calculation.

Worked Example: A Cubic Reparameterization

Consider the same normal-mean model with known variance \(9\), now with \(n=36\) and \(\mu>0\). Let \(\eta=\mu^3\). The original-scale information is \(I_\mu=36/9=4\), and \(d\mu/d\eta=1/(3\mu^2)\). Hence

I_\eta(\eta)=\frac{4}{9\mu^4}, \qquad \operatorname{SE}(\hat\eta)\approx\frac{1}{\sqrt{I_\eta(\hat\eta)}}=\frac{3\hat\mu^2}{2}.

If \(\bar X=\hat\mu=2.2\), the original-scale standard error is \(3/\sqrt{36}=0.5\). The transformed-scale standard error is \(3(2.2)^2/2=7.26\). This also follows from the derivative \(d\eta/d\mu=3(2.2)^2=14.52\), since \(14.52(0.5)=7.26\).

For testing \(\mu_0=2\), the original-scale statistic is \((2.2-2)/0.5=0.4\). The transformed estimate and null value are \(2.2^3=10.648\) and \(2^3=8\), respectively, giving the transformed statistic \((10.648-8)/7.26\approx0.365\). Both use the same data and model, but the curved scale gives a different finite-sample Wald statistic.

Observed Information and Wald Statistics

The expected information transformation is exact under the regularity conditions above. Observed information—the negative second derivative of the log-likelihood for the data actually observed—has an additional term away from a likelihood maximum. With \(j_\theta(\theta)=-\ell_\theta''(\theta)\), direct differentiation gives

j_\eta(\eta) = j_\theta(h(\eta))[h'(\eta)]^2 - U_\theta(h(\eta))h''(\eta).

This follows by differentiating \(U_\eta=U_\theta h'\): its derivative is \(\ell_\theta''(h')^2+U_\theta h''\), and observed information is the negative of that derivative. At an interior maximum likelihood estimate, the score is zero, so the extra term vanishes. Thus observed information at the MLE transforms by the same squared inverse derivative. If using observed information away from the MLE, omitting the score term is generally incorrect.

A Wald statistic compares an estimate with a hypothesized value after dividing by its estimated standard error. For a null value \(\theta_0\), define \(\eta_0=g(\theta_0)\). If the MLE is consistent and asymptotically normal, and standard errors are obtained from information and evaluated consistently, the two-sided Wald statistics are asymptotically equivalent. To see this, Taylor expansion around \(\theta_0\) gives

\hat\eta-\eta_0 = g'(\theta_0)(\hat\theta-\theta_0)+o_p(n^{-1/2}).

The standard-error relationship and consistency give \(\widehat{\operatorname{SE}}(\hat\eta)=|g'(\theta_0)|\widehat{\operatorname{SE}}(\hat\theta)\{1+o_p(1)\}\). Since \(\hat\theta-\theta_0=O_p(n^{-1/2})\) and the standard error is also of order \(n^{-1/2}\), dividing the Taylor expansion by the transformed standard error yields

Z_\eta = \operatorname{sign}\{g'(\theta_0)\}Z_\theta+o_p(1).

Consequently \(Z_\eta^2-Z_\theta^2\xrightarrow{p}0\), and the two-sided tests have the same first-order asymptotic behavior under the null. This is not exact finite-sample invariance: a nonlinear transformation changes the numerator, and its derivative in the standard error is evaluated at the estimate. For a decreasing transformation, the signed statistic reverses direction, so the direction of a one-sided alternative must also be transformed.

Common Mistake and Practical Interpretation

Using the forward derivative for information. The correct formula is \(I_\eta=I_\theta(d\theta/d\eta)^2\), not \(I_\theta(d\eta/d\theta)^2\). Standard errors, in contrast, multiply by \(|d\eta/d\theta|\).
Assuming Wald statistics are exactly invariant. They are generally only first-order equivalent. The Bernoulli and normal examples show that finite-sample values can differ.
Forgetting the direction of a decreasing map. A negative derivative reverses a signed statistic and the interpretation of a one-sided test; squaring removes the sign only for a two-sided Wald statistic.
Applying the observed-information rule at any point. Away from the MLE, the score-dependent second-derivative term need not vanish.

These relationships help distinguish a change in the model from a change in the units used to describe its parameter. Fisher information and likelihood curvature can look numerically different across scales, while properly transformed estimates and standard errors describe the same local uncertainty. The finite-sample dependence of Wald procedures on parameterization is also a reason to state clearly which scale a reported test uses.

Check Your Understanding

Use the derivative rules and the examples above to answer the following questions.

If \(\eta=g(\theta)\), write the formula for \(I_\eta\) using \(I_\theta\) and \(d\theta/d\eta\).
If \(g'(\theta_0)<0\), how are the signed Wald statistics related to first order near the null?
Why does observed information at an interior MLE transform more simply than observed information at an arbitrary parameter value?
In the Bernoulli example, why do the probability-scale and logit-scale Wald statistics differ for the observed sample?
What additional care is needed when converting a one-sided test through a decreasing transformation?