Information Under Smooth Reparameterization

Why Information Depends on Parameter Coordinates

A statistical model does not change when its parameter is given a different coordinate system, but the numerical information matrix does. This is not a contradiction: the coordinates describe changes in the model in different units and directions. For instance, a change of one unit in a probability parameter is not the same change as one unit in its logit. The Jacobian converts between these changes.

The key rule is that Fisher information transforms by multiplication on both sides by the Jacobian of the original parameter with respect to the new one. The direction of this derivative matters. We derive the rule, identify the assumptions needed for its expected-Hessian form, and work through scalar and vector calculations.

Definition and Transformation Rule

Definition: For a model with density or mass function $f(x;\theta)$, where $\theta$ is a $p$-dimensional parameter, the score is $s_\theta(x)=\nabla_\theta\log f(x;\theta)$. When the score has finite second moments, the Fisher information is $$ I_\theta(\theta)=E_\theta[s_\theta(X)s_\theta(X)^T]. $$

Suppose $\theta=h(\eta)$, where $h$ is a one-to-one, twice continuously differentiable map between open parameter sets, with a differentiable inverse and nonsingular Jacobian. Write $J(\eta)$ for the $p\times p$ matrix with entries $J_{ai}=\partial\theta_a/\partial\eta_i$. The data density in the new coordinates is $f(x;h(\eta))$; this is a change of parameter, not a transformation of the data.

Theorem: Suppose the score has finite second moments and the model is differentiable in the parameter. Then the Fisher information in the $\eta$ coordinates is $$ I_\eta(\eta)=J(\eta)^T I_\theta(h(\eta))J(\eta). $$ For the expected-negative-Hessian definition of information to give the same result, also assume the relevant second derivatives can be passed under the integral defining the density, so that $E_\theta[s_\theta(X)]=0$ and $I_\theta(\theta)=-E_\theta[\nabla_\theta^2\log f(X;\theta)]$.

Proof. Let $\ell_\theta(x)=\log f(x;\theta)$. By the chain rule, the score in the new coordinates is $$ s_\eta(x)=\nabla_\eta\ell_\theta(x)=J^T s_\theta(x). $$ Taking the expected outer product gives $$ I_\eta=E_\eta[s_\eta s_\eta^T] =E_\theta[J^T s_\theta s_\theta^T J] =J^T I_\theta J, $$ because $J$ is fixed at the parameter value and the model for the data is unchanged. This proves the score-covariance formula.

For the expected-Hessian statement, apply the chain rule a second time: $$ \nabla_\eta^2\ell_\theta =J^T(\nabla_\theta^2\ell_\theta)J +\sum_{a=1}^p (s_\theta)_a\,\nabla_\eta^2 h_a. $$ Taking expectations, the second term vanishes because $E_\theta[s_\theta]=0$. Consequently, $$ -E_\eta[\nabla_\eta^2\ell_\theta] =J^T\{-E_\theta[\nabla_\theta^2\ell_\theta]\}J =J^T I_\theta J. $$ The differentiation conditions in the theorem justify the zero-mean score and expected-Hessian identities used here. $\square$

For $n$ independent, identically distributed observations, the log-likelihood is a sum of $n$ individual log-likelihoods and the information is $n$ times the one-observation information. The same transformation rule applies with that sample information.

Scalar Reparameterizations

For a scalar parameter $\theta=h(\eta)$, the Jacobian is the derivative $h'(\eta)$, so the rule reduces to $$ I_\eta(\eta)=\left(\frac{d\theta}{d\eta}\right)^2 I_\theta(\theta). $$ The derivative is squared: information is nonnegative, and reversing the direction of a coordinate does not make it negative. The derivative must still be nonzero for a smooth one-to-one coordinate change with a differentiable inverse.

Worked Example: Bernoulli Probability and Logit

Let $X\sim\operatorname{Bernoulli}(p)$, with $0<p<1$. The log-likelihood for one observation is $x\log p+(1-x)\log(1-p)$, and its score is $$ s_p(x)=\frac{x}{p}-\frac{1-x}{1-p}=\frac{x-p}{p(1-p)}. $$ Since $\operatorname{Var}(X)=p(1-p)$, the information is $$ I_p(p)=E[s_p(X)^2]=\frac{\operatorname{Var}(X)}{p^2(1-p)^2} =\frac{1}{p(1-p)}. $$

Use the logit coordinate $\eta=\log\{p/(1-p)\}$. Its inverse is $p=h(\eta)=e^\eta/(1+e^\eta)$, and differentiation gives $dp/d\eta=p(1-p)$. Therefore $$ I_\eta(\eta)=\{p(1-p)\}^2\frac{1}{p(1-p)}=p(1-p). $$ At $p=0.2$, the probability-coordinate information is $1/(0.2\cdot0.8)=6.25$, while the logit-coordinate information is $0.2\cdot0.8=0.16$. These numbers differ because they measure information per squared unit in different coordinates, not because the underlying Bernoulli model changed.

Worked Example: A Nonlinear Coordinate for a Normal Mean

Suppose $X\sim N(\mu,\sigma^2)$, where $\sigma^2$ is known and $\mu>0$. The score for $\mu$ is $(X-\mu)/\sigma^2$, so $$ I_\mu(\mu)=E\left[\frac{(X-\mu)^2}{\sigma^4}\right]=\frac{1}{\sigma^2}. $$ Set $\mu=\eta^2$ with $\eta>0$. This is a smooth one-to-one parameterization on the stated domain, and $d\mu/d\eta=2\eta$. Hence $$ I_\eta(\eta)=(2\eta)^2\frac{1}{\sigma^2}=\frac{4\eta^2}{\sigma^2}. $$ For example, if $\eta=2$ and $\sigma^2=4$, then $\mu=4$, $I_\mu=1/4$, and $I_\eta=16/4=4$. Directly, $I_\eta=(d\mu/d\eta)^2 I_\mu=4^2(1/4)=4$, confirming the result.

Vector Reparameterizations

For vector parameters, the scalar derivative becomes a Jacobian matrix. If $\theta=h(\eta)$, the columns of $J$ describe how each new coordinate changes the original parameter vector. Multiplication by $J$ on the right and $J^T$ on the left transforms information into the new coordinate system. This also preserves positive definiteness when $J$ is nonsingular: for any nonzero vector $v$, $v^T I_\eta v=(Jv)^T I_\theta(Jv)>0$ whenever $I_\theta$ is positive definite.

Worked Example: Linear Coordinates for a Bivariate Normal Mean

Let $X_1,\ldots,X_n$ be independent $N_2(\theta,\Sigma)$ observations, where the covariance matrix is known and $$ \Sigma=\begin{pmatrix}2&1\\1&2\end{pmatrix}, \qquad \Sigma^{-1}=\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}. $$ For one observation the score for $\theta$ is $\Sigma^{-1}(X-\theta)$. Its covariance is $$ \Sigma^{-1}\operatorname{Cov}(X)\Sigma^{-1} =\Sigma^{-1}\Sigma\Sigma^{-1} =\Sigma^{-1}, $$ so the information for $n$ observations in the $\theta$ coordinates is $n\Sigma^{-1}$.

Now define $\theta=A\eta$, where $$ A=\begin{pmatrix}1&0\\-1&1\end{pmatrix}. $$ Since $\det(A)=1$, this is an invertible linear reparameterization and its Jacobian is $J=A$. First calculate $$ \Sigma^{-1}A =\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix} \begin{pmatrix}1&0\\-1&1\end{pmatrix} =\begin{pmatrix}1&-1/3\\-1&2/3\end{pmatrix}. $$ Thus $$ I_\eta =nA^T\Sigma^{-1}A =n\begin{pmatrix}1&-1\\0&1\end{pmatrix} \begin{pmatrix}1&-1/3\\-1&2/3\end{pmatrix} =n\begin{pmatrix}2&-1\\-1&2/3\end{pmatrix}. $$ The off-diagonal entries are $-1$, not zero. For a check, the determinant of the per-observation matrix is $2(2/3)-(-1)^2=1/3>0$, and its top-left entry is positive, so it is positive definite as expected from the invertible transformation.

Worked Example: Log Rates for Independent Poisson Counts

Suppose $X_1$ and $X_2$ are independent Poisson variables with respective rates $\lambda_1>0$ and $\lambda_2>0$. The score for $\lambda_j$ is $X_j/\lambda_j-1$, whose variance is $\operatorname{Var}(X_j)/\lambda_j^2=1/\lambda_j$. Independence makes the cross-information zero, so $$ I_\lambda=\begin{pmatrix}1/\lambda_1&0\\0&1/\lambda_2\end{pmatrix}. $$ Use log-rate coordinates $\eta_j=\log\lambda_j$, so $\lambda_j=e^{\eta_j}$ and $J=\partial\lambda/\partial\eta=\operatorname{diag}(\lambda_1,\lambda_2)$. The transformed information is $$ I_\eta=J^T I_\lambda J =\begin{pmatrix}\lambda_1&0\\0&\lambda_2\end{pmatrix}. $$ At rates $(\lambda_1,\lambda_2)=(2,5)$, this is $\operatorname{diag}(2,5)$. For $n$ independent pairs of counts, it becomes $n\operatorname{diag}(2,5)$.

Common Mistakes and Interpretation

A frequent error is to use the Jacobian in the opposite direction. If the new parameter is $\eta=g(\theta)$, the transformation formula still requires $d\theta/d\eta$, not $d\eta/d\theta$. One can either differentiate the inverse map or rewrite the rule using the inverse Jacobian. For a scalar parameter, confusing these derivatives changes the answer by a reciprocal square.

A second error is to apply the expected-information rule directly to the observed negative Hessian. For a nonlinear transformation, the Hessian chain rule contains the additional term $\sum_a(s_\theta)_a\nabla_\eta^2h_a$. It vanishes after expectation under the stated regularity conditions and the zero-mean score identity, but it need not vanish for a particular observed data set. Thus the observed Hessian generally does not transform by $J^T(\cdot)J$ alone.

Key takeaway: Under a smooth, invertible parameter change $\theta=h(\eta)$, transform expected Fisher information as $I_\eta=J^T I_\theta J$, with $J=\partial\theta/\partial\eta$. Check the direction and dimensions of $J$, and distinguish expected information from the observed Hessian.

This rule matters whenever a model is expressed in coordinates chosen for interpretation or computation—for example, probabilities versus logits, or rates versus log rates. It also makes clear that an information matrix is coordinate-dependent: its entries change with the scale of the parameter, while the underlying statistical model remains the same.

Check Your Understanding

Use the Jacobian rule and the regularity qualifications in your reasoning.

If $\theta=3\eta$ and $I_\theta=4$, what is $I_\eta$?
For a vector transformation $\theta=h(\eta)$, what does the Jacobian entry $J_{ai}$ represent?
Why does the extra Hessian term in the second-derivative chain rule disappear from expected information under the stated conditions?
For $\eta=\log\lambda$, should the scalar information transformation use $d\lambda/d\eta$ or $d\eta/d\lambda$? Explain.
Why can the observed negative Hessian fail to transform by the Jacobian formula alone for a nonlinear reparameterization?