Information Under Parameter Transformations

Why Information Changes With the Parameter

A statistical model can describe the same family of distributions using different parameters. A Bernoulli probability $p$, for example, can instead be represented by its log-odds $\psi=\log\{p/(1-p)\}$. The distributions have not changed, but the numerical scale used to describe them has. Fisher information must therefore be translated to the new scale before it is interpreted or compared.

For a scalar parameter, the translation depends on the derivative of the old parameter with respect to the new one. The square in the transformation rule is important: information is a second-moment measure of the score, and the score itself changes by a first-derivative factor. The result is local, so the derivative is evaluated at the parameter value under consideration.

Definition: Let $X$ have a density or mass function $f(x;\theta)$, where $\theta$ lies in an open interval. The score is $U_\theta(X)=\partial\log f(X;\theta)/\partial\theta$. When its second moment is finite and its mean is zero, the scalar Fisher information is $I_\theta=E_\theta[U_\theta(X)^2]$.

We use the regularity conditions that the model has a common support not depending on $\theta$, the log-density is differentiable in $\theta$, and the score has finite second moment and mean zero. These are standard sufficient conditions for the score-variance definition used here. The mean-zero condition is also what makes the second moment equal to the variance of the score. For a sample, $X$ denotes the full observed data, so the same definition applies to the sample information.

The Scalar Transformation Rule

Let $\phi=g(\theta)$ be a one-to-one reparameterization on the parameter interval, with a differentiable inverse $\theta=h(\phi)$ and $h'(\phi)\ne 0$. The transformed model has density $f(x;h(\phi))$. Its score is obtained by differentiating that log-density with respect to $\phi$.

Theorem: Under the regularity conditions above, the Fisher information for $\phi$ is $$ I_\phi(\phi)=I_\theta\{h(\phi)\}\left(\frac{d\theta}{d\phi}\right)^2. $$ Equivalently, if $\phi=g(\theta)$, then $$ I_\phi\{g(\theta)\}=\frac{I_\theta(\theta)}{\{g'(\theta)\}^2}. $$

Proof. Write $U_\theta(x)=\partial\log f(x;\theta)/\partial\theta$. By the chain rule, the score in the transformed parameter is

U_\phi(x) =\frac{\partial}{\partial\phi}\log f\{x;h(\phi)\} =U_\theta(x)\frac{d\theta}{d\phi}.

At a fixed value of $\phi$, the data distribution is the original model at $\theta=h(\phi)$. Thus the derivative factor is constant with respect to the data, and

I_\phi(\phi) =E_\phi[U_\phi(X)^2] =E_\theta\left[U_\theta(X)^2\left(\frac{d\theta}{d\phi}\right)^2\right] =I_\theta\{h(\phi)\}\left(\frac{d\theta}{d\phi}\right)^2.

Since $d\theta/d\phi=1/g'(\theta)$ for an invertible differentiable transformation with nonzero derivative, the equivalent expression follows. $\square$

The rule is a change-of-scale formula, not a claim that information is unchanged. If one unit of $\phi$ corresponds locally to a different number of units of $\theta$, the score per unit of $\phi$ changes by that conversion factor. Squaring it gives the corresponding change in information.

Worked Example: Information for a Bernoulli Probability

Let $X_1,\ldots,X_n$ be independent Bernoulli observations with success probability $p$, where $0<p<1$. For one observation,

\ell(p)=X\log p+(1-X)\log(1-p), \qquad U_p(X)=\frac{X}{p}-\frac{1-X}{1-p} =\frac{X-p}{p(1-p)}.

Because $\operatorname{Var}(X)=p(1-p)$ and $E[X]=p$, the score has mean zero and its information is

I_p^{(1)} =E[U_p(X)^2] =\frac{E[(X-p)^2]}{p^2(1-p)^2} =\frac{p(1-p)}{p^2(1-p)^2} =\frac{1}{p(1-p)}.

The independent observations have additive information, so $I_p=n/\{p(1-p)\}$. At $p=0.2$ with $n=10$, $p(1-p)=0.2(0.8)=0.16$, giving $I_p=10/0.16=62.5$. This is information measured on the probability scale.

Worked Example: Transforming Probability to Log-Odds

For the same Bernoulli sample, define the log-odds parameter $\psi=\log\{p/(1-p)\}$. Its inverse transformation is $p=e^\psi/(1+e^\psi)$. Differentiating this inverse gives

\frac{dp}{d\psi} =\frac{e^\psi}{(1+e^\psi)^2} =p(1-p).

Apply the information transformation rule, using $p$ as the original parameter and $\psi$ as the new one:

I_\psi =I_p\left(\frac{dp}{d\psi}\right)^2 =\frac{n}{p(1-p)}\{p(1-p)\}^2 =np(1-p).

At $n=10$ and $p=0.2$, this is $I_\psi=10(0.2)(0.8)=1.6$. The value $1.6$ does not contradict the probability-scale value $62.5$: the two numbers describe information per squared unit of different parameter scales. The inverse relationship also checks the calculation: $d\psi/dp=1/\{p(1-p)\}$, so $I_\psi\{d\psi/dp\}^2=1.6/(0.16)^2=62.5$.

Rates and Log-Rates

Rate parameters are often positive and can vary across several orders of magnitude. Representing a positive rate by its logarithm is therefore common. The transformation rule gives the information on the log-rate scale directly, without changing the probability model.

Worked Example: Information for a Poisson Rate

Suppose $X_1,\ldots,X_n$ are independent Poisson observations with rate $\lambda>0$. For one observation, the log-likelihood, score, and information are

\ell(\lambda)=X\log\lambda-\lambda-\log(X!), \qquad U_\lambda(X)=\frac{X}{\lambda}-1, \qquad I_\lambda^{(1)}=E[U_\lambda(X)^2]=\frac{\operatorname{Var}(X)}{\lambda^2}=\frac{1}{\lambda}.

Here $E[X]=\operatorname{Var}(X)=\lambda$, so the score has mean zero. By additivity for independent observations, $I_\lambda=n/\lambda$. For $n=8$ observations at $\lambda=2$, the information is $8/2=4$. This is information expressed per squared unit of the Poisson rate.

Worked Example: Transforming a Poisson Rate to a Log-Rate

Set $\xi=\log\lambda$, so $\lambda=e^\xi$ and $d\lambda/d\xi=\lambda$. The transformed information is

I_\xi =I_\lambda\left(\frac{d\lambda}{d\xi}\right)^2 =\frac{n}{\lambda}\lambda^2 =n\lambda.

For $n=8$ and $\lambda=2$, $I_\xi=8(2)=16$. Checking in the reverse direction, $d\xi/d\lambda=1/\lambda=1/2$, and $I_\xi(d\xi/d\lambda)^2=16(1/2)^2=4$, the original rate-scale information. The two calculations agree exactly.

Composition and Interpretation

Reparameterizations can be applied in stages. For example, a positive rate might first be expressed as a log-rate and then multiplied by a constant. The derivative factors multiply, so information can be transformed either in one step or successively.

Result: Suppose $\phi=g(\theta)$ and $\alpha=q(\phi)$ are differentiable one-to-one transformations with nonzero derivatives. Then $$ I_\alpha =I_\theta\left(\frac{d\theta}{d\phi}\right)^2 \left(\frac{d\phi}{d\alpha}\right)^2 =I_\theta\left(\frac{d\theta}{d\alpha}\right)^2. $$

Proof. Applying the transformation rule first from $\theta$ to $\phi$ gives $I_\phi=I_\theta(d\theta/d\phi)^2$. Applying it again from $\phi$ to $\alpha$ gives $I_\alpha=I_\phi(d\phi/d\alpha)^2$. Substitution yields the product of squared derivatives in the display. The ordinary chain rule gives $d\theta/d\alpha=(d\theta/d\phi)(d\phi/d\alpha)$, so that product is exactly $I_\theta(d\theta/d\alpha)^2$. $\square$

This composition property is useful as a consistency check. A direct transformation and a sequence of transformations must give the same information when they describe the same final parameter. It also explains why the derivative must be taken in the direction from the original parameter to the new one: the rule uses $d\theta/d\phi$, not $d\phi/d\theta$.

Common Mistake

A common error is to say that information is invariant because the probability model has not changed. The model is indeed the same, but the score is a derivative with respect to a particular parameter. Changing that parameter changes the size of the derivative, and therefore changes the numerical information.

For example, in the Bernoulli calculation one might incorrectly carry $I_p=n/\{p(1-p)\}$ over unchanged to the log-odds scale. The correction is to differentiate the inverse map: $dp/d\psi=p(1-p)$, then multiply by its square. The correct result is $I_\psi=np(1-p)$, not $n/\{p(1-p)\}$. Another frequent error is to use the reciprocal derivative in the formula; doing so reverses the scale conversion.

Key takeaway: If $\phi=g(\theta)$, transform scalar Fisher information using the derivative of the inverse map: $I_\phi=I_\theta(d\theta/d\phi)^2$. Evaluate both the information and derivative at the same underlying model value.

Check Your Understanding

Use the transformation rule and show the derivative factor in each calculation.

If $\phi=3\theta$, express $I_\phi$ in terms of $I_\theta$.
For a Bernoulli sample, why does the information for log-odds contain the factor $p(1-p)$, rather than its reciprocal?
For a Poisson sample with $n$ observations, what is the information for $\xi=\log\lambda$?
State one regularity condition used in the information transformation theorem and explain its role in the proof.
Suppose a parameter is transformed in two differentiable stages. Why must the successive information calculations agree with the direct transformation?