Parameterization Effects on the Fisher Matrix

Why Parameterization Changes the Fisher Matrix

A population model may describe random-effect variability using variances, standard deviations, covariances, or correlations. These are different coordinates for the same underlying quantities. The likelihood describes the same statistical model under each coordinate system, but the numerical entries of its Fisher information matrix change when the coordinates change. That matters when reporting uncertainty, comparing parameter scales, or using the matrix in a design calculation.

The essential bookkeeping rule is to differentiate the old parameter vector with respect to the new one. If the Fisher matrix is known in coordinates $\theta$, and the desired coordinates are $\eta$, write $\theta=g(\eta)$ and use the Jacobian $J=\partial\theta/\partial\eta$. Reversing this Jacobian is a common source of incorrect results.

Definition: Let $\theta$ be a parameter vector and let $\eta$ be a smooth, one-to-one reparameterization with $\theta=g(\eta)$. The Jacobian $J$ has entries $J_{ij}=\partial\theta_i/\partial\eta_j$. If $I_\theta$ is the Fisher information matrix in $\theta$ coordinates, then the Fisher information in $\eta$ coordinates is $$ I_\eta=J^\mathsf{T} I_\theta J. $$

This is a congruence transformation, not a similarity transformation. In particular, it does not generally preserve the numerical entries, eigenvalues, or determinant of the matrix. The result is information in the new coordinates, including their units and scale.

Justification of the Transformation

Let $\ell(\theta)$ be the log likelihood and $U_\theta=\partial\ell/\partial\theta$ its score vector. The chain rule gives the score in the new coordinates:

U_\eta=\frac{\partial\ell}{\partial\eta} =J^\mathsf{T}U_\theta.

The Fisher information can be defined, under the usual regularity conditions, as the expected outer product of the score. Substituting the transformed score and treating the parameter transformation as fixed gives

I_\eta =E(U_\eta U_\eta^\mathsf{T}) =J^\mathsf{T}E(U_\theta U_\theta^\mathsf{T})J =J^\mathsf{T}I_\theta J.

The relevant conditions include differentiability of the likelihood and transformation, a locally one-to-one parameterization, and regularity sufficient for the score-based Fisher information to exist. The parameter point must also be in the interior of the admissible parameter space for the usual regular interpretation to apply.

There is a related caution when information is calculated from the observed Hessian in a particular data set. The Hessian chain rule has an additional term involving the score and second derivatives of the transformation. Thus, the observed information does not in general transform by the congruence formula at every data set and every parameter value. For expected Fisher information, the expected score is zero under regularity, and the additional term vanishes in expectation.

Variance, Standard Deviation, and Log-Variance

For a positive variance $v$, let $\tau=\sqrt{v}$ be its standard deviation and let $\lambda=\log v$ be its log-variance. Their inverse mappings from new coordinates to variance are $v=\tau^2$ and $v=\exp(\lambda)$. The derivatives needed for the Fisher transformation are therefore $dv/d\tau=2\tau$ and $dv/d\lambda=v$.

I_\tau=(2\tau)^2 I_v=4vI_v, \qquad I_\lambda=v^2I_v.

These scalar results are special cases of the matrix rule. The numerical values have different units: information for a variance is measured per squared variance unit, whereas information for a standard deviation is measured per squared standard-deviation unit. Log-variance is dimensionless when the logarithm is defined relative to a fixed reference unit. Consequently, a larger information number in one parameterization does not, by itself, mean that the experiment is more informative.

Worked Example: Transforming Information for One Variance

Suppose a hypothetical model has a random-effect variance $v=9$ in squared units, so its standard deviation is $\tau=3$ units. Assume the Fisher information in variance coordinates is $I_v=2$, in the corresponding inverse-squared-variance units. Find the information for $\tau$ and $\lambda=\log v$.

For standard-deviation coordinates, $dv/d\tau=2\tau=6$. Hence

I_\tau =\left(\frac{dv}{d\tau}\right)^2 I_v =6^2(2) =72.

For log-variance coordinates, $dv/d\lambda=v=9$. Thus

I_\lambda =\left(\frac{dv}{d\lambda}\right)^2 I_v =9^2(2) =162.

The values 2, 72, and 162 represent information in three different coordinate scales. They are not competing estimates of one scale-free quantity.

Covariance and Correlation Coordinates

For two random effects, write their covariance matrix as

\Omega= \begin{pmatrix} v_1 & c_{12}\\ c_{12} & v_2 \end{pmatrix}, \qquad c_{12}=\rho\sqrt{v_1v_2}.

Here $v_1,v_2$ are variances, $c_{12}$ is a covariance, and $\rho$ is a correlation. A covariance and a correlation are not interchangeable coordinates: the covariance depends on the scales of both random effects, while the correlation is dimensionless and constrained to $(-1,1)$ for a positive-definite two-dimensional covariance matrix.

If the old coordinates are $(v_1,v_2,c_{12})$ and the new coordinates are $(\tau_1,\tau_2,\rho)$, then $v_1=\tau_1^2$, $v_2=\tau_2^2$, and $c_{12}=\rho\tau_1\tau_2$. The Jacobian must differentiate this complete mapping:

J= \frac{\partial(v_1,v_2,c_{12})} {\partial(\tau_1,\tau_2,\rho)} = \begin{pmatrix} 2\tau_1 & 0 & 0\\ 0 & 2\tau_2 & 0\\ \rho\tau_2 & \rho\tau_1 & \tau_1\tau_2 \end{pmatrix}.

By contrast, if the old coordinates are already $(v_1,v_2,\rho)$, and the new coordinates are $(\tau_1,\tau_2,\rho)$, the mapping is simply $v_i=\tau_i^2$, with $\rho$ unchanged. In that case the Jacobian is $\operatorname{diag}(2\tau_1,2\tau_2,1)$. These are distinct transformations: the former changes covariance coordinates to standard-deviation-and-correlation coordinates; the latter changes variances to standard deviations while retaining correlation coordinates.

Worked Example: From Covariance to Standard-Deviation and Correlation Coordinates

Consider a hypothetical random-effect covariance matrix with $\tau_1=2$ units, $\tau_2=3$ units, and $\rho=0.5$. Then $v_1=4$, $v_2=9$, and $c_{12}=0.5(2)(3)=3$. Suppose the Fisher matrix in old coordinates $(v_1,v_2,c_{12})$ is diagonal with entries $2,1,0.5$. For clarity, these are hypothetical coordinate-specific information values.

The Jacobian from the old coordinates to the new coordinates, evaluated at these values, is

J= \begin{pmatrix} 4 & 0 & 0\\ 0 & 6 & 0\\ 1.5 & 1 & 6 \end{pmatrix}.

Applying $I_\eta=J^\mathsf{T}I_\theta J$, with $I_\theta=\operatorname{diag}(2,1,0.5)$, gives

I_\eta= \begin{pmatrix} 33.125 & 0.75 & 4.5\\ 0.75 & 36.5 & 3\\ 4.5 & 3 & 18 \end{pmatrix}.

For example, the first diagonal element is $2(4^2)+0.5(1.5^2)=32+1.125=33.125$. The first off-diagonal element is $0.5(1.5)(1)=0.75$. The off-diagonal entries arise because a change in either standard deviation also changes the covariance when correlation is held fixed.

Log-Variance Coordinates While Retaining Correlation

A different useful change keeps correlation as a coordinate but replaces variances by log-variances. Let $\lambda_i=\log v_i$, so $v_i=\exp(\lambda_i)$. Starting from old coordinates $(v_1,v_2,\rho)$ and using new coordinates $(\lambda_1,\lambda_2,\rho)$, the Jacobian is diagonal:

J= \frac{\partial(v_1,v_2,\rho)} {\partial(\lambda_1,\lambda_2,\rho)} = \begin{pmatrix} v_1 & 0 & 0\\ 0 & v_2 & 0\\ 0 & 0 & 1 \end{pmatrix}.

This differs from the covariance-to-correlation Jacobian. Here the old parameter vector contains $\rho$, not $c_{12}$, so no covariance derivative belongs in this Jacobian. If the old coordinates instead contain covariance, then $c_{12}=\rho\exp((\lambda_1+\lambda_2)/2)$, and its row of the Jacobian is $(c_{12}/2,c_{12}/2,\sqrt{v_1v_2})$.

Worked Example: Variances to Log-Variances With Correlation Fixed as a Coordinate

Suppose the old coordinates are $(v_1,v_2,\rho)$, with $v_1=4$ and $v_2=9$. Assume a hypothetical old-coordinate Fisher matrix $I_\theta=\operatorname{diag}(2,1,5)$. For new coordinates $(\lambda_1,\lambda_2,\rho)$, the Jacobian is $\operatorname{diag}(4,9,1)$. Therefore,

I_\eta = \begin{pmatrix} 4&0&0\\ 0&9&0\\ 0&0&1 \end{pmatrix} \begin{pmatrix} 2&0&0\\ 0&1&0\\ 0&0&5 \end{pmatrix} \begin{pmatrix} 4&0&0\\ 0&9&0\\ 0&0&1 \end{pmatrix} = \begin{pmatrix} 32&0&0\\ 0&81&0\\ 0&0&5 \end{pmatrix}.

The variance information for $v_1$ is multiplied by $v_1^2=16$, giving 32; that for $v_2$ is multiplied by $v_2^2=81$, giving 81. The correlation coordinate is unchanged, so its diagonal information remains 5.

Checks, Comparisons, and Limitations

A reliable transformation starts by writing both parameter vectors in order and expressing every old coordinate as a function of the new ones. Differentiate those functions, evaluate the Jacobian at the parameter point of interest, and calculate $J^\mathsf{T}I_\theta J$. The result should be symmetric; if $I_\theta$ is positive semidefinite, the transformed matrix should also be positive semidefinite. A failure of either check often indicates a transposition, ordering, or arithmetic error.

Directly calculating information in the target coordinates is an alternative to transforming an existing matrix. When both calculations use the same model, parameter point, and regularity conditions, they should agree. Direct calculation can be useful as a check, but may be less convenient when a population Fisher matrix has already been derived in another coordinate system. Neither approach removes the need to specify the parameterization and ordering explicitly.

Some design criteria respond differently to reparameterization. If $J$ is nonsingular, then

\det(I_\eta)=\det(J)^2\det(I_\theta).

At a fixed parameter point, a smooth transformation that is the same for every candidate design therefore multiplies the determinant by the same positive factor; determinant-based design rankings are unchanged. However, the numerical determinant changes, and criteria based on particular diagonal elements or inverse-matrix elements can change with the chosen coordinates. Always state the parameter scale when interpreting a variance estimate or its uncertainty.

The usual transformation also has limits. At $\rho=\pm1$, a two-effect covariance matrix is singular and the correlation lies at the boundary; regular interior-parameter arguments may fail. Near boundaries, numerical derivatives and matrix calculations can be poorly conditioned. A transformation such as $z=\operatorname{atanh}(\rho)$ can represent correlations on an unconstrained scale, but it requires its own Jacobian, $d\rho/dz=1-\rho^2$, and does not make boundary information problems disappear.

Common Mistakes

Using the inverse Jacobian in the wrong direction. If information is known in $\theta$ and needed in $\eta$, use $\partial\theta/\partial\eta$, not $\partial\eta/\partial\theta$.
Confusing covariance with correlation. The mapping $c_{12}=\rho\tau_1\tau_2$ is needed when covariance is an old coordinate. It does not belong in a transformation whose old coordinates already include $\rho$.
Comparing raw information numbers across scales. Information values depend on units and parameterization. Interpret them in the coordinates in which they were calculated.
Applying the expected-information rule blindly to an observed Hessian. The observed Hessian has a score-dependent curvature term under nonlinear transformations; the simple congruence rule applies directly to expected Fisher information.

Key takeaway: Write the old parameters as functions of the new parameters, differentiate that mapping, and use $I_\eta=J^\mathsf{T}I_\theta J$. In variance, standard-deviation, covariance, and correlation calculations, the exact choice and ordering of coordinates determine the Jacobian.

Check Your Understanding

Answer each question using the stated coordinate order. Solutions are included after the questions.

If $I_v=3$ at $v=4$, what are $I_\tau$ and $I_\lambda$ for $\tau=\sqrt v$ and $\lambda=\log v$?
For old coordinates $(v_1,v_2,\rho)$ and new coordinates $(\tau_1,\tau_2,\rho)$, what is the Jacobian?
Why does the covariance-to-standard-deviation-and-correlation Jacobian generally have off-diagonal entries?
What two matrix checks can help identify an error in a transformed Fisher matrix?
Does a change of parameterization necessarily change determinant-based design rankings across candidate designs? State the condition for rankings to remain unchanged.

Solutions. (1) Since $v=4$, $\tau=2$, and $dv/d\tau=4$, $I_\tau=4^2(3)=48$. Also $dv/d\lambda=v=4$, so $I_\lambda=4^2(3)=48$. (2) The Jacobian is $\operatorname{diag}(2\tau_1,2\tau_2,1)$, because $v_i=\tau_i^2$ and $\rho$ is unchanged. (3) Holding correlation fixed, changing either standard deviation changes the covariance $c_{12}=\rho\tau_1\tau_2$, so the covariance row has nonzero derivatives with respect to both standard deviations. (4) Check that the result is symmetric and positive semidefinite when the original Fisher matrix is positive semidefinite. (5) No. At a fixed parameter point, the rankings remain unchanged if the same nonsingular transformation applies to every candidate design, because its squared Jacobian determinant multiplies every Fisher determinant by the same factor.