Reparameterization And Information Transformation in Optimal Design

Why Parameter Scale Matters

A model may describe the same scientific quantities using different parameter coordinates. A clearance might be expressed directly in volume per time, for example, or through its logarithm. The underlying model has not changed, but the numerical information matrix does: its entries depend on the coordinates used to represent the parameters. Transforming the matrix correctly is essential when interpreting precision or comparing designs on different scales.

The central result is a Jacobian transformation rule. It follows from how the likelihood score changes coordinates, and it applies to expected Fisher information when the parameter transformation is smooth and locally invertible. Observed information requires an additional qualification: away from a likelihood maximum, curvature in the transformation contributes an extra term.

Definition: Let $\theta$ be a $p$-dimensional parameter vector and let $\eta=h(\theta)$ be a one-to-one, twice differentiable reparameterization. Write the inverse map as $\theta=q(\eta)$, and define its Jacobian by $K(\eta)=\partial\theta/\partial\eta^\mathsf{T}$, with entry $K_{ij}=\partial\theta_i/\partial\eta_j$. If $K$ is nonsingular, the expected Fisher information transforms as $$ I_\eta(\eta)=K(\eta)^\mathsf{T} I_\theta(\theta)K(\eta). $$

Here $I_\theta$ is the information expressed in $\theta$-coordinates, and $I_\eta$ is the information for the same observations expressed in $\eta$-coordinates. The parameter values in the two matrices correspond through $\theta=q(\eta)$. The nonsingularity condition means the transformation has a valid local inverse: no parameter direction has been lost or collapsed.

Why the Jacobian Rule Holds

Let $\ell(\theta)$ be the log likelihood for the observed data, and let $s_\theta=\partial\ell/\partial\theta$ be its score column vector. The chain rule gives the score in the new coordinates:

s_\eta=\frac{\partial\ell}{\partial\eta} =K^\mathsf{T}s_\theta.

Fisher information can be defined as the expected outer product of the score, $I_\theta=E_\theta[s_\theta s_\theta^\mathsf{T}]$. Substituting the transformed score and treating $K$ as fixed at the parameter value being evaluated gives:

\begin{aligned} I_\eta &=E_\theta[s_\eta s_\eta^\mathsf{T}]\\ &=E_\theta[(K^\mathsf{T}s_\theta)(K^\mathsf{T}s_\theta)^\mathsf{T}]\\ &=K^\mathsf{T}E_\theta[s_\theta s_\theta^\mathsf{T}]K\\ &=K^\mathsf{T}I_\theta K. \end{aligned}

Under the usual regularity conditions, expected information is also the expected negative Hessian of the log likelihood. The same transformation follows from that definition because the extra second-derivative terms have expectation zero when the expected score is zero. This relies on a regular likelihood model, including differentiability and the usual interchange of differentiation and integration. The score-outer-product derivation makes the coordinate transformation especially direct.

This is a congruence transformation, not an ordinary similarity transformation. In particular, the entries, eigenvalues, and determinant of the matrix generally change. The information continues to describe the same likelihood, but its numerical representation reflects the chosen parameter coordinates.

Worked Example: Information for a Log-Rate Parameter

Suppose $n=8$ independent observations follow an exponential distribution with rate $\lambda$, measured in inverse hours. The Fisher information for the rate is $I_\lambda=n/\lambda^2$. At the hypothetical nominal value $\lambda=0.4\ \mathrm{h}^{-1}$, calculate the information for $\eta=\log(\lambda/(1\ \mathrm{h}^{-1}))$.

The reference rate makes the logarithm's argument dimensionless. The inverse transformation is $\lambda=(1\ \mathrm{h}^{-1})e^\eta$, so $d\lambda/d\eta=\lambda$. The scalar Jacobian rule is therefore $I_\eta=(d\lambda/d\eta)^2I_\lambda$. Substituting the information and nominal rate:

I_\eta =\lambda^2\frac{n}{\lambda^2} =n=8.

Thus the information for this log-rate coordinate is $8$, while the information for the rate coordinate is $8/(0.4\ \mathrm{h}^{-1})^2=50\ \mathrm{h}^2$. The numerical values differ because the coordinates and their units differ. In this model, the transformed information is constant in $\eta$; that does not mean the rate itself is known without uncertainty.

Two-Parameter Transformations and Approximate Covariance

The matrix rule matters especially when parameters are correlated. Suppose $\theta=(\mathrm{CL},V)^\mathsf{T}$, with clearance $\mathrm{CL}$ in L/h and volume $V$ in L. Use the coordinates $\eta=(\log(\mathrm{CL}/(1\ \mathrm{L/h})),\log(V/(1\ \mathrm{L})))^\mathsf{T}$. The inverse map has Jacobian $K=\operatorname{diag}(\mathrm{CL},V)$, where the numerical values are understood in the stated units.

Worked Example: Transforming a Two-Parameter Matrix

At the hypothetical nominal values $\mathrm{CL}=5\ \mathrm{L/h}$ and $V=20\ \mathrm{L}$, suppose the information in the original coordinates is

I_\theta= \begin{pmatrix} 0.04 & 0.01\\ 0.01 & 0.09 \end{pmatrix}.

The entries have the corresponding inverse squared or inverse-product units. The Jacobian is $K=\begin{pmatrix}5&0\\0&20\end{pmatrix}$. Apply the rule by multiplying on both sides:

I_\eta =K^\mathsf{T}I_\theta K = \begin{pmatrix}5&0\\0&20\end{pmatrix} \begin{pmatrix}0.04&0.01\\0.01&0.09\end{pmatrix} \begin{pmatrix}5&0\\0&20\end{pmatrix} = \begin{pmatrix}1&1\\1&36\end{pmatrix}.

For example, the upper-right entry is $5(0.01)(20)=1$, and the lower-right entry is $20^2(0.09)=36$. The determinant is $35$, so this transformed matrix is nonsingular. Its information entries are on the log-parameter scale.

For a regular model with nonsingular information, the inverse information approximates the covariance matrix of an efficient estimator at large sample sizes. In the original coordinates, the inverse matrix is

I_\theta^{-1} =\frac{1}{0.0035} \begin{pmatrix}0.09&-0.01\\-0.01&0.04\end{pmatrix} \approx \begin{pmatrix}25.714&-2.857\\-2.857&11.429\end{pmatrix}.

On the log scale, the corresponding inverse is

I_\eta^{-1} =\frac{1}{35} \begin{pmatrix}36&-1\\-1&1\end{pmatrix} \approx \begin{pmatrix}1.0286&-0.02857\\-0.02857&0.02857\end{pmatrix}.

These matrices agree with the covariance transformation $I_\eta^{-1}=K^{-1}I_\theta^{-1}K^{-\mathsf{T}}$. For example, the approximate log-clearance variance is $25.714/5^2=1.0286$. This relation also follows by locally transforming estimation errors: a small change satisfies $d\eta=K^{-1}d\theta$. The covariance transformation is a first-order approximation for estimator uncertainty, not an exact statement about every finite-sample estimator distribution.

Observed Information Is Not Always a Simple Congruence

Observed information is the negative Hessian evaluated at the observed data, rather than its expectation. For a nonlinear transformation, differentiating twice introduces a term involving the score. In scalar notation, with $\theta=q(\eta)$:

-\frac{d^2\ell}{d\eta^2} = \left(\frac{d\theta}{d\eta}\right)^2 \left(-\frac{d^2\ell}{d\theta^2}\right) - \frac{d\ell}{d\theta}\frac{d^2\theta}{d\eta^2}.

The second term vanishes at a maximum-likelihood estimate when the score is zero, but need not vanish at other parameter values. Expected information transforms by the Jacobian rule under regularity conditions; observed information at an arbitrary point generally does not.

Worked Example: A Curvature Term in Observed Information

Consider a hypothetical observed log likelihood, evaluated locally at $\theta=2$, with score $d\ell/d\theta=4$ and observed information $-d^2\ell/d\theta^2=3$. Let $\eta=\log\theta$, so $\theta=e^\eta$. At this point, $d\theta/d\eta=2$ and $d^2\theta/d\eta^2=2$.

Using the full observed-information formula gives

I_{\mathrm{obs},\eta} =2^2(3)-4(2)=12-8=4.

Simply applying the congruence expression to observed information would give $2^2(3)=12$, which is incorrect here because the score is nonzero. At a likelihood maximum, the score term would be zero, and the transformed observed information would equal $12$ at that point. This example shows why the expected-information rule should not be applied uncritically to an observed Hessian away from a maximum.

What Changes—and What Does Not—in Design Comparisons

Information-based design calculations are made at specified nominal parameter values. If the same smooth reparameterization is applied to every candidate design, its Jacobian at those fixed parameter values is the same across designs. Thus, for two designs $A$ and $B$,

I_{\eta,A}=K^\mathsf{T}I_{\theta,A}K, \qquad I_{\eta,B}=K^\mathsf{T}I_{\theta,B}K.

For example, the determinants satisfy $\det(I_\eta)=(\det K)^2\det(I_\theta)$. The common positive multiplier means determinant rankings are unchanged when comparing the designs at the same parameter point. This does not make every design criterion coordinate-invariant. Criteria based on particular variances or weighted combinations can change when the parameter scale changes, unless the scientific target is transformed consistently as well. The important practice is to state which parameters or estimands matter and use coordinates that preserve that target.

Key takeaway: For expected Fisher information, transform with the Jacobian of the inverse parameter map: $I_\eta=K^\mathsf{T}I_\theta K$. Check the direction of the Jacobian, evaluate it at the corresponding parameter values, and distinguish expected information from observed curvature away from a likelihood maximum.

Diagnostics and Common Mistakes

Before interpreting a transformed information matrix, check the transformation and the calculation. A nonsingular, correctly oriented Jacobian should preserve positive definiteness when the original information is positive definite. If a positive-definite matrix becomes singular or indefinite after a purported smooth bijection with nonsingular Jacobian, suspect an algebra or implementation error.

Using the forward Jacobian. If $\eta=h(\theta)$, the rule as written needs $K=\partial\theta/\partial\eta^\mathsf{T}$, not $\partial\eta/\partial\theta^\mathsf{T}$. Derive the inverse map explicitly.
Transforming only diagonal entries. Off-diagonal entries must also be multiplied by the appropriate Jacobian factors; otherwise the dependence structure is misrepresented.
Ignoring units in logarithms. A logarithm should have a dimensionless argument. Specify a reference unit or scale when defining log parameters.
Applying the expected-information rule to any observed Hessian. Away from a likelihood maximum, nonlinear transformation curvature contributes a score-dependent term.
Comparing raw information entries across scales. Their numerical magnitudes depend on the parameterization. Interpret the chosen scale and scientific target, rather than treating a larger entry alone as universal evidence of a better design.

A parameter transformation also cannot repair a genuine loss of local identifiability. If the Jacobian is nonsingular, the transformed matrix has the same rank as the original. If the Jacobian is singular, it is not a valid change of coordinates for all parameter directions, and the usual inverse-based information interpretations may fail.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

If $\eta=\log(\theta/\theta_{\mathrm{ref}})$, what derivative belongs in the scalar information transformation rule?
A two-parameter Jacobian is $K=\operatorname{diag}(2,3)$, and $I_\theta=\begin{pmatrix}4&1\\1&5\end{pmatrix}$. Calculate $I_\eta$.
Why can determinant-based design rankings remain unchanged under a fixed reparameterization, while individual parameter-variance criteria may not?
When does the score-dependent term in the observed-information transformation vanish?

Solutions. (1) The inverse map is $\theta=\theta_{\mathrm{ref}}e^\eta$, so $d\theta/d\eta=\theta$, and $I_\eta=\theta^2 I_\theta$. (2) $K^\mathsf{T}I_\theta K=\begin{pmatrix}16&6\\6&45\end{pmatrix}$: the off-diagonal entry is $2(1)(3)=6$. (3) At a fixed nominal parameter point, the Jacobian is common to all designs, so their determinants all acquire the same positive multiplier $(\det K)^2$. Variances and their combinations refer to coordinates and targets that can change with scale. (4) It vanishes when the score is zero, as at an interior maximum-likelihood estimate satisfying the score equation.