Reparameterization Invariance of Firth Estimates

What Does Reparameterization Invariance Mean?

A statistical model can often be written using several parameter codings. For example, a two-group logistic model may use the logit for the first group and a treatment contrast, or it may use the two group logits directly. These are different coordinates for the same fitted probabilities. An estimator is invariant under a change of coding when the estimate in the new coordinates represents exactly the same parameter point as the estimate in the original coordinates.

For maximum likelihood, this invariance follows because the likelihood is a scalar function of the model point: changing coordinates does not change which point maximizes it. Firth’s method adds a term involving the Fisher information determinant. That determinant transforms with a Jacobian, so the invariance question requires care. In particular, a smooth one-to-one reparameterization alone is not enough to guarantee invariance of a Jeffreys-penalized maximum.

Definition: Let $\theta$ be a parameter in an open subset of $\mathbb{R}^p$, with log-likelihood $\ell(\theta)$ and nonsingular expected Fisher information matrix $I_\theta(\theta)$. The Jeffreys-penalized log-likelihood is $$ \ell^*(\theta)=\ell(\theta)+\frac{1}{2}\log |I_\theta(\theta)|. $$ A Jeffreys-penalized estimate is a maximizer of $\ell^*(\theta)$, when such a maximizer exists. The determinant is taken in the coordinates $\theta$; consequently, this expression is a log density relative to coordinate volume, not a coordinate-free scalar objective.

The discussion below assumes a regular model: the parameter space is open, the likelihood and expected information are sufficiently differentiable, and $I_\theta$ is positive definite at the parameter values considered. For a reparameterization $\theta=g(\phi)$, assume $g$ is a differentiable one-to-one map with differentiable inverse and nonsingular Jacobian $J(\phi)=\partial\theta/\partial\phi^\mathsf{T}$. These conditions make the information transformation and determinant calculation well-defined.

How the Penalty Changes Coordinates

The likelihood itself transforms by substitution: in $\phi$-coordinates it is $\ell(g(\phi))$. The Fisher information transforms as a matrix under the Jacobian. The next result gives the corresponding transformation of the Jeffreys penalty.

Result: Under the regularity and differentiability conditions above, $$ I_\phi(\phi)=J(\phi)^\mathsf{T} I_\theta(g(\phi))J(\phi), $$ and hence $$ \ell^*_\phi(\phi)=\ell^*_\theta(g(\phi))+\log|\det J(\phi)|. $$

Proof. Write $U_\theta=\partial\ell(\theta)/\partial\theta$ for the score. By the chain rule, the score in $\phi$-coordinates is $U_\phi=J^\mathsf{T}U_\theta$. Fisher information is the covariance matrix of the score (equivalently, under the usual regularity conditions, the negative expected Hessian). Therefore

I_\phi=E[U_\phi U_\phi^\mathsf{T}] =J^\mathsf{T}E[U_\theta U_\theta^\mathsf{T}]J =J^\mathsf{T}I_\theta J.

\ell^*_\phi(\phi) =\ell(g(\phi))+\frac12\log|I_\phi(\phi)| =\ell^*_\theta(g(\phi))+\log|\det J(\phi)|.

This proves the stated transformation. $\square$

The extra term is constant when the absolute Jacobian determinant is constant. In particular, for an invertible affine transformation $\theta=A\phi+b$, the Jacobian is the constant matrix $A$, so the two penalized objectives differ only by $\log|\det A|$. Adding a constant cannot change a maximizer. This is the invariance relevant to ordinary changes among logistic contrast codings.

Theorem: If $\theta=A\phi+b$ with $A$ nonsingular, then every maximizer $\hat\theta$ of $\ell^*_\theta$ corresponds to a maximizer $\hat\phi=A^{-1}(\hat\theta-b)$ of $\ell^*_\phi$, and conversely. More generally, the same conclusion holds for a smooth one-to-one reparameterization whose Jacobian determinant has constant absolute value over the parameter space.

Proof. For an affine transformation, the preceding result gives $\ell^*_\phi(\phi)=\ell^*_\theta(A\phi+b)+\log|\det A|$. The map $\phi\mapsto A\phi+b$ is a bijection, and the added term is independent of $\phi$. Thus comparing the objective at any two $\phi$-values is exactly equivalent to comparing $\ell^*_\theta$ at their corresponding $\theta$-values. The maximizing sets therefore map onto one another. The same argument applies when $\log|\det J(\phi)|$ is constant. $\square$

Worked Logistic Contrast Codings

Worked Example: Baseline Coding Versus Two Group Logits

Suppose two independent groups have binomial responses, with $y_1=2$ successes out of $n_1=5$ in group 1 and $y_2=1$ out of $n_2=4$ in group 2. First use the two logits $\eta_1,\eta_2$, where $p_j=\operatorname{logit}^{-1}(\eta_j)$. In this saturated model the log-likelihood, apart from binomial constants, is

\ell(\eta_1,\eta_2)=\sum_{j=1}^2\{y_j\eta_j-n_j\log(1+e^{\eta_j})\}.

For group $j$, the information for $\eta_j$ is $n_jp_j(1-p_j)$. The information matrix is diagonal, so the penalty is one-half the sum of the logarithms of these two entries. Differentiating the penalized objective with respect to $\eta_j$ gives

y_j-n_jp_j+\frac12(1-2p_j)=0, \qquad p_j=\frac{y_j+1/2}{n_j+1}.

Thus $\hat p_1=(2+1/2)/6=5/12$ and $\hat p_2=(1+1/2)/5=3/10$. Their estimated logits are

\hat\eta_1=\log\frac{5/12}{7/12}=\log\frac57, \qquad \hat\eta_2=\log\frac{3/10}{7/10}=\log\frac37.

Now write the same model as $\eta_1=\alpha$ and $\eta_2=\alpha+\beta$, so that $\beta$ is the group-2 versus group-1 log-odds contrast. The map from $(\alpha,\beta)$ to $(\eta_1,\eta_2)$ has matrix $\begin{pmatrix}1&0\\1&1\end{pmatrix}$ and determinant $1$. The affine invariance result therefore applies. Transforming the estimates gives

\hat\alpha=\log\frac57, \qquad \hat\beta=\hat\eta_2-\hat\eta_1 =\log\frac37-\log\frac57 =\log\frac35.

The treatment contrast is also checked directly by the estimated odds ratio: group 2 has odds $3/7$, group 1 has odds $5/7$, and their ratio is $(3/7)/(5/7)=3/5$. The baseline-coded fit and the two-logit fit describe identical fitted probabilities.

Worked Example: Centered and Scaled Contrast Coding

Use the same two fitted group logits, but label the groups by covariate values $x=0$ and $x=2$. In the model $\eta(x)=\alpha+\beta x$, the group logits are $\eta_1=\alpha$ and $\eta_2=\alpha+2\beta$. The estimates from the preceding calculation imply

\hat\alpha=\log\frac57, \qquad \hat\beta=\frac{\hat\eta_2-\hat\eta_1}{2} =\frac12\log\frac35.

Instead code the predictor as $z=(x-1)/2$, giving values $-1/2$ and $1/2$, and write $\eta(x)=\gamma+\delta z$. At the two groups, $\eta_1=\gamma-\delta/2$ and $\eta_2=\gamma+\delta/2$, so

\gamma=\frac{\eta_1+\eta_2}{2}, \qquad \delta=\eta_2-\eta_1.

This is an invertible affine transformation of the two model coefficients. Its determinant has absolute value $2$, which changes the penalized objective by the constant $\log 2$, not its maximizer. Numerically in exact log form,

\hat\gamma=\frac12\log\frac{15}{49}, \qquad \hat\delta=\log\frac35.

The logits at the observed covariate values are unchanged: $\hat\gamma-\hat\delta/2=\log(5/7)$ and $\hat\gamma+\hat\delta/2=\log(3/7)$. Thus centering and scaling the contrast changes the coefficient values, as intended, but not the fitted model.

Why Arbitrary Smooth Reparameterizations Can Fail

A nonlinear change of coordinates can have a Jacobian determinant that varies with the parameter. Then the added term $\log|\det J(\phi)|$ is not constant, and it can move the maximum. This is not a failure of the information transformation: it is a consequence of maximizing a density whose reference coordinate volume has changed.

Worked Example: A Nonlinear Change That Moves the Estimate

Observe one Bernoulli success, and let $\theta$ be the logit of the success probability $p$, so $p=e^\theta/(1+e^\theta)$. The ordinary log-likelihood is $\ell(\theta)=\theta-\log(1+e^\theta)$, and the information is $p(1-p)$. The Jeffreys-penalized score is

\frac{d\ell^*}{d\theta} =(1-p)+\frac12(1-2p) =\frac32-2p.

Setting this equal to zero gives $\hat p=3/4$, hence $\hat\theta=\log 3$. Now use the smooth one-to-one coordinate $\phi=e^\theta$, with $\phi>0$. Then $\theta=\log\phi$ and $d\theta/d\phi=1/\phi$. By the transformation formula, the penalized objective in $\phi$-coordinates is the original objective evaluated at $\theta=\log\phi$, plus $\log(1/\phi)=-\theta$. Its derivative with respect to $\theta$ is therefore

\frac{d}{d\theta}\{\ell^*(\theta)-\theta\} =\frac32-2p-1 =\frac12-2p.

The transformed-coordinate maximum has $p=1/4$, not $3/4$. It is an interior maximum: the derivative is positive when $p<1/4$, negative when $p>1/4$, and $p$ increases strictly with $\theta$. The transformed estimate is $\hat\theta=\log(1/3)$, or $\hat\phi=1/3$. This explicit calculation disproves invariance under arbitrary smooth one-to-one changes.

Interpretation and a Common Pitfall

The Jeffreys prior defines an invariant measure, but its density in a particular coordinate system includes the coordinate-volume factor. A density mode can change when that factor changes. Accordingly, when discussing Firth estimates, distinguish an affine recoding of the same regression coefficients from an arbitrary nonlinear transformation. Standard logistic contrast changes—such as switching reference groups, replacing cell logits by an intercept and contrast, or centering and scaling predictors—are invertible affine transformations. Their Jacobians are constant, and the fitted model and transformed estimates agree exactly.

Wrong reasoning: “The Fisher information transforms by a Jacobian, so the determinant penalty must make the estimate invariant under every smooth reparameterization.”
Correction: The determinant contributes $\log|\det J(\phi)|$ to the transformed penalized objective. Only when that contribution is constant does it leave the maximizer unchanged.
Practical check: Write the parameter map and compute its Jacobian determinant. Do not infer invariance from one-to-one-ness alone.

Key takeaway: Jeffreys-penalized estimates are equivariant under affine changes of parameter coding, and more generally under smooth changes with constant absolute Jacobian determinant. A general nonlinear reparameterization changes the coordinate density and can change its maximizer.

Check Your Understanding

Use the transformation formula and the worked calculations to answer these questions.

For $\theta=A\phi+b$, state how the Fisher information determinant changes and why the maximizing model point is preserved.
What additional term appears in the transformed Jeffreys-penalized log-likelihood under $\theta=g(\phi)$?
In the two-group example, why does the baseline-coded contrast estimate equal $\log(3/5)$?
For the nonlinear example, what are the original and transformed estimates of the success probability?
Does a smooth one-to-one transformation guarantee that a Jeffreys-penalized estimate is unchanged? Explain the role of the Jacobian determinant.