Deriving Firth's Adjusted Score Equations

From Jeffreys Penalty to an Adjusted Score

Firth’s method changes a likelihood equation by adding a term determined by how the expected information varies with the parameter. That term is not an arbitrary correction: it is the derivative of the Jeffreys penalty. Its form also explains why, under suitable regularity conditions, solving the adjusted equation removes the leading small-sample bias of maximum likelihood.

Let \(\beta=(\beta^1,\ldots,\beta^p)^\mathsf{T}\) be a parameter vector, let \(\ell(\beta)\) be the log-likelihood, and write \(U_a(\beta)=\partial\ell(\beta)/\partial\beta^a\) for its score component. The expected Fisher information is \(I_{rs}(\beta)=E_\beta[U_rU_s]\). Assume it is finite, differentiable, and positive definite in the parameter region considered.

The determinant in the penalty is the determinant of the expected information, not the observed information. Positive definiteness ensures \(|I(\beta)|>0\), so the logarithm and its derivative are defined. The adjustment is the derivative of this penalty; it is not generally a constant shift in the score.

Differentiating the Penalty

For a differentiable nonsingular matrix \(I(\beta)\), the determinant derivative identity is \(\partial\log|I|/\partial\beta^a=\operatorname{tr}(I^{-1}\partial I/\partial\beta^a)\). Define \(A_a(\beta)\) to be the added score component. Applying the identity gives the adjusted score directly.

This result follows by differentiating the two terms in \(\ell^*\): the derivative of \(\ell\) is \(U_a\), and the derivative of \(\tfrac12\log|I|\) is \(A_a\). The inverse information in the trace weights changes in information across parameter directions; the trace contracts that matrix derivative to one scalar adjustment for each score component.

Why the Adjustment Targets First-Order Bias

A precise bias argument can be given for a regular canonical exponential-family model with fixed design and a fixed-dimensional parameter. Write its log-likelihood as \(\ell(\beta)=\sum_i\{y_i\eta_i(\beta)-b_i(\eta_i(\beta))\}+c(y)\), where \(\eta_i(\beta)\) is linear in \(\beta\), the natural parameter. Then the Hessian is nonrandom, and the score has mean zero and covariance \(I\). Assume the true parameter is an interior point, \(I\) is nonsingular, the likelihood is sufficiently smooth, the required moments exist, and the maximum-likelihood estimator is consistent with the usual Taylor expansion and bias remainder of order \(O(n^{-2})\). These conditions support the expansion below; they are not claims about models with nonexistence or irregular boundary behavior.

Use \(\ell_{rst}=\partial^3\ell/(\partial\beta^r\partial\beta^s\partial\beta^t)\), and let \(\kappa_{rst}=E[\ell_{rst}]\). In a canonical model with fixed design, the Hessian and third derivatives are nonrandom. Also, differentiating \(I_{rs}=-E[\ell_{rs}]\) gives \(\partial I_{rs}/\partial\beta^a=-\kappa_{rsa}\). Consequently, \(A_a=-\tfrac12 I^{rs}\kappa_{ars}\).

To see the source of this formula, let \(\delta=\hat\beta_{\mathrm{ML}}-\beta\). Expand the score equation around the true parameter. Because the Hessian is nonrandom and equals \(-I\), its leading terms are \(0=U_r-I_{rs}\delta^s+\tfrac12\ell_{rst}\delta^s\delta^t+\cdots\). The first-order solution is \(\delta^a=I^{ar}U_r\). Substituting it into the quadratic term gives the second-order contribution \(\tfrac12 I^{ar}\ell_{rst}I^{su}U_u I^{tv}U_v\). Taking expectations uses \(E[U_uU_v]=I_{uv}\), so the contribution is \(\tfrac12 I^{ar}I^{st}\kappa_{rst}\). The regular Taylor and moment conditions ensure that the omitted terms contribute \(O(n^{-2})\) to the bias.

Now contract that leading bias with the information. This step identifies exactly how the adjustment acts:

The order changes in the remainder because \(I_{ab}\) is of order \(n\). In particular, the middle expression is \( -A_a\), not \(I_{ab}A_b\): \(A_a=-\tfrac12 I^{st}\kappa_{ast}\) is already the adjustment component. This distinction matters when translating between a score-scale correction and a parameter-scale bias.

Finally, compare the adjusted root to the maximum-likelihood root. To first order, adding \(A\), which is order one, shifts the root by \(I^{-1}A\), which is order \(n^{-1}\). The same expansion yields \(\operatorname{Bias}(\hat\beta_F)=\operatorname{Bias}(\hat\beta_{\mathrm{ML}})+I^{-1}A+O(n^{-2})\). Since \(I^{-1}A=-\operatorname{Bias}(\hat\beta_{\mathrm{ML}})+O(n^{-2})\) by the contraction just shown, the order-\(n^{-1}\) bias cancels. Thus the adjusted estimator has no first-order bias term under these assumptions. This is a first-order statement, not a guarantee of zero finite-sample bias in every model.

Scope, Interpretation, and a Common Pitfall

The trace formula applies whenever the expected information is differentiable and nonsingular, whether or not a particular likelihood has a canonical representation. The bias derivation above was deliberately stated for a regular canonical model with fixed design, where the Hessian structure makes the leading calculation transparent. In other settings, establishing first-order bias reduction requires the corresponding regularity conditions and bias expansion; the trace formula alone is not a proof that every finite-sample estimate is unbiased.

• Pitfall: confusing the adjustment with its information-weighted version. From \(A_a=-\tfrac12 I^{rs}\kappa_{ars}\), one may correctly conclude \(I_{ab}\operatorname{Bias}(\hat\beta^b_{\mathrm{ML}})=-A_a+O(n^{-1})\). Replacing \(A_a\) on the right with \(I_{ab}A_b\) is wrong: the latter has a different scale and is not the adjustment component.
• Pitfall: treating the penalty as a fixed constant. Although the Jeffreys penalty is defined from the information, it can vary with \(\beta\). Its derivative vanishes only when \(|I(\beta)|\) is locally constant.
• Pitfall: claiming exact unbiasedness. The argument cancels the leading \(n^{-1}\) bias term under regularity conditions. Higher-order bias and finite-sample behavior need not vanish.