Firth's Jeffreys-Penalized Likelihood

From Logistic Likelihood to an Information-Based Penalty

In logistic regression, the ordinary log-likelihood measures how well a coefficient vector explains the observed binary outcomes. In small or sparse samples, maximizing that likelihood can produce estimates with substantial bias, and the likelihood may not have a finite maximizer. Firth’s method modifies the likelihood using the model’s expected information. The resulting adjustment is not an arbitrary fixed penalty on the coefficients: its value depends on how much information the model has at each parameter value.

This tutorial develops the objective itself. Let \(Y_1,\ldots,Y_n\) be independent Bernoulli responses with fixed covariate vectors \(x_i\in\mathbb R^p\), and suppose \(\operatorname{logit}(P(Y_i=1\mid x_i))=x_i^\mathsf{T}\beta\), where \(\beta\in\mathbb R^p\). Write \(p_i=P(Y_i=1\mid x_i)\), so \(p_i=\exp(x_i^\mathsf{T}\beta)/\{1+\exp(x_i^\mathsf{T}\beta)\}\). Assume the design matrix \(X\), whose \(i\)th row is \(x_i^\mathsf{T}\), has full column rank. Then the expected information is positive definite for every finite \(\beta\), and its determinant is positive.

The absolute determinant measures the volume of information across parameter directions. The logarithm converts that volume factor into an additive term on the log-likelihood scale. In the logistic model, the expected information has a particularly direct form. Let \(W\) be the diagonal matrix with entries \(p_i(1-p_i)\). Then \(I(\beta)=X^\mathsf{T}WX\), and hence the objective is \(\ell_J(\beta)=\ell(\beta)+\tfrac12\log|X^\mathsf{T}WX|\).

The likelihood formula omits only terms that do not depend on \(\beta\). To see why the information takes this form, the derivative of the \(i\)th log-likelihood contribution with respect to its linear predictor is \(y_i-p_i\), and its second derivative is \(-p_i(1-p_i)\). Since the covariates are fixed, taking the negative expected Hessian gives \(\sum_i p_i(1-p_i)x_ix_i^\mathsf{T}=X^\mathsf{T}WX\). Full column rank of \(X\) and positivity of every \(p_i(1-p_i)\) at finite coefficients ensure positive definiteness: for nonzero \(v\), \(v^\mathsf{T}X^\mathsf{T}WXv=(Xv)^\mathsf{T}W(Xv)>0\).

Worked Calculations

Why the Penalty Is Connected to Jeffreys

Jeffreys’s rule associates a prior measure with expected information. In coordinates \(\theta\in\mathbb R^p\), its density with respect to ordinary coordinate volume is proportional to \(|I(\theta)|^{1/2}\). Multiplying the likelihood by this density and taking logarithms gives \(\ell(\theta)+\tfrac12\log|I(\theta)|\), up to an additive constant. This explains the name and construction of Firth’s objective. It does not mean that a fixed quadratic or absolute-value penalty has been added to the coefficients.

To verify the result, the score in the \(\gamma\) coordinates is \(J^\mathsf{T}\) times the score in the \(\theta\) coordinates, by the chain rule. Taking the expected outer product of the scores, or equivalently transforming the information as a quadratic form, yields the displayed identity. The determinant product rule then gives \[ |I_\gamma(\gamma)|^{1/2} =|J(\gamma)|\,|I_\theta(g(\gamma))|^{1/2}, \] where \(|J|\) denotes the absolute value of the Jacobian determinant. The change-of-variables formula for integration is \(d\theta=|J(\gamma)|\,d\gamma\). Thus \(|I_\theta(\theta)|^{1/2}d\theta =|I_\gamma(\gamma)|^{1/2}d\gamma\), which proves the measure is the same under the coordinate change. \(\square\)

There is an important distinction between invariance of this measure and invariance of the mode of a density. In the new coordinates, the log-likelihood plus the log Jeffreys density is \(\ell(g(\gamma))+\tfrac12\log|I_\gamma(\gamma)|\). By the determinant identity, this equals \(\ell(g(\gamma))+\tfrac12\log|I_\theta(g(\gamma))|+\log|J(\gamma)|\). If the Jacobian determinant varies with \(\gamma\), the additional term can change the location of a density mode. If the change of coordinates is affine with constant nonsingular Jacobian, \(\log|J|\) is constant, so maximizing the objective gives corresponding estimates in either coordinate system. The precise invariance claim is therefore that Jeffreys’s measure transforms correctly, not that every coordinate density has the same numerical mode.

Using the Objective Carefully

The information penalty is model-dependent and parameter-dependent. In logistic regression, changing the design, adding a covariate, or changing fitted probabilities changes \(X^\mathsf{T}WX\), and therefore can change the determinant and its contribution to the objective. By contrast, a positive multiplicative constant in the information determinant adds only a constant to the penalized log-likelihood and does not change its maximizer.

The formula requires a nonsingular expected information matrix for the stated determinant to be positive and its logarithm to be defined. In the logistic model, a full-column-rank design ensures this for finite coefficients. If the design is rank deficient, the determinant is zero and this objective is not defined as written; one cannot treat the log determinant as an ordinary finite penalty without first addressing the model’s identifiability.

• Common mistake: treating the correction as a fixed coefficient penalty. The term is \(\tfrac12\log|I(\beta)|\), not a preset multiple of \(\|\beta\|^2\) or \(\|\beta\|_1\). Its value depends on the expected information at the current coefficients.
• Common mistake: applying Jeffreys invariance to density modes. Jeffreys’s prior measure is invariant under smooth one-to-one reparameterization. A density mode need not be, because a coordinate change with nonconstant Jacobian adds \(\log|J(\gamma)|\) to the log density.
• Common mistake: miscounting failures in grouped data. For \(y\) successes and \(m\) failures, the two coefficients are \(y+\tfrac12\) and \(m+\tfrac12\). The failure coefficient comes from \(m\), not from \(y\) or the group total.