The Leverage Adjustment in Logistic Scores

From Jeffreys’ Penalty to Leverage

Firth’s adjustment to the logistic score can be written in terms of fitted probabilities and leverage values. This form reveals where the adjustment enters: each observation contributes according to both its fitted probability and its share of the model’s information. It also clarifies how to adapt the score when observations are reported as binomial counts for groups rather than as individual zero-one outcomes.

Consider independent Bernoulli responses \(Y_1,\ldots,Y_n\), with \(Y_i\in\{0,1\}\), and a logistic regression model with \(p\) coefficients. Write \(x_i\) for the \(p\)-vector of covariates for observation \(i\), and let \(X\) be the \(n\times p\) matrix with rows \(x_i^\mathsf{T}\). Assume \(X\) has full column rank. At a finite coefficient vector \(\beta\), define \(\eta_i=x_i^\mathsf{T}\beta\), \(\pi_i=P(Y_i=1)=\exp(\eta_i)/(1+\exp(\eta_i))\), and \(w_i=\pi_i(1-\pi_i)\).

Because every \(w_i>0\) at finite \(\beta\) and \(X\) has full column rank, \(I(\beta)\) is positive definite and its inverse exists. The ordinary logistic score is \(U(\beta)=X^\mathsf{T}(y-\pi)\), where \(y=(Y_1,\ldots,Y_n)^\mathsf{T}\) and \(\pi=(\pi_1,\ldots,\pi_n)^\mathsf{T}\). Firth’s adjusted score adds the derivative of half the log determinant of the information matrix to this ordinary score.

Here the likelihood is the ordinary Bernoulli likelihood, and the penalty is defined wherever \(I(\beta)\) is positive definite. The leverage values depend on \(\beta\), just as the fitted probabilities do. The displayed equation gives the adjusted score at each candidate \(\beta\); it does not assert that the score equation has a solution or describe an algorithm for finding one.

Deriving the Adjustment

The key calculation is the derivative of \(\log|I(\beta)|\). For a differentiable positive-definite matrix \(I\), the determinant identity gives \(\partial\log|I|/\partial\beta_j=\operatorname{tr}(I^{-1}\partial I/\partial\beta_j)\). We now apply it to the logistic information matrix.

Putting these steps together, and using the linearity of the trace, yields

The derivative of the penalty \(\tfrac12\log|I(\beta)|\) is therefore \(\tfrac12\sum_i h_i(1-2\pi_i)x_{ij}\) in component \(j\). Since \(\tfrac12(1-2\pi_i)=\tfrac12-\pi_i\), the full adjusted score has component

Stacking these components over \(j=1,\ldots,p\) gives the theorem’s vector expression. This derivation also shows why the correction is not simply an added constant to the response: both \(h_i\) and \(\pi_i\) vary with the model parameters. \(\square\)

Worked Examples

Grouped Binomial Responses

Now suppose the data consist of \(G\) independent groups. Group \(g\) has \(m_g\) Bernoulli trials, of which \(Y_g\) are successes, with common probability \(\pi_g\) within the group and logistic linear predictor \(\eta_g=x_g^\mathsf{T}\beta\). The grouped response has distribution \(Y_g\sim\operatorname{Bin}(m_g,\pi_g)\). Assume the \(G\times p\) design matrix with rows \(x_g^\mathsf{T}\) has full column rank and \(m_g>0\).

The grouped score and information are

Define the grouped-row leverage

This is the diagonal element of the weighted hat matrix formed from the \(G\) grouped rows, whose row weight for group \(g\) is \(m_g\pi_g(1-\pi_g)\). It equals the sum of the individual-trial leverages for the \(m_g\) replicated rows in that group.

To verify the result, write the information as \(I=\sum_g m_gw_gx_gx_g^\mathsf{T}\), where \(w_g=\pi_g(1-\pi_g)\). Differentiation gives \(\partial w_g/\partial\beta_j=w_g(1-2\pi_g)x_{gj}\), so

Applying the determinant derivative and taking the trace produces

Adding this derivative to the grouped ordinary score gives the stated expression component by component. Equivalently, expand each group into \(m_g\) Bernoulli rows: each replicated row has leverage \(w_gx_g^\mathsf{T}I^{-1}x_g\), so their total leverage is exactly \(H_g\). The grouped formula is therefore the individual-trial adjustment summed within groups. \(\square\)

Interpreting Leverage and Avoiding a Common Error

Leverage summarizes how strongly a row contributes to the fitted model’s information geometry. In the individual Bernoulli form, the correction for observation \(i\) is \(h_i(\tfrac12-\pi_i)\), before multiplication by its covariate vector. If \(\pi_i\) is below one half, that scalar correction is positive; if it is above one half, it is negative. Its effect on a particular coefficient also depends on the sign and size of the corresponding covariate.

A useful check is \(\sum_i h_i=p\). Indeed, the sum of the diagonal elements of \(H\) is its trace, and

For grouped data, the corresponding identity is \(\sum_g H_g=p\). These identities are useful consistency checks, but they do not imply that all observations or groups have the same leverage.

• Common mistake: Use the grouped-row hat diagonal as though it were the leverage of each individual trial, then multiply it by the group size again. This counts the replication twice.
• Correction: The grouped-row leverage \(H_g\) already includes \(m_g\). If working from individual trials, each trial has leverage \(H_g/m_g\), and summing the \(m_g\) corrections gives \(H_g(\tfrac12-\pi_g)x_g\).
• Another mistake: Add \(h_i(\tfrac12-\pi_i)\) to the score without multiplying by \(x_i\). The scalar changes the residual contribution, but the coefficient score receives it in the direction of the observation’s covariate vector.

The leverage form is valuable because it connects the Jeffreys penalty to the design and gives a direct translation between individual and grouped data representations. It also makes clear that the same fitted probability can receive different adjustment weight at different design locations. The formula is an expression for the adjusted score; conclusions about existence, numerical solution, or inferential procedures require separate arguments.