Penalized Fisher Scoring and IRLS

From a GLM Likelihood to a Quadratic Step

Penalized GLM estimation is usually not a single closed-form calculation. At a current coefficient vector, Fisher scoring replaces the likelihood locally with a quadratic expression; adding the coefficient penalty then gives a penalized quadratic subproblem. For GLMs, that subproblem has the form of weighted least squares, with a working response and weights recalculated at each iteration.

A crucial distinction is that Fisher scoring uses expected information, whereas a second-order Taylor expansion of the log-likelihood uses the observed Hessian. These are generally different matrices. The Fisher-scoring quadratic is a useful local surrogate, not a claim that the observed log-likelihood itself has that exact quadratic expansion.

Proof. The second-order Taylor theorem for a twice continuously differentiable function gives the displayed expansion, with the observed Hessian \(H(\beta)\). Separately, the Fisher-scoring surrogate has gradient with respect to \(\delta\) equal to \(U(\beta)-I(\beta)\delta\). Setting this gradient to zero gives \(\delta=I(\beta)^{-1}U(\beta)\). Its Hessian is \(-I(\beta)\), which is negative definite when \(I(\beta)\) is positive definite, so this stationary point is the unique maximizer. \(\square\)

The GLM Working Response and Weights

Consider independent observations \(Y_i\) from a GLM with mean \(\mu_i\), variance \(\operatorname{Var}(Y_i)=\phi V(\mu_i)/a_i\), and linear predictor \(\eta_i=o_i+x_i^T\beta\). Here \(x_i^T\) is row \(i\) of the design matrix \(X\), \(o_i\) is a fixed offset, \(a_i\) is a known prior weight, and \(\phi\) is the dispersion parameter. The mean is related to the linear predictor through a differentiable link, so that \(\mu_i=g^{-1}(\eta_i)\). Assume the derivative \(d\mu_i/d\eta_i\) is nonzero at the current iterate.

At iteration \(t\), evaluate \(\eta_i\), \(\mu_i\), and the derivative at \(\beta^{(t)}\). Define the working response \(z_i\) and working weight \(w_i\) by

The working response is on the linear-predictor scale. It is a local adjustment: it starts at the current \(\eta_i\) and adds a residual multiplied by the local slope \(d\eta_i/d\mu_i\). The weights reflect both the variance of the response and the sensitivity of the mean to changes in the predictor.

Derivation. For an observation with variance \(\phi V(\mu_i)/a_i\), the score contribution for \(\eta_i\) is \[ \frac{a_i}{\phi}\frac{Y_i-\mu_i}{V(\mu_i)}\frac{d\mu_i}{d\eta_i}. \] Multiplication by \(x_i\) gives its contribution to the coefficient score. On the other hand, \[ w_i(z_i-\eta_i) =\frac{a_i}{\phi}\frac{(d\mu_i/d\eta_i)^2}{V(\mu_i)} \frac{Y_i-\mu_i}{d\mu_i/d\eta_i} =\frac{a_i}{\phi}\frac{Y_i-\mu_i}{V(\mu_i)} \frac{d\mu_i}{d\eta_i}. \] Since \(\eta_i=o_i+x_i^T\beta^{(t)}\), summing these contributions gives \(X^TW(z-o-X\beta^{(t)})\). The expected information has entries obtained from the variance of the score contributions; independence makes cross-observation terms zero, and the resulting matrix is \(X^TWX\). This uses the standard likelihood regularity conditions under which the score has mean zero and expected information equals the score covariance.

Adding a Coefficient Penalty

Suppose the target is to maximize \(\ell(\beta)-\lambda P(\beta)\), where \(\lambda\geq 0\) and \(P\) is a coefficient penalty. For example, ridge uses \(P(\beta)=\tfrac12\|\beta\|_2^2\), while lasso uses \(P(\beta)=\|\beta\|_1\); an unpenalized intercept can be excluded from \(P\). The penalty convention and which coefficients are penalized must be fixed as part of the model specification.

At iteration \(t\), use the Fisher-scoring quadratic surrogate for \(\ell\), then include the penalty evaluated at the candidate coefficient vector \(b\). Up to a term independent of \(b\), maximizing that local penalized surrogate is equivalent to minimizing

The factor \(1/2\) and the definition of \(w_i\) matter: with the weights above, the quadratic part has Hessian \(X^TWX=I(\beta^{(t)})\), matching the Fisher-scoring surrogate. A penalty added on a different likelihood scale would require a correspondingly adjusted \(\lambda\).

Proof. Write \(\eta^{(t)}=o+X\beta^{(t)}\). The derivative of the weighted least-squares part of \(Q_t\) with respect to \(b\) is \(X^TW(o+Xb-z)\). At \(b=\beta^{(t)}\), this equals \(-X^TW(z-o-X\beta^{(t)})=-U(\beta^{(t)})\). Its Hessian is \(X^TWX=I(\beta^{(t)})\). Thus its quadratic expansion about \(\beta^{(t)}\), up to an additive constant, is \[ -\;U(\beta^{(t)})^T(b-\beta^{(t)}) +\frac12(b-\beta^{(t)})^TI(\beta^{(t)})(b-\beta^{(t)}). \] This is the negative of the Fisher-scoring log-likelihood surrogate, up to a constant. Adding \(\lambda P(b)\) therefore gives precisely the minimization form of the local penalized surrogate. Differentiating when \(P\) is smooth gives the stated condition; for a nonsmooth convex penalty, the gradient condition is replaced by the subgradient condition. \(\square\)

Worked Iterations

Practical Interpretation and Common Pitfalls

The calculation separates two tasks. The GLM determines the current working response and weights; the penalty determines how the coefficients are chosen within the resulting weighted least-squares problem. As the iteration proceeds, the working quantities are recalculated from the updated coefficients. This is why the overall procedure is called iteratively reweighted least squares, or IRLS.

Common mistake: treating expected information as the Taylor Hessian. The incorrect argument says that the second-order Taylor expansion of \(\ell\) has quadratic term \(-\tfrac12\delta^TI(\beta)\delta\). The Taylor term is instead \(\tfrac12\delta^TH(\beta)\delta\). The expected information gives a different matrix in general; substituting \(-I(\beta)\) defines the Fisher-scoring surrogate. Keeping these statements separate prevents an incorrect claim about the local expansion.

Common mistake: solving an unpenalized weighted least-squares problem and then applying the penalty afterward. In general, shrinking the unpenalized answer does not minimize the stated penalized subproblem, especially when predictors are correlated or a penalty is nonsmooth. The penalty belongs inside \(Q_t(b)\), because the update is the minimizer of the entire local penalized objective.

Finally, a Fisher-scoring step is based on a local approximation. Even if the quadratic subproblem is solved exactly, the actual penalized likelihood need not increase by taking the full step. When a full step is unsuitable, implementations may use step damping or a line search that checks the actual objective. Positive definiteness of the local information ensures a well-defined unpenalized quadratic direction, but by itself does not guarantee global convergence.