Deriving The GLM Hat Matrix

From Fisher Scoring to a Hat Matrix

In ordinary least squares, the hat matrix maps the observed response to its fitted values. In a generalized linear model (GLM), a Fisher-scoring step instead fits a working response by weighted least squares. Its hat matrix is therefore local: it describes the projection for the weights and working response at the current iteration, not a single fixed projection for the entire nonlinear fitting procedure.

We derive that matrix from the Fisher-scoring update. We then express it in weighted coordinates, where it is an ordinary orthogonal projection. This gives precise meanings to its symmetry, idempotence, and diagonal entries.

Deriving the Matrix in a Fisher-Scoring Step

Suppose \(Y_1,\ldots,Y_n\) are independent GLM responses, with means \(\mu_i\), linear predictors \(\eta_i=o_i+x_i^\mathsf{T}\beta\), and fixed offsets \(o_i\). Here \(x_i^\mathsf{T}\) is row \(i\) of \(X\). Use the usual GLM variance specification \(\operatorname{Var}(Y_i)=\phi V(\mu_i)/a_i\), where \(a_i>0\) is a prior weight and \(\phi>0\) is the dispersion parameter. Write \(\mu_i=g^{-1}(\eta_i)\), and assume the link and mean functions are differentiable with \(d\mu_i/d\eta_i\ne 0\), and that \(V(\mu_i)>0\).

At the current iterate, define the working response and working weight by

Both \(z_i\) and \(w_i\) are evaluated at the current parameter value. To see why these quantities produce the scoring update, the GLM score for \(\beta\) is

The expected information is \(X^\mathsf{T}WX/\phi\). If \(X\) has full column rank, then \(X^\mathsf{T}WX\) is invertible: for every nonzero \(b\in\mathbb{R}^p\), \(b^\mathsf{T}X^\mathsf{T}WXb=(Xb)^\mathsf{T}W(Xb)>0\). Thus a full Fisher-scoring step is

Since \(\eta=o+X\beta\), this is equivalently the weighted least-squares solution

Consequently, the updated fitted linear predictor is \(o+H_W(z-o)\). With no offset, this reduces to \(H_Wz\). The matrix is called local because the weights and working response depend on the current iterate. A step-size adjustment or damping also changes the update map; the matrix derived here is for a full Fisher-scoring step with the weights held fixed during that step.

Why the Weighted-Coordinate Matrix Is an Orthogonal Projection

The weighted hat matrix \(H_W\) is generally not symmetric in the usual Euclidean coordinates. Multiplying by square roots of the weights changes coordinates to the geometry in which weighted least squares is ordinary least squares. The resulting matrix \(P_W\) is symmetric and idempotent.

Proof. Let \(A=W^{1/2}X\). Then \(A^\mathsf{T}A=X^\mathsf{T}WX\), which is invertible by the full-rank and positive-weight assumptions. In this notation, \(P_W=A(A^\mathsf{T}A)^{-1}A^\mathsf{T}\). Taking the transpose gives

because \(A^\mathsf{T}A\) is symmetric and so is its inverse. Multiplication gives

Moreover, \(P_W\) maps every vector into the column space of \(A\), and \(P_WAv=Av\) for every \(v\in\mathbb{R}^p\). These properties, together with symmetry, identify \(P_W\) as the orthogonal projection onto that column space. Finally, substituting \(A=W^{1/2}X\) shows that \(P_W=W^{1/2}H_WW^{-1/2}\), or equivalently \(H_W=W^{-1/2}P_WW^{1/2}\). Therefore \(H_W^2=W^{-1/2}P_W^2W^{1/2}=H_W\). This establishes idempotence, but does not imply that \(H_W\) is symmetric. \(\square\)

Diagonal Entries as Local Leverage

The diagonal of \(H_W\) agrees entry by entry with the diagonal of \(P_W\). In fact, writing \(x_i^\mathsf{T}\) for row \(i\) of \(X\),

These diagonal entries are the local leverage values: they measure the projection weight assigned to each observation in the current weighted working geometry. Because \(P_W\) is an orthogonal projection, its diagonal entries lie between zero and one. To verify this, let \(e_i\) be the \(i\)th standard basis vector. Symmetry and idempotence give \((P_W)_{ii}=e_i^\mathsf{T}P_We_i=\|P_We_i\|^2\). An orthogonal projection cannot increase length, so \(0\leq \|P_We_i\|^2\leq \|e_i\|^2=1\). Also, the trace of a rank-\(p\) projection is \(p\), so

Thus, the diagonal entries describe how the \(p\)-dimensional fitted subspace is represented across the \(n\) observations. They do not, by themselves, describe the size of an observation’s residual. Nor is this local working-response projection the derivative of fitted means with respect to observed responses; that is a different sensitivity concept.

Common Mistakes

• Using the unweighted formula at every iteration. The GLM working least-squares step uses \(X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W\). Equal weights up to a common factor ensure this agrees with the unweighted hat matrix, but unequal weights do not guarantee a difference in every design.
• Calling \(H_W\) symmetric. It is idempotent, but generally not symmetric in ordinary coordinates. The symmetric orthogonal projection is \(P_W\), the hat matrix in weighted coordinates.
• Treating the matrix as fixed across the whole GLM fit. Its weights are evaluated at the current iterate, so a later Fisher-scoring step can have a different local hat matrix.
• Interpreting leverage as a residual or influence measure. The diagonal describes the local projection geometry; it does not measure response discrepancy on its own.