Generalized Leverage As A Response Sensitivity

Fitted Means as a Response-Sensitivity Question

Leverage is often introduced through the diagonal of a weighted hat matrix. That diagonal describes how the fitted working responses depend on the working responses in a local weighted least-squares problem. A different, direct question is: if one observed response changes slightly, how much does each fitted mean change? The derivative answering that question is called generalized leverage.

The distinction matters because a GLM fit is usually nonlinear in the responses. The weighted least-squares representation used in Fisher scoring is a local approximation; its hat matrix need not capture every feature of the derivative of the actual fitted means. The two measures agree in important cases, but the agreement is not automatic.

Definition: For fixed design matrix $X$, let $\hat{\mu}(y)$ be the vector of fitted means as a function of the response vector $y$. When this function is differentiable at the observed data, its generalized leverage matrix is $$ G=\frac{\partial \hat{\mu}}{\partial y^\mathsf{T}}, $$ so $G_{ij}=\partial\hat{\mu}_i/\partial y_j$. The diagonal element $G_{ii}$ measures the local sensitivity of fitted mean $i$ to its own response $y_i$.

This is a local derivative, not the result of deleting an observation or making a large change to it. In models with discrete responses, the response itself takes only discrete values. The derivative is understood through the smooth score equations, extended locally as a function of response values; it describes the sensitivity of the fitted solution under that local perturbation.

Deriving the Exact Local Derivative

Consider independent observations from a GLM with fixed design rows $x_i^\mathsf{T}$, linear predictors $\eta_i=x_i^\mathsf{T}\beta$, means $\mu_i=g^{-1}(\eta_i)$, and variance $\operatorname{Var}(Y_i)=\phi V(\mu_i)/a_i$. Here $a_i$ is a known positive prior weight or precision multiplier, $\phi$ is treated as fixed, and $g$ is a differentiable link. Assume the usual exponential-family likelihood, so its regression score has the form below. Write $\mu_i'=d\mu_i/d\eta_i$, and define $$ c_i=\frac{a_i}{\phi}\frac{\mu_i'}{V(\mu_i)},\qquad C=\operatorname{diag}(c_1,\ldots,c_n),\qquad D_\mu=\operatorname{diag}(\mu_1',\ldots,\mu_n'). $$

The score equation is $U(\beta,y)=X^\mathsf{T}C(y-\mu)=0$. The weights in $C$ depend on $\beta$ through the fitted means. Consequently, differentiating the score with respect to $\beta$ includes a term involving the residuals; dropping that term is generally not exact.

Theorem: Suppose $U(\hat{\beta},y)=0$, the score is continuously differentiable in a neighborhood of $(\hat{\beta},y)$, the link and variance function are differentiable there, and the observed information $$ J=-\frac{\partial U}{\partial\beta^\mathsf{T}}\bigg|_{(\hat{\beta},y)} $$ is nonsingular. Then the local solution $\hat{\beta}(y)$ is differentiable in $y$, and the generalized leverage matrix is $$ G=D_\mu XJ^{-1}X^\mathsf{T}C, \qquad J=\sum_{i=1}^n\left(c_i\mu_i'-\frac{dc_i}{d\eta_i}(y_i-\mu_i)\right)x_ix_i^\mathsf{T}, $$ with all quantities evaluated at the fitted solution.

Proof. The implicit function theorem applies because the score is continuously differentiable and its derivative with respect to $\beta$ is $-J$, which is nonsingular. Differentiate the score equation with respect to the response vector: $$ \frac{\partial U}{\partial\beta^\mathsf{T}}\frac{\partial\hat{\beta}}{\partial y^\mathsf{T}}+\frac{\partial U}{\partial y^\mathsf{T}}=0. $$ At fixed $\beta$, $\partial U/\partial y^\mathsf{T}=X^\mathsf{T}C$. Therefore $$ -J\frac{\partial\hat{\beta}}{\partial y^\mathsf{T}}+X^\mathsf{T}C=0, \qquad \frac{\partial\hat{\beta}}{\partial y^\mathsf{T}}=J^{-1}X^\mathsf{T}C. $$ Since $\hat{\mu}_i=g^{-1}(x_i^\mathsf{T}\hat{\beta})$, the derivative of the mean vector with respect to $\beta$ is $D_\mu X$. The chain rule gives $G=D_\mu XJ^{-1}X^\mathsf{T}C$.

For the displayed form of $J$, differentiate each score contribution $c_i(y_i-\mu_i)x_i$ with respect to $\beta$. Its derivative is $$ \left(\frac{dc_i}{d\eta_i}(y_i-\mu_i)-c_i\mu_i'\right)x_ix_i^\mathsf{T}. $$ Negating and summing these derivatives gives the stated observed information. The residual term is part of the exact derivative unless $dc_i/d\eta_i=0$, or its contribution happens to vanish. $\square$

How This Relates to the Weighted Hat Matrix

The expected information replaces the residual-dependent part of $J$ by its expectation, since $E[Y_i-\mu_i]=0$. It is $X^\mathsf{T}WX$, where $$ W=\operatorname{diag}\left(\frac{a_i}{\phi}\frac{(\mu_i')^2}{V(\mu_i)}\right). $$ Using this information in place of $J$ gives the Fisher-scoring, or local working-response, approximation to response sensitivity: $$ G_F=D_\mu X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}C. $$

Result: When $X^\mathsf{T}WX$ is nonsingular and each $\mu_i'\ne0$, the diagonal of $G_F$ equals the diagonal of the weighted hat matrix $$ H_W=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}. $$ This is a statement about the Fisher-information approximation. In general, the exact generalized leverage diagonal, formed using $J$, need not equal that weighted hat-matrix diagonal.

Proof. Let $W_i=a_i(\mu_i')^2/(\phi V(\mu_i))$ be the $i$th diagonal entry of $W$. The definitions give $c_i=W_i/\mu_i'$. The $i$th diagonal element of $G_F$ is therefore $$ (G_F)_{ii}=\mu_i' x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i c_i =W_i x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i. $$ The right-hand side is $(H_W)_{ii}$, by direct multiplication of the weighted hat matrix. This proves the result. $\square$

Thus, even for a nonlinear link, the Fisher approximation has the same diagonal as weighted working-response leverage. But the exact derivative uses $J$, which contains response residuals when the link is noncanonical. The exact generalized leverage matrix and the working hat matrix are different objects; when the information matrices differ, their diagonals can differ too.

Worked Examples

Worked Example: Gaussian Regression with the Identity Link

Take a Gaussian GLM with identity link, unit prior weights, and the design rows $x_i^\mathsf{T}=(1,0),(1,1),(1,2)$. The fitted means are $X\hat{\beta}$, so ordinary least squares gives $$ \hat{\mu}=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}y. $$ Differentiating with respect to $y^\mathsf{T}$ shows that $G=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$, the ordinary hat matrix.

Here $X^\mathsf{T}X=\begin{pmatrix}3&3\\3&5\end{pmatrix}$, whose inverse is $\frac{1}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix}$. For rows $(1,0),(1,1),(1,2)$, the diagonal entries are respectively $5/6,1/3,5/6$. Because the identity-link Gaussian fit is linear in $y$, these are exact response sensitivities as well as hat-matrix leverages.

Worked Example: Logistic Regression with a Canonical Link

Suppose $Y_i$ are Bernoulli, with $x_i^\mathsf{T}=(1,0),(1,1),(1,2)$ and observed responses $y=(0,1,0)^\mathsf{T}$. The logistic link is canonical, so the score is $X^\mathsf{T}(y-\mu)$. At $\hat{\beta}=(-\log 2,0)^\mathsf{T}$, all fitted probabilities are $1/3$. The intercept score is $1-3(1/3)=0$, and the slope score is $1-(0+1+2)/3=0$.

For the canonical link, $c_i=1$, so $dc_i/d\eta_i=0$. The exact observed information is $J=X^\mathsf{T}WX$, with $W_i=\mu_i(1-\mu_i)=2/9$. The design is full rank, and the information is nonsingular. The weighted hat diagonal is $$ h_{ii}=W_i x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i. $$ Because all $W_i$ are equal, these values are the unweighted design leverages: $5/6,1/3,5/6$. The exact generalized leverage has this same diagonal. This example shows an important equality case: canonical-link score curvature does not add a residual-dependent term to $J$.

Worked Example: A Noncanonical Link Changes Exact Sensitivity

Consider a Gaussian model with variance one and the log link $\mu_i=\exp(o_i+x_i\beta)$, where $o_i$ is a fixed offset. Take $x_1=1,x_2=2$, offsets $o_1=0,o_2=\log 2$, and responses $y_1=0,y_2=9/4$. At $\beta=0$, the fitted means are $\mu_1=1,\mu_2=2$. Here $V(\mu)=1$, $\mu_i'=\mu_i$, and $c_i=\mu_i$.

The score at $\beta=0$ is $$ U=1(1)(0-1)+2(2)(9/4-2)=-1+1=0. $$ The observed information is the sum of $x_i^2\mu_i(2\mu_i-y_i)$: $$ J=1^2(1)(2-0)+2^2(2)(4-9/4)=2+14=16. $$ It is positive, so this stationary fit is locally nonsingular. The exact derivative of $\hat{\beta}$ with respect to response $y_j$ is $J^{-1}x_j\mu_j$. Applying the chain rule to $\hat{\mu}_i$ gives $$ G_{ij}=\frac{\mu_i x_i x_j\mu_j}{16}. $$ Thus the exact diagonal is $1/16,16/16$, or $1/16,1$.

For comparison, the expected information is $X^\mathsf{T}WX$, with weights $W_i=(\mu_i')^2=\mu_i^2$. It equals $1^2(1)^2+2^2(2)^2=17$. The weighted hat diagonal is consequently $1/17,16/17$. Both entries differ from the exact generalized leverage diagonal. The cause is the residual-dependent term in $J$: the observed and expected information are not the same at this fit.

Interpretation and a Common Pitfall

A generalized leverage value is a local sensitivity, and its interpretation depends on scale. $G_{ii}$ asks how fitted mean $i$ responds to a small change in response $i$; an off-diagonal entry $G_{ij}$ asks how fitted mean $i$ responds to a change in response $j$. The matrix need not be symmetric, and its diagonal is not automatically bounded by one in every nonlinear model. It should not be read as a universal probability or as a direct measure of case deletion.

Common mistake: “The weighted hat diagonal is the derivative of the fitted means with respect to the observed responses, so it is always the exact generalized leverage.” This confuses the local Fisher-scoring working problem with the full fitted-score equations. The correction is to differentiate the score equations themselves: the exact formula uses observed information $J$, including residual-dependent curvature. The weighted hat diagonal equals the derivative diagonal under the Fisher approximation, and it agrees exactly in cases such as canonical-link GLMs with fixed dispersion. Outside those settings, check the observed-information term rather than assuming equality.

Key takeaway: Generalized leverage is $\partial\hat{\mu}/\partial y^\mathsf{T}$. Its exact form is $D_\mu XJ^{-1}X^\mathsf{T}C$. Weighted hat-matrix leverage describes the Fisher-scoring working problem; nonlinear-link residual curvature can make its diagonal differ from exact response sensitivity.

Check Your Understanding

Use the score-derivative distinction to answer these questions.

In the generalized leverage matrix, what does the entry $G_{ij}$ measure?
Which term in the observed information can cause exact generalized leverage to differ from Fisher-scoring leverage?
Why does the diagonal of the Fisher-information sensitivity matrix equal the weighted hat-matrix diagonal?
For a canonical-link GLM with fixed dispersion, why does the residual-dependent term in the observed information vanish?
In the Gaussian log-link example, which information matrix produces the exact derivative, and which produces the weighted hat diagonal?