GLM Leverage and the Hat Matrix

Why Leverage Matters in a GLM

A generalized linear model (GLM) estimates its mean structure from both the predictors and the response distribution. Two observations with different predictor values can therefore have different potential to affect the fitted mean pattern, even if their observed responses are unremarkable. Leverage measures this potential in predictor space, accounting for the weights used by the fitted GLM.

The diagonal elements of the GLM hat matrix provide observation-specific leverage values. They are related to the hat values from linear regression, but GLM leverage is local: it is calculated using the fitted working weights, which depend on the fitted means. It is not a residual, nor does a high value by itself establish that an observation is influential.

Definition: In a GLM with design matrix $X$, full column rank $p$, and fitted working-weight matrix $W$, the weighted hat matrix is $$ H=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}. $$ The leverage of observation $i$ is the diagonal element $h_{ii}$. Equivalently, $$ h_{ii}=w_i x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i, $$ where $x_i^\mathsf{T}$ is row $i$ of $X$, and $w_i$ is its fitted working weight.

Working Weights and the Hat Matrix

For independent observations in a GLM, let $\mu_i=E[Y_i]$, let $g(\mu_i)=\eta_i$, and write the linear predictor as $\eta_i=x_i^\mathsf{T}\beta+\text{offset}_i$. The variance is commonly written $\operatorname{Var}(Y_i)=\phi V(\mu_i)/a_i$, where $a_i$ is a prior weight and $\phi$ is a dispersion parameter. In iteratively reweighted least squares, the working weight is proportional to $$ w_i=a_i\frac{\left(d\mu_i/d\eta_i\right)^2}{V(\mu_i)}. $$ If the model has a common dispersion parameter, including the factor $1/\phi$ in every weight does not change the hat matrix: multiplying all weights by the same positive constant cancels between the matrix and its inverse. The formula assumes that $X^\mathsf{T}WX$ is invertible; if the design is rank deficient, the model must instead be expressed using its estimable rank and an appropriate generalized inverse.

For a Poisson model with a log link and unit prior weights, $V(\mu_i)=\mu_i$ and $d\mu_i/d\eta_i=\mu_i$, so $w_i=\mu_i$. For a Bernoulli logistic model, $V(\mu_i)=\mu_i(1-\mu_i)$ and $d\mu_i/d\eta_i=\mu_i(1-\mu_i)$, giving $w_i=\mu_i(1-\mu_i)$. For grouped binomial data with $n_i$ trials, the corresponding weight is $n_i\mu_i(1-\mu_i)$. These weights describe how strongly each observation contributes to the local weighted fit.

Form the fitted design and weights.
Use the model’s predictor rows $x_i^\mathsf{T}$ and calculate each working weight from its fitted mean and variance function.

Calculate the weighted information matrix.
Compute $X^\mathsf{T}WX$, where $W$ is diagonal with entries $w_i$.

Evaluate each diagonal.
Use $h_{ii}=w_i x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i$. Compare the values in context, then check their bounds and sum.

Why the Diagonal Measures Leverage

The hat matrix arises from the local weighted least-squares problem used by iteratively reweighted least squares. At a given fitted model, let $z$ denote the working response. The weighted least-squares coefficient estimate for that local problem is $(X^\mathsf{T}WX)^{-1}X^\mathsf{T}Wz$. Thus its fitted values on the linear-predictor scale are obtained by applying $X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W$ to $z$.

To see why the weighted hat matrix is used, multiply the working response by $W^{1/2}$. The resulting fitted values in weighted coordinates are $H W^{1/2}z$, with $H$ as defined above. This matrix is a symmetric projection onto the column space of $W^{1/2}X$: it is symmetric, and multiplying it by itself gives $H$. Its diagonal elements quantify how strongly each weighted coordinate participates in that projection. Returning to the unweighted working-response representation gives the same diagonal values.

This derivation is local to the fitted means because changing the model changes $W$. It explains why GLM leverage depends on more than the raw predictor values. An offset can also change fitted means and therefore working weights, even though the offset is not an estimated column of $X$.

Theorem: With positive working weights and a full-rank design with $p$ columns, $0\le h_{ii}\le 1$ for every observation and $\sum_i h_{ii}=p$. More generally, the sum is the rank of the weighted design. These are useful arithmetic checks; they do not make a particular leverage value universally “high.”

Worked Examples

Worked Example: Poisson Intercept With Exposure Offsets

A hypothetical portfolio model uses a Poisson distribution with a log link. Four cells have claim counts $y=(1,1,4,4)$ and exposures $e=(1,2,3,4)$. The model is $\log(\mu_i)=\beta_0+\log(e_i)$, with unit prior weights. Calculate the fitted means and leverage values. The offset is specified explicitly: this is not a Poisson intercept-only model without offsets.

Method. The offset model has one estimated coefficient, the log claim rate. Its likelihood estimate is the total count divided by total exposure: $$ \hat{\lambda}=\frac{\sum_i y_i}{\sum_i e_i} =\frac{1+1+4+4}{1+2+3+4} =\frac{10}{10}=1. $$

The fitted means are $\hat{\mu}_i=e_i\hat{\lambda}$, so they are $1,2,3,4$. For a Poisson log-link model, $w_i=\hat{\mu}_i$. With an intercept-only design column of ones, the leverage reduces to $h_{ii}=w_i/\sum_j w_j$. Therefore: $$ (h_{11},h_{22},h_{33},h_{44}) =\left(\frac{1}{10},\frac{2}{10},\frac{3}{10},\frac{4}{10}\right) =(0.1000,0.2000,0.3000,0.4000). $$

Check. All values are between zero and one, and they sum to $1$, the number of estimated coefficients. The larger-exposure cells have more leverage in this offset model because their fitted Poisson means, and hence their working weights, are larger.

Mistake avoided. Assigning $0.2500$ to each cell would be incorrect here. Equal leverage applies to a Poisson intercept-only model without offsets, where all fitted means are equal. With the stated offsets, fitted means differ and leverage is proportional to exposure. In an intercept-only Poisson model without offsets and with four observations, each leverage would indeed be $1/4=0.2500$.

Worked Example: Poisson Leverage Across Predictor Values

A hypothetical count model has a Poisson response, a log link, and predictor values $x=(0,1,2)$, with design rows $x_i^\mathsf{T}=(1,x_i)$. Suppose the observed counts are $y=(1,2,4)$. Calculate the fitted means and leverages.

Method. The model $\log(\mu_i)=\beta_0+\beta_1x_i$ fits these three counts exactly at $\hat{\beta}_0=0$ and $\hat{\beta}_1=\log 2$, giving $\hat{\mu}=(1,2,4)$. The score equations are satisfied because each fitted mean equals its observation. The working weights are consequently $w=(1,2,4)$.

For a two-column design, write $S_0=\sum_i w_i$, $S_1=\sum_i w_ix_i$, and $S_2=\sum_i w_ix_i^2$. Then $X^\mathsf{T}WX$ is $\begin{pmatrix}S_0&S_1\\S_1&S_2\end{pmatrix}$. Here $S_0=7$, $S_1=10$, $S_2=18$, and its determinant is $7(18)-10^2=26$. For each $x_i$, the diagonal formula simplifies to $$ h_{ii}=\frac{w_i(S_2-2x_iS_1+x_i^2S_0)}{S_0S_2-S_1^2}. $$

At $x=0$, $h_{11}=1(18)/26=9/13=0.6923$. At $x=1$, $h_{22}=2(18-20+7)/26=10/26=5/13=0.3846$. At $x=2$, $h_{33}=4(18-40+28)/26=24/26=12/13=0.9231$.

Check. The leverages sum to $9/13+5/13+12/13=2$, the number of model coefficients. The point at $x=2$ has the greatest leverage: it is at the edge of the observed predictor range and has the largest working weight. The calculation avoids treating distance in $x$ as the only determinant of leverage.

Worked Example: Grouped-Binomial Logistic Leverage

A hypothetical grouped-binomial logistic model records 100 trials at each predictor value $x=0,1,2$, with 20, 50, and 80 successes, respectively. The model includes an intercept and a linear predictor $x$. Find the leverage for each grouped observation.

Method. The fitted probabilities $0.2,0.5,0.8$ correspond to logits $-\log 4,0,\log 4$, which lie on a straight line in $x$. The fitted probabilities equal the observed proportions, so the grouped-binomial score equations are satisfied. The working weights are $w_i=n_i\hat{\mu}_i(1-\hat{\mu}_i)$, giving $16,25,16$.

With design rows $(1,0),(1,1),(1,2)$, the weighted sums are $S_0=57$, $S_1=57$, and $S_2=89$. The determinant is $57(89)-57^2=1824$. Applying the same two-column leverage formula: $$ h_{11}=\frac{16(89)}{1824}=\frac{89}{114}=0.7807,\qquad h_{22}=\frac{25(89-114+57)}{1824}=\frac{25}{57}=0.4386, $$ $$ h_{33}=\frac{16(89-228+228)}{1824}=\frac{89}{114}=0.7807. $$

Check. The values sum to $2$, as required for the intercept and slope. The center predictor value has lower leverage than either endpoint. This comparison reflects both the predictor positions and the logistic working weights; the middle group’s weight is larger, but its predictor position is less extreme.

Interpretation and Common Traps

A high $h_{ii}$ indicates that the fitted local predictor structure gives observation $i$ relatively strong potential to affect its own fitted value. It does not say whether the response is unusual. Residual diagnostics address discrepancies between observed and fitted responses; leverage describes the design-and-weight geometry. An observation can have high leverage and a small residual, or low leverage and a large residual.

Leverage should be interpreted relative to the number of observations, the number of fitted coefficients, the predictor patterns, and the working weights. The identity $\sum_i h_{ii}=p$ gives an average leverage of $p/n$ when there are $n$ observations, but this average is not a universal cutoff for flagging an observation. In particular, repeated predictor patterns, offsets, and fitted probabilities near zero or one can make GLM working weights vary substantially.

Using the ordinary unweighted hat matrix. A GLM generally requires fitted working weights. Omitting them can give the wrong leverage ranking and values.
Confusing offsets with estimated predictors. An offset is fixed and is not an additional coefficient, but it can alter fitted means and therefore alter the weights.
Calling leverage an outlier measure. A response residual and a predictor-space leverage value answer different questions.
Ignoring basic checks. A value outside $[0,1]$, or a sum different from the design rank, signals a calculation or setup error under the stated conditions.

Careful use of the hat matrix is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus. The practical sequence is to identify the design and fitted weights, calculate the diagonal elements, and interpret them separately from residual size.

Check Your Understanding

For each problem, calculate the requested leverage values and use the relevant sum or bounds as a check.

A Poisson log-link model without offsets has only an intercept and four observations. What is each leverage? (Answer: $0.2500$ for each observation.)
A Poisson log-link model has an intercept and an offset $\log(e_i)$, with exposures $e=(2,3,5)$, unit prior weights, and no other predictors. Give the three leverage values. (Answer: $0.2000,\ 0.3000,\ 0.5000$.)
A Poisson model has design rows $(1,x_i)$ for $x=(0,1,2)$, with fitted means $(2,2,2)$. Find the three leverages. (Answer: $0.8333,\ 0.3333,\ 0.8333$.)
For a grouped-binomial logistic model with one intercept and one binary predictor, there are two grouped observations at $x=0$ and one at $x=1$. Suppose each group has the same number of trials and all fitted probabilities are $0.4$. What are the leverages, in order? (Answer: $0.5000,\ 0.5000,\ 1.0000$.)
A Poisson model has rows $(1,x_i)$ at $x=(0,1,2)$ and working weights $w=(1,1,2)$. Use the weighted sums to calculate the leverages. (Answer: $9/11,\ 3/11,\ 10/11$, or $0.8182,\ 0.2727,\ 0.9091$ to four decimals.)

What You'll Learn