Interpreting GLM Leverage Under Changing Weights

Why Leverage Depends on More Than Predictor Values

In ordinary least squares, leverage is often described as a property of the predictors: an observation far from the center of the design may have high leverage. In a generalized linear model, that description is incomplete. The local weighted least-squares problem used by iteratively reweighted least squares gives each observation a weight determined by the fitted mean, the variance function, and the link. Thus GLM leverage reflects both the design matrix and the information weights at the fitted model.

This distinction matters when interpreting a case with an extreme predictor. If its fitted information weight is small, that case may contribute little to the weighted geometry despite its unusual predictor value. But a small weight does not guarantee small leverage: the other observations’ weights and the directions represented in the design also matter.

Definition and Interpretation

Let $X$ be the $n$-by-$p$ design matrix, with row $x_i^\mathsf{T}$ for observation $i$. Assume $X$ has full column rank. At a fitted GLM, let $W$ be the diagonal matrix of positive working weights, with diagonal entries $w_i$. For a model with mean $\mu_i$, linear predictor $\eta_i$, variance function $V(\mu_i)$, and link $g(\mu_i)=\eta_i$, the usual working weight, before any observation-level prior weight, is $ (d\mu_i/d\eta_i)^2/V(\mu_i) $. An observation-level prior weight multiplies this quantity. A common scalar dispersion factor does not change the hat matrix.

Definition: The GLM working hat matrix is $$ H=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}. $$ The leverage of observation $i$ is its diagonal entry $$ h_i=H_{ii}=w_i x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i. $$ It describes the observation’s diagonal contribution in the local, weighted least-squares geometry at the fitted model.

The formula has two coupled parts. The row $x_i$ locates the observation in predictor space, while the inverse information matrix $(X^\mathsf{T}WX)^{-1}$ describes how well the weighted design as a whole supports coefficient directions. The factor $w_i$ is the observation’s own fitted information weight. Because changing a weight also changes the information matrix, leverage cannot generally be inferred by looking at $w_i$ or $x_i$ in isolation.

Worked Example: An Extreme Logistic Predictor With Small Leverage

Consider a logistic regression with intercept and one predictor, $x_i$, and fitted linear predictor $\hat\eta_i=-2+x_i$. Take three predictor values $0,1,20$. The fitted binomial information weights for individual Bernoulli observations are $w_i=\hat p_i(1-\hat p_i)$, where $\hat p_i=1/(1+\exp(-\hat\eta_i))$.

At $x=0$, $\hat\eta=-2$, so $w_1\approx0.104994$. At $x=1$, $\hat\eta=-1$, so $w_2\approx0.196612$. At $x=20$, $\hat\eta=18$, so $w_3\approx1.5230\times10^{-8}$. The last predictor is far outside the first two, but the fitted probability is nearly one and its Bernoulli variance—and hence its working weight—is very small.

For a two-column design with rows $(1,x_i)$, write $S_0=\sum_iw_i$, $S_1=\sum_iw_ix_i$, and $S_2=\sum_iw_ix_i^2$. Then $$ X^\mathsf{T}WX= \begin{pmatrix} S_0&S_1\\ S_1&S_2 \end{pmatrix}, \qquad h_i=w_i\frac{S_2-2x_iS_1+x_i^2S_0}{S_0S_2-S_1^2}. $$

Using the weights above gives $S_0\approx0.3016055$, $S_1\approx0.1966122$, $S_2\approx0.1966180$, and $S_0S_2-S_1^2\approx0.0206448$. For the extreme point $x=20$, the quadratic expression in the numerator is approximately $0.1966180-40(0.1966122)+400(0.3016055)=112.9743$. Therefore $$ h_3\approx \frac{(1.5230\times10^{-8})(112.9743)}{0.0206448} \approx0.0000833. $$

The extreme predictor has very small leverage in this fitted weighted geometry. Its predictor distance alone would not reveal that result; the near-zero fitted information weight is crucial. This calculation is local to the stated fitted model, rather than a claim that every extreme logistic predictor must have low leverage.

What the Leverage Identities Tell Us

The following properties give useful checks on leverage values and clarify their interpretation. They concern the working hat matrix at a fixed fitted model; they do not turn leverage into a measure of response discrepancy.

Theorem: If $W$ has positive diagonal entries and $X$ has full column rank, then $H$ is symmetric and idempotent, $0\leq h_i\leq1$ for every $i$, and $\sum_{i=1}^n h_i=p$.

Proof. Set $B=W^{1/2}X$. Full column rank of $X$, together with positive diagonal entries in $W$, implies that $B$ has full column rank. The hat matrix is $H=B(B^\mathsf{T}B)^{-1}B^\mathsf{T}$, so it is symmetric. Multiplying it by itself and using $B^\mathsf{T}B$ gives $$ H^2=B(B^\mathsf{T}B)^{-1}(B^\mathsf{T}B)(B^\mathsf{T}B)^{-1}B^\mathsf{T}=H. $$ Thus $H$ is an orthogonal projection matrix. For the $i$th coordinate vector $e_i$, $h_i=e_i^\mathsf{T}He_i=\|He_i\|^2$, so $h_i\geq0$. Also, $I-H$ is an orthogonal projection, and $1-h_i=e_i^\mathsf{T}(I-H)e_i=\|(I-H)e_i\|^2\geq0$. Hence $h_i\leq1$. Finally, using the trace identity for products, $$ \sum_{i=1}^n h_i=\operatorname{tr}(H) =\operatorname{tr}\{(B^\mathsf{T}B)^{-1}B^\mathsf{T}B\} =\operatorname{tr}(I_p)=p. $$ This proves all three claims. $\square$

The sum identity is a useful diagnostic check: with full rank and positive weights, the average leverage is $p/n$. It does not say that each observation has leverage near $p/n$. Weights and predictor geometry can make the individual values quite uneven.

Worked Example: Reweighting a Fixed Design

Take four observations with predictor values $0,1,2,3$, an intercept, and a slope. First use unit weights. Here $S_0=4$, $S_1=6$, $S_2=14$, and $S_0S_2-S_1^2=20$. Substituting into the leverage formula gives $$ (h_1,h_2,h_3,h_4)=(0.70,0.30,0.30,0.70). $$

Now keep the same design but change the weights to $(1,1,1,0.1)$. The weighted sums are $S_0=3.1$, $S_1=3.3$, and $S_2=5.9$, giving determinant $3.1(5.9)-3.3^2=7.4$. For $x=0$, $h=1(5.9)/7.4\approx0.7973$. For $x=1$, $h=1(5.9-6.6+3.1)/7.4\approx0.3243$. For $x=2$, $h=1(5.9-13.2+12.4)/7.4\approx0.6892$. For $x=3$, $h=0.1(5.9-19.8+27.9)/7.4\approx0.1892$.

The reweighted leverages are approximately $(0.7973,0.3243,0.6892,0.1892)$, which sum to $2$, the number of coefficients. The fourth observation’s leverage falls, but the others redistribute rather than staying fixed. In particular, the leverage at $x=2$ rises from $0.30$ to about $0.6892$. A change in one fitted weight changes the matrix used to measure every row.

Changing One Weight Versus Refitting the Model

There is a precise monotonicity statement for changing an observation’s weight while holding the design and all other weights fixed. It is useful, but it must not be confused with the effect of refitting a GLM, where fitted means and therefore multiple working weights may change together.

Result: Suppose $X$ is fixed and the weighted information matrix formed from all observations other than $i$, $A_{-i}=\sum_{j\ne i}w_jx_jx_j^\mathsf{T}$, is positive definite. If only $w_i>0$ varies, then $h_i$ is an increasing function of $w_i$.

Proof. Let $q=x_i^\mathsf{T}A_{-i}^{-1}x_i$, which is nonnegative because $A_{-i}^{-1}$ is positive definite. The full information matrix is $A_{-i}+w_ix_ix_i^\mathsf{T}$. The rank-one inverse identity gives $$ (A_{-i}+w_ix_ix_i^\mathsf{T})^{-1} =A_{-i}^{-1} -\frac{w_iA_{-i}^{-1}x_ix_i^\mathsf{T}A_{-i}^{-1}}{1+w_iq}. $$ Multiplying on both sides by $x_i^\mathsf{T}$ and $x_i$ yields $x_i^\mathsf{T}(A_{-i}+w_ix_ix_i^\mathsf{T})^{-1}x_i=q/(1+w_iq)$. Consequently, $$ h_i(w_i)=\frac{w_iq}{1+w_iq}, \qquad \frac{d h_i(w_i)}{dw_i}=\frac{q}{(1+w_iq)^2}\geq0. $$ Thus increasing only this observation’s weight cannot decrease its own leverage under the stated condition. In a GLM refit, however, changing the data or model fit can alter $X$, fitted means, and other weights, so this one-weight result does not establish monotonicity across refits. $\square$

Worked Example: Leverage in an Intercept-Only Model

For an intercept-only design, every row of $X$ is $1$, so $X^\mathsf{T}WX=\sum_jw_j$. The leverage formula reduces to $h_i=w_i/\sum_jw_j$. With weights $1,1,0.01$, the leverages are $$ h_1=\frac{1}{2.01}\approx0.4975,\qquad h_2=\frac{1}{2.01}\approx0.4975,\qquad h_3=\frac{0.01}{2.01}\approx0.0050. $$

This simplest case isolates the role of relative information weight: the third observation receives very little leverage because its weight is small compared with the total. With predictors in the model, the design directions and the inverse information matrix also affect the allocation, as in the preceding examples.

Common Mistakes and Practical Interpretation

Equating an extreme predictor with high leverage. Extreme predictor values can increase a row’s geometric reach, but a small fitted information weight can reduce its contribution. Check the fitted weights and the information matrix as well as the predictor values.
Assuming a small weight always means small leverage. Leverage depends on the entire weighted design. If an observation is important for a direction that other weighted observations barely support, it can still have substantial leverage.
Treating leverage as a response residual. Leverage is determined by the design and fitted working weights, not by whether the observed response is surprising given the fitted mean. Residuals address response discrepancy; leverage describes the local design geometry.
Assuming the one-weight result applies to a refitted model. The monotonicity derivation holds with the design and all other weights fixed. A GLM refit changes fitted means and can change many weights simultaneously.
Ignoring near-zero weights. A very small working weight can make the information matrix poorly conditioned or leave some directions weakly supported. In such cases, numerical calculations and leverage interpretations require particular care.

In practice, read a GLM leverage value as a property of the fitted, weighted design—not as a permanent label attached to an observation’s predictor values. The same design matrix can yield different leverages under different fitted means, and the same fitted weights can distribute leverage differently under a different design. This is why comparisons of leverage across fits should account for both components.

Check Your Understanding

Use the definitions and results above to answer the following questions.

In the GLM working hat matrix, where do the fitted information weights enter, and what part of the formula depends on the design matrix?
Why can an observation with an extreme predictor value have small leverage in a fitted logistic regression?
Under the assumptions of the leverage theorem, what is the sum of all $n$ leverage values?
For the one-weight monotonicity result, which quantities must remain fixed as $w_i$ changes?
Why does the one-weight monotonicity result not imply that a refitted GLM must have higher leverage for an observation whose fitted weight increased?