Why Leverage Matters in a GLM
A fitted generalized linear model (GLM) is shaped not only by how unusual an outcome is, but also by where its predictors place it in the design. An observation with an unusual predictor profile can have substantial potential to change fitted values, even if its observed outcome is close to its prediction. Leverage quantifies that potential locally, at the fitted model.
Leverage is not the same as a large residual. A residual describes the discrepancy between an observed outcome and its fitted value. Leverage describes how unusual an observation’s predictor information is relative to the other observations, taking account of the GLM’s working weights. Both help interpret a diagnostic picture; neither, by itself, proves that a case is erroneous or damaging.
The GLM Hat Matrix and Its Conditions
Let \(X\) be the \(n \times p\) design matrix, with row \(x_i^{\mathsf T}\) for observation \(i\). It includes the model’s predictor columns, such as an intercept and any encoded factors or transformations. At the converged fit, let \(W\) be the diagonal matrix of final IRLS working weights, with diagonal entries \(w_i\). For a GLM with variance function \(V(\mu_i)\), prior weight \(a_i\), and link derivative \(d\eta_i/d\mu_i\), a standard form of the working weight is \[ w_i = \frac{a_i}{V(\mu_i)}\left(\frac{d\mu_i}{d\eta_i}\right)^2 . \]
A common dispersion factor is omitted from this expression because multiplying every working weight by the same positive constant does not change the hat matrix. Offsets affect the fitted means and therefore can affect the final weights, even though an offset is not an estimated column of \(X\).
The matrix inverse in the leverage formula exists only if \(X^{\mathsf T}WX\) is nonsingular. For example, full column rank of \(X\) together with strictly positive working weights is sufficient. Full column rank of \(X\) alone is not sufficient if some weights are zero.
The symmetric matrix \(H\) expresses the projection in weighted coordinates. In the original coordinates for the working response, the corresponding local fitted-value mapping is \(X(X^{\mathsf T}WX)^{-1}X^{\mathsf T}W\); it has the same diagonal entries. Thus, leverage can be computed from the final fitted weights and design, but it is a local linear approximation for a GLM, not a complete description of what refitting after a large change would do.
Why the Formula Holds
IRLS approximates the GLM fitting problem near its current parameter estimate by a weighted least-squares problem. Let \(z\) denote the working response at convergence. The weighted least-squares estimate for this local problem minimizes \((z-X\beta)^{\mathsf T}W(z-X\beta)\). Setting its derivative with respect to \(\beta\) to zero gives the normal equations.
The normal equations are \(X^{\mathsf T}WX\hat{\beta}=X^{\mathsf T}Wz\).
If \(X^{\mathsf T}WX\) is nonsingular, then \(\hat{\beta}=(X^{\mathsf T}WX)^{-1}X^{\mathsf T}Wz\).
The local fitted values are \(X\hat{\beta}=X(X^{\mathsf T}WX)^{-1}X^{\mathsf T}Wz\). The diagonal entry for observation \(i\) is \(h_{ii}\).
The sum of the leverages is a useful check. When \(X^{\mathsf T}WX\) is nonsingular, the weighted fitted design has rank \(p\), the number of estimable columns, and \(\sum_i h_{ii}=p\). More generally, when using a rank-reduced formulation, the sum is the rank of the weighted fitted design. This identity follows because the symmetric weighted-coordinate matrix is an idempotent projection, whose trace equals its rank; its trace is also the sum of its diagonal entries.
Worked Examples
Worked Example: Leverage in a Gaussian Identity-Link Model
A hypothetical linear model has an intercept and one predictor \(x\), with observations at \(x=0,1,2,3\). The response values are \(10,11,12,30\). For a Gaussian identity-link GLM with equal working weights, calculate the four leverages and briefly compare them with the residuals.
Method. With equal weights, the intercept-and-slope leverage formula is \(h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}\), where \(S_{xx}=\sum_i(x_i-\bar{x})^2\). Here \(n=4\), \(\bar{x}=(0+1+2+3)/4=1.5\), and \(S_{xx}=2.25+0.25+0.25+2.25=5\).
For \(x=0\), \(h_{11}=0.25+2.25/5=0.70\). For \(x=1\), \(h_{22}=0.25+0.25/5=0.30\). For \(x=2\), \(h_{33}=0.25+0.25/5=0.30\). For \(x=3\), \(h_{44}=0.25+2.25/5=0.70\). The sum is \(0.70+0.30+0.30+0.70=2\), matching the two design columns.
To compare with residuals, \(\bar{y}=(10+11+12+30)/4=15.75\). The slope is \(\sum_i(x_i-\bar{x})(y_i-\bar{y})/S_{xx}=30.5/5=6.1\), and the intercept is \(15.75-6.1(1.5)=6.6\). The fitted values are \(6.6,12.7,18.8,24.9\), giving residuals \(3.4,-1.7,-6.8,5.1\).
The observation at \(x=2\) has the largest absolute residual, \(6.8\), but leverage \(0.30\); the endpoint observations have leverage \(0.70\). The endpoint at \(x=3\) has both high leverage and a residual of \(5.1\), while high leverage alone does not imply a large residual. Common mistake avoided: treating the largest residual as automatically the highest-leverage case. The leverage calculation uses predictor position, not residual magnitude.
Worked Example: Leverage in a Logistic GLM
A hypothetical logistic regression has an intercept and a continuous predictor with observed values \(x=0,1,2\). At convergence, its fitted probabilities follow the logistic model with \(\beta_0=0\) and \(\beta_1=\ln(3)\). Calculate each leverage using the final working weights.
Method. For individual Bernoulli observations with a logit link and no additional prior weights, \(w_i=\hat{p}_i(1-\hat{p}_i)\). The fitted odds at \(x=0,1,2\) are \(1,3,9\), so the fitted probabilities are \(0.5,0.75,0.9\), respectively. The weights are \(0.25,0.1875,0.09\).
For \(X\) with rows \((1,0),(1,1),(1,2)\), the entries of \(X^{\mathsf T}WX\) are \(S_0=\sum w_i=0.5275\), \(S_1=\sum w_ix_i=0.3675\), and \(S_2=\sum w_ix_i^2=0.5475\). Thus \(X^{\mathsf T}WX=\begin{pmatrix}0.5275&0.3675\\0.3675&0.5475\end{pmatrix}\). Its determinant is \(0.5275(0.5475)-0.3675^2=0.28880625-0.13505625=0.15375\), which is nonzero.
For a two-column design, \(h_{ii}=w_i(S_2-2x_iS_1+x_i^2S_0)/0.15375\). At \(x=0\), \(h_{11}=0.25(0.5475)/0.15375=0.8902\). At \(x=1\), \(h_{22}=0.1875(0.5475-0.735+0.5275)/0.15375=0.06375/0.15375=0.4146\). At \(x=2\), \(h_{33}=0.09(0.5475-1.47+2.11)/0.15375=0.106875/0.15375=0.6951\). Values are rounded to four decimals.
The leverage sum is \(0.8902+0.4146+0.6951=1.9999\), the small difference from \(2\) being rounding. Common mistake avoided: using the unweighted formula here would give \(5/6=0.8333\) at \(x=0\), instead of \(0.8902\). In a logistic GLM, leverage uses the final probability-dependent working weights; the unweighted calculation is not the correct GLM leverage.
Worked Example: Poisson Leverage with an Exposure Offset
A hypothetical Poisson rate model uses a group indicator and an offset equal to the log of exposure. The data are: group A has exposures \(1\) and \(3\), with counts \(1\) and \(7\); group B has exposures \(1\) and \(2\), with counts \(5\) and \(7\). Estimate each group’s rate, find the fitted means and working weights, and calculate leverage.
Method. The model has a separate log-rate for each group. With a log link and offset \(\log(e_i)\), the fitted mean is \(\hat{\mu}_i=e_i\hat{\lambda}_{g(i)}\), where \(e_i\) is exposure and \(\hat{\lambda}_{g(i)}\) is the fitted rate for the observation’s group. The Poisson working weight is \(w_i=\hat{\mu}_i\).
For group A, total count is \(1+7=8\) and total exposure is \(1+3=4\), so \(\hat{\lambda}_A=8/4=2\). Its fitted means are \(1(2)=2\) and \(3(2)=6\), with weights \(2\) and \(6\). For group B, total count is \(5+7=12\) and total exposure is \(1+2=3\), so \(\hat{\lambda}_B=12/3=4\). Its fitted means are \(1(4)=4\) and \(2(4)=8\), with weights \(4\) and \(8\).
Because the design has a separate indicator column for each group, leverage within each group is its observation’s weight divided by the sum of weights in that group. Group A has leverages \(2/(2+6)=0.25\) and \(6/(2+6)=0.75\). Group B has leverages \(4/(4+8)=0.3333\) and \(8/(4+8)=0.6667\). Their total is \(2\), matching the two group-rate parameters.
The raw residuals, observed count minus fitted mean, are \(-1,+1,+1,-1\). Within group A, the exposure-3 observation has higher leverage than the exposure-1 observation, although their raw residual magnitudes are equal. Common mistake avoided: treating the offset as an estimated predictor column or ignoring its effect on fitted means. The offset is fixed, but it affects the working weights through \(\hat{\mu}_i\).
Interpreting Leverage Alongside Residuals
A large \(h_{ii}\) means that observation \(i\) occupies a predictor position that gives it relatively substantial weight in the local fitted-value calculation. It does not say whether the observation is poorly fitted. That question is assessed using a suitable residual, such as a Pearson or deviance residual, interpreted in the context of the response distribution and model.
A useful diagnostic reading distinguishes two questions:
- Is the observation unusual in predictor space? Look at its leverage relative to the other observations and the model’s design.
- Is its outcome poorly fitted? Examine an appropriate residual and the surrounding diagnostic pattern.
An observation with high leverage and a small residual is well fitted at the current estimate, but its predictor position still merits attention. An observation with a large residual and ordinary leverage is poorly fitted but may have less potential to alter fitted values. A case with both high leverage and a large residual deserves particular scrutiny. These are diagnostic descriptions, not automatic rules for deleting observations.
There is no universal cutoff that determines whether leverage is “high” in every GLM. The average leverage is the rank of the weighted fitted design divided by \(n\). Rules of thumb based on multiples of that average can help flag cases for review, but they are not formal tests and depend on model size, sample size, weights, and design. Interpret a flagged value in context and alongside residual diagnostics.
Common Calculation and Interpretation Traps
- Using the ordinary least-squares hat matrix for every GLM. The GLM uses final IRLS working weights. An unweighted calculation generally differs for logistic and Poisson models.
- Assuming full rank of \(X\) guarantees the inverse. It does not if zero weights remove information. The required condition is that \(X^{\mathsf T}WX\) is nonsingular.
- Calling leverage an outlier measure. Leverage concerns predictor position and the local fitting weights; residuals describe outcome discrepancy.
- Treating a high-leverage observation as automatically influential. Leverage measures potential, not the realized change in coefficients or predictions. That requires additional influence analysis.
- Ignoring model structure. Factors, interactions, offsets, and weights can all change the design or final working weights and therefore the leverage pattern.
Check Your Understanding
For each problem, calculate or interpret leverage using the stated GLM structure. Answers are provided at the end of each item.
- A Gaussian identity-link model has an intercept and \(x=0,1,2\), with equal weights. What is the leverage at \(x=0\), and what is the sum of the three leverages? (Answer: \(0.8333\); sum \(=2\).)
- A logistic model has one Bernoulli observation with fitted probability \(0.2\) and no prior weight. What is its IRLS working weight? (Answer: \(0.1600\).)
- A two-group Poisson model with log exposure offsets has fitted means \(3,9\) in group A. What are the two leverages within group A? (Answer: \(0.2500\) and \(0.7500\).)
- A logistic fit uses a full-column-rank design matrix \(X=I_2\), but its final weights are \(W=\operatorname{diag}(1,0)\). Does full rank of \(X\) ensure the displayed inverse exists? (Answer: No; \(X^{\mathsf T}WX=\operatorname{diag}(1,0)\) is singular.)
- In a fitted GLM, one observation has high leverage and a small deviance residual, while another has ordinary leverage and a large deviance residual. Which observation is more unusual in predictor space, and which is more poorly fitted? (Answer: The high-leverage observation is more unusual in predictor space; the large-residual observation is more poorly fitted.)