Separating Design Leverage From Response Discrepancy

Two Different Questions in a Diagnostic

When an observation looks unusual, two questions should be kept separate. Is its predictor configuration unusual relative to the design? And does its response disagree with the model’s fitted pattern? Leverage addresses the first question; residuals address the second. They describe different features of a case, so neither alone establishes that removing or changing it would substantially alter a fitted model.

In a generalized linear model (GLM), this distinction is usually examined through the weighted least-squares problem underlying an iteratively reweighted least-squares (IRLS) fit. At a given iteration, the design matrix and working weights define a local weighted regression. Leverage is a property of that weighted design; a working residual measures disagreement on the working-response scale. Response, Pearson, and deviance residuals provide other discrepancy summaries, but they are not interchangeable with the working residual.

The word local matters. In a GLM, the weights depend on the fitted model and can change during fitting; leverage is therefore tied to the current weighted design, not just to the raw predictor values. The formula provides a diagnostic summary at that fit or iteration, not a universal measure of how influential the observation must be.

What Leverage Measures

To see the design component, transform the weighted problem using \(W^{1/2}\). Its whitened design matrix is \(W^{1/2}X\), and the projection matrix onto the columns of that design is \[ H=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}. \] The diagonal entry \(H_{ii}\) equals \(h_i\). This representation gives useful facts about leverage without making it a measure of response disagreement.

Proof. The matrix \(H\) is symmetric. If \(A=W^{1/2}X\), then \(H=A(A^\mathsf{T}A)^{-1}A^\mathsf{T}\), so \[ H^2=A(A^\mathsf{T}A)^{-1}(A^\mathsf{T}A)(A^\mathsf{T}A)^{-1}A^\mathsf{T}=H. \] Thus \(H\) is a symmetric projection matrix. For the \(i\)th coordinate vector \(\mathbf{e}_i\), \[ h_i=\mathbf{e}_i^\mathsf{T}H\mathbf{e}_i =\mathbf{e}_i^\mathsf{T}H^\mathsf{T}H\mathbf{e}_i =\|H\mathbf{e}_i\|^2\geq 0. \] An orthogonal projection cannot increase length, so \(\|H\mathbf{e}_i\|^2\leq\|\mathbf{e}_i\|^2=1\). Finally, the trace of a projection onto a \(p\)-dimensional column space is \(p\), and trace is the sum of diagonal entries. Hence \(\sum_i h_i=\operatorname{tr}(H)=p\). \(\square\)

The sum identity is a useful check on calculated leverages. It also gives a reference point: their average is \(p/n\). But an observation with leverage above that average is not automatically problematic. Leverage says that the design gives the case a particular geometric position; it says nothing by itself about whether its response agrees with the fitted pattern.

Working-Response Perturbations and Leverage

Leverage also describes how a change in one working response is reflected in the fitted working responses, with the weights and design held fixed. It is important to distinguish the fitted working response on its original scale from the fitted whitened response.

Proof. Minimizing \((\mathbf{z}-X\boldsymbol{\beta})^\mathsf{T}W(\mathbf{z}-X\boldsymbol{\beta})\) gives the normal equations \(X^\mathsf{T}WX\hat{\boldsymbol{\beta}}=X^\mathsf{T}W\mathbf{z}\). Since \(X^\mathsf{T}WX\) is invertible, \(\hat{\boldsymbol{\beta}}=(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W\mathbf{z}\). Multiplying by \(X\) gives \(\hat{\mathbf{z}}=P\mathbf{z}\). Changing only \(z_i\) adds \(\delta\mathbf{e}_i\) to \(\mathbf{z}\), so the fitted vector changes by \(P\delta\mathbf{e}_i\), whose \(j\)th component is \(P_{ji}\delta\). On the diagonal, \[ P_{ii}=\mathbf{x}_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}\mathbf{x}_i\,w_i=h_i. \] For comparison, whitening gives \(W^{1/2}\hat{\mathbf{z}}=H W^{1/2}\mathbf{z}\), so \(H\) acts on the whitened working response, not directly on \(\mathbf{z}\). \(\square\)

This result describes a fixed local weighted least-squares problem. It does not say that a case with high leverage must have a large residual, or that a response perturbation of a given size will have the same effect on every coefficient or fitted response. It is one reason to keep leverage and discrepancy conceptually distinct.

Worked Examples

How to Read the Pair

A practical diagnostic interpretation starts by asking what each quantity measures and on which scale it is computed. High leverage indicates that the observation’s predictor configuration is unusual relative to the current weighted design. A large residual indicates disagreement, but the meaning of “large” depends on the residual definition: a raw response residual, Pearson residual, deviance residual, and working residual use different scales and constructions. Use a residual appropriate to the diagnostic question, and do not treat their numerical magnitudes as interchangeable.

The combination is informative but not a mechanical verdict. High leverage together with a substantial residual is a reason to inspect the observation and the model carefully. High leverage with little discrepancy may have little immediate evidence of tension with the fitted pattern. A large residual with modest leverage signals a response mismatch without, on its own, establishing that the overall fit changes greatly. The detailed relation between these quantities and case-deletion changes requires additional analysis; neither diagnostic summary is itself a case-deletion calculation.

Common Mistake

Wrong reasoning: “This case has high leverage, so it must strongly affect the fit,” or, conversely, “Its residual is large, so it must be influential.” Both claims jump from one diagnostic dimension to a conclusion about fit change. High leverage describes predictor geometry, while residual size describes response disagreement. As the worked examples show, either can occur without the other, and a large raw residual can coexist with a tiny weighted contribution. Treat them as complementary clues, not as guarantees.