Common Errors In GLM Influence Analysis

Why Influence Diagnostics Are Easy to Misread

A case can be unusual in its predictors without changing the fitted model much, and it can change one aspect of a fit more than another. In a generalized linear model (GLM), that distinction matters because the fit involves both the design and the mean–variance relationship. An influence diagnostic is not a verdict about whether an observation is erroneous; it is evidence about how much a specified fitted quantity changes under a specified perturbation, often deletion of one case.

Let $x_i$ be the design vector for observation $i$, $\hat\beta$ the fitted coefficient vector, and $\hat\mu_i$ the fitted mean. Depending on the question, the target might be coefficients, a linear predictor, or a fitted mean at a particular covariate value. These quantities are on different scales and need not respond equally. The following distinctions help prevent a large diagnostic value from being given a meaning it does not have.

Definition: A case is influential with respect to a specified fitted quantity if perturbing or deleting that case produces a consequential change in that quantity. Influence is therefore relative to the target and the perturbation. Leverage describes a case’s position in the fitted design geometry; it is not, by itself, a measure of the change caused by deleting the case.

Error 1: Treating High Leverage as Influence

In a GLM, the usual working leverage is calculated from the design matrix $X$ and the diagonal working-weight matrix $W$ at the fitted model. If $X$ has full column rank and all working weights are positive, the weighted hat matrix is

H=W^{1/2}X(X^\top W X)^{-1}X^\top W^{1/2}, \qquad h_i=H_{ii}.

A large $h_i$ says that the observation has a position in the weighted design that can give it substantial geometric scope to affect the fit. Whether it actually does so also depends on its discrepancy from the fitted model. In a local weighted-least-squares calculation, this is visible in the case-deletion formula below.

Result (fixed-weight deletion identity): Suppose $W$ is positive diagonal, $X$ has full column rank, and $z$ is a working response. Let $\hat\beta=(X^\top W X)^{-1}X^\top Wz$, and let $\hat\beta_{-i}$ be the weighted least-squares estimate after removing row $i$, assuming the reduced design still has full column rank. Then $$ \hat\beta_{-i}-\hat\beta = -\frac{(X^\top W X)^{-1}x_i\,w_i(z_i-x_i^\top\hat\beta)} {1-h_i}. $$ This identity is exact for the fixed weighted least-squares problem. Applied to a GLM, it describes a frozen working step; it is not generally the exact change between two separately refitted GLMs.

To verify the identity, write $A=X^\top W X$, $b=X^\top Wz$, and $r_i=z_i-x_i^\top\hat\beta$. After deleting row $i$, the normal-equation matrix and right-hand side are $A_{-i}=A-w_ix_ix_i^\top$ and $b_{-i}=b-w_ix_iz_i$. Since $b-A\hat\beta=0$, the reduced normal equations evaluated at $\hat\beta$ have discrepancy

b_{-i}-A_{-i}\hat\beta=-w_ix_i r_i.

Consequently, $\hat\beta_{-i}-\hat\beta=-w_iA_{-i}^{-1}x_i r_i$. The Sherman–Morrison identity gives $A_{-i}^{-1}x_i=A^{-1}x_i/(1-w_ix_i^\top A^{-1}x_i)$, and $w_ix_i^\top A^{-1}x_i=h_i$. Substitution proves the formula. It shows why leverage alone is insufficient: the change also depends on the working residual, and on the direction $A^{-1}x_i$.

Worked Example: A High-Leverage Poisson Case With No Deletion Change

Fit a Poisson regression with an intercept and one predictor to four observations: $x=(0,1,2,100)$ and $y=(1,1,1,1)$. The fitted model has $\hat\beta_0=\hat\beta_1=0$, so every fitted mean is 1. Indeed, its score is zero because each contribution is $x_i(y_i-\hat\mu_i)=0$. The design has full rank, and the Poisson log-likelihood is strictly concave in these coefficients, so this is the unique maximum.

At this fit, the Poisson working weights are all 1. For an intercept and slope, the leverage at $x=100$ is

h_{100}=\frac14+ \frac{(100-25.75)^2}{\sum_i(x_i-25.75)^2} =\frac14+\frac{5513.0625}{7352.75} \approx 0.9998.

This is extremely high leverage. Yet the deleted data still have $x=0,1,2$ and $y=1,1,1$. The intercept-only mean of 1 with slope 0 has zero score in the reduced model as well, and the reduced design has full rank. Its unique fitted coefficients are again $(0,0)$. Thus this case has very high leverage but no coefficient change on deletion. Its large $x$-value alone does not establish influence.

Error 2: Treating a Cutoff as a Universal Verdict

Rules such as flagging leverage above $2p/n$, where $p$ is the number of fitted coefficients including the intercept, are screening conventions. They do not establish that a case is influential, invalid, or in need of removal. A threshold can also change the screening result: for $n=20$ and $p=3$, $2p/n=0.30$ and $3p/n=0.45$. A leverage of $0.35$ exceeds the first rule and not the second. That disagreement is a reason to examine the case and the analysis context, not to declare one rule a proof.

Different diagnostics also measure different things. For example, Cook’s distance summarizes a coefficient change relative to the estimated coefficient covariance, whereas leverage is a diagonal of a weighted projection matrix. A cutoff proposed for one quantity must not be applied to another merely because both are reported as diagnostic numbers.

Result (mean working leverage): Under the full-rank, positive-weight conditions above, $\sum_i h_i=p$, so the average working leverage is $p/n$. This is a geometric identity, not a threshold for influence.

For the proof, $H$ is symmetric and idempotent: direct multiplication gives $H^2=H$. Its trace is therefore its rank, which is $p$ because $X$ has full column rank and $W^{1/2}$ is nonsingular. Since the trace is the sum of the diagonal entries, $\sum_i h_i=\operatorname{tr}(H)=p$. The identity describes the overall allocation of leverage; it does not say which cases matter for a particular scientific conclusion. $\square$

Error 3: Mixing Predictor, Linear-Predictor, and Response Scales

A raw predictor value is not itself a leverage measure. Leverage depends on the predictor’s position relative to the other design rows and on the fitted working weights. Rescaling or recoding predictors can change their numerical values without changing the model space, and hence without changing ordinary design leverage under an invertible reparameterization. Conversely, the same numerical distance in a predictor can have different geometric importance in two different designs.

A second scale confusion occurs when a deletion change in the linear predictor is described as a change in the response. In a GLM, $\mu=g^{-1}(\eta)$. The response-scale change depends on the location of $\eta$, not just on the size of a change in $\eta$.

Worked Example: One Linear-Predictor Change, Different Risk Changes

For logistic regression, $\mu=\operatorname{logit}^{-1}(\eta)=1/(1+e^{-\eta})$. Consider a change $\Delta\eta=1$. At $\eta=0$, the fitted probability changes from $0.5000$ to $1/(1+e^{-1})\approx0.7311$, a change of about $0.2311$. At $\eta=4$, it changes from $1/(1+e^{-4})\approx0.9820$ to $1/(1+e^{-5})\approx0.9933$, a change of about $0.0113$.

The linear-predictor change is identical in both cases, but the probability changes differ by about a factor of 20. Therefore, report which scale a diagnostic or deletion comparison uses. If the practical question concerns predicted risks, inspect the fitted probabilities at relevant covariate values rather than treating a coefficient or $\eta$ change as though it were itself a risk change.

Error 4: Ignoring GLM Working Weights

GLM leverage is weighted leverage. The weights reflect the local information in the fitted model, not simply the numerical size of a predictor or the number of rows in a data file. For instance, grouped binomial observations have trial counts that affect their working weights. Treating each group as equally informative can give the wrong leverage and influence interpretation.

Worked Example: A Large Grouped-Binomial Leverage With No Coefficient Change

Consider an intercept-only logistic model with two grouped observations: 1 success out of 2 trials, and 9 successes out of 18 trials. The pooled success proportion is $10/20=0.5$, so the fitted intercept is $\hat\beta_0=\operatorname{logit}(0.5)=0$. For an intercept-only grouped-binomial model, the working weight at fitted probability $0.5$ is $m_i(0.5)(0.5)=m_i/4$, where $m_i$ is the trial count. The weights are $0.5$ and $4.5$, so the leverages are

h_1=\frac{0.5}{0.5+4.5}=0.1, \qquad h_2=\frac{4.5}{0.5+4.5}=0.9.

The second group has leverage $0.9$, in part because it contains most of the trials. But each group’s observed proportion is exactly $0.5$, the fitted probability. Removing either group leaves the other group’s observed proportion at $0.5$, so the refitted intercept remains 0. This example reinforces two points: the grouped trial counts matter to leverage, and high leverage alone does not guarantee a deletion change.

Error 5: Trusting a Deletion Approximation Without Refitting

A one-step deletion approximation holds the working weights and working response fixed, or linearizes the likelihood near the full-data fit. A true deletion analysis changes the data and then refits the model. The weights, fitted means, and curvature may change; the reduced fit may also fail to converge, lose rank, or have no finite maximum-likelihood estimate. An approximation cannot establish that the refit exists or behaves regularly.

Worked Example: Deletion Creates Separation in Logistic Regression

Suppose four binary logistic-regression observations have predictor values and outcomes $(x,y)=(-1,0),(0,1),(1,1),(2,0)$, with an intercept and a slope. In the full data, the positive outcomes occur at 0 and 1, while the negative outcomes occur at -1 and 2. No affine function of $x$ can place both positive points on one side of a threshold and both negative points on the other: the convex hulls of the positive and negative predictor values, $[0,1]$ and $[-1,2]$, overlap. Thus there is no separating direction, and the full logistic model has a finite maximum-likelihood estimate.

Now delete the observation $(2,0)$. The remaining data are $(-1,0),(0,1),(1,1)$. The affine predictor $\eta=x+0.5$ is negative at $x=-1$ and positive at $x=0,1$, so the reduced data are completely separated. Increasing the slope and adjusting the intercept along a separating direction can increase the likelihood toward its supremum without attaining it at finite coefficients. The reduced model therefore has no finite maximum-likelihood estimate.

In this case, a numerical deletion approximation around the finite full-data fit cannot be interpreted as a reliable finite coefficient change for the reduced fit. The exact refit reveals a property the approximation cannot settle: the reduced estimate does not exist as a finite vector.

A Practical Interpretation Sequence

Influence analysis is most useful when each number is connected to a clearly stated question. A large value is a prompt for investigation, not an automatic instruction to discard a case. Before drawing a conclusion, identify the target of the diagnostic, its scale, and whether the reported change comes from a local approximation or an actual refit.

Name the target.
Specify whether you are assessing coefficients, a linear predictor, fitted means, or predictions at particular covariate values.

Check the geometry and weights.
Interpret leverage relative to the design and the GLM working weights; do not substitute raw predictor size for either.

Use cutoffs as screens.
Record the rule used, but do not treat a threshold crossing as a universal definition of influence or an error in the data.

Refit and compare on the relevant scale.
Check convergence, estimability, and the exact deletion change before interpreting a one-step approximation as a model-level result.

A common mistake is to reason, “This case has high leverage, so it must be influential; its diagnostic exceeds the cutoff, so it should be removed.” The correction is to ask what quantity changed, by how much, and under what perturbation; check whether the chosen diagnostic and scale answer that question; then refit when an exact deletion comparison is intended. Any decision to exclude an observation also requires a substantive justification, not a diagnostic threshold alone.

Key takeaway: Leverage measures geometric opportunity for influence, not influence itself. GLM weights, residual discrepancy, response scale, and the behavior of the exact reduced-data fit all matter. No single cutoff replaces those checks.

Check Your Understanding

For each question, explain the reasoning rather than relying on a cutoff alone.

Why does a high diagonal working leverage not, by itself, imply a large change in fitted coefficients after deletion?
For $n=30$ and $p=4$, calculate the average working leverage under the full-rank, positive-weight conditions. Does that value determine an influence cutoff?
In logistic regression, why can the same change in linear predictor correspond to very different changes in fitted probabilities?
How can grouped-binomial trial counts affect leverage even when two groups have the same observed success proportion?
What should be checked when a one-step deletion approximation is large or when the reduced model may be separated?