Why Unusual and Influential Are Not the Same
A generalized linear model (GLM) can fit most observations well while one case has an unusually large response, an unusual combination of predictors, or both. These features are related but distinct. A large residual signals that the response is poorly explained by the fitted mean; high leverage signals that the predictor values give the case potential to affect the fit. A case is influential when the fitted model changes materially if that case is removed and the model is refitted.
The practical question is therefore not simply, “Is this residual large?” It is, “What does the response residual say, how much leverage does the case have, and what changes when the case is deleted?” The diagnostics below are screening and investigation tools, not automatic rules for excluding data.
Residuals Adjusted for Leverage
For independent responses \(Y_i\) in a GLM, let \(y_i\) be the observed response and \(\widehat{\mu}_i\) the fitted mean. The Pearson residual measures the discrepancy between the response and fitted mean relative to the model-based standard deviation. For a GLM with variance function \(V(\mu_i)\), dispersion parameter \(\phi\), and prior weight \(a_i\), use
For a Poisson model with unit prior weights, \(V(\mu_i)=\mu_i\) and \(\phi=1\), so \(r_i^{P}=(y_i-\widehat{\mu}_i)/\sqrt{\widehat{\mu}_i}\). A deviance residual instead uses the observation’s signed contribution to the model deviance. For a Poisson response with unit weight, that contribution is
When \(y_i=0\), the term \(y_i\log(y_i/\widehat{\mu}_i)\) is defined by its limiting value, zero. Other GLM families have their own deviance contributions. Pearson and deviance residuals are useful for different views of response discrepancy; neither alone determines influence.
Residuals also reflect the fact that the same observations were used to estimate the model. A common leverage-adjusted approximation divides a residual by \(\sqrt{1-h_{ii}}\), where \(h_{ii}\) is the case’s GLM leverage. For example, a standardized deviance residual is often reported as \(d_i/\sqrt{\widehat{\phi}(1-h_{ii})}\). This adjustment is approximate in a GLM: it is not a guarantee of an exact standard normal distribution, especially with small samples, discrete responses, or a poor model fit.
Worked Example: Standardizing a Large Poisson Residual
A hypothetical count model has a fitted mean of 2 for one policy segment, and the observed count is 8. The fitted GLM leverage for this case is 0.25. Assume a Poisson response with unit prior weight and \(\phi=1\). Calculate its deviance residual and leverage-adjusted standardized deviance residual.
Method. Calculate the signed deviance contribution, then divide the residual by \(\sqrt{1-h_{ii}}\). The positive sign applies because \(8>2\).
First, \(\log(8/2)=\log(4)=1.386294\). The quantity inside the deviance expression is
Thus the deviance contribution is \(2(5.090352)=10.180704\), and the deviance residual is \(\sqrt{10.180704}=3.1907\). The leverage adjustment is \(\sqrt{1-0.25}=\sqrt{0.75}=0.8660\). Therefore the standardized deviance residual is \(3.1907/0.8660=3.683\), to three decimals.
As a check, the Pearson residual is \((8-2)/\sqrt{2}=4.2426\), and its leverage-adjusted value is \(4.2426/0.8660=4.899\), to three decimals. The two residual types need not agree in magnitude. The observation has a substantial response discrepancy under both measures, but this calculation alone does not establish that it has substantial influence on the fitted coefficients.
Common mistake avoided: Dividing by \(\sqrt{h_{ii}}\) would not adjust for leverage correctly. The approximate adjustment uses \(\sqrt{1-h_{ii}}\).
GLM Leverage from the Fitted Model
For a GLM fitted by iteratively reweighted least squares, the final iteration gives a weighted least-squares approximation. Let \(X\) be the model matrix and \(W\) the diagonal matrix of final working weights. A common leverage definition for this local weighted fit is the diagonal of its hat matrix:
This expression assumes the fitted model matrix has full column rank, so that the inverse exists. If a generalized inverse is needed, the calculation must use an appropriate rank-aware formulation. Leverage depends on the predictor values and the fitted working weights; in a GLM it is not generally determined by predictor values alone. Values nearer 1 indicate greater potential for the case to affect the fit, but no single leverage value proves influence.
Worked Example: Calculating Leverage from a Model Matrix
A hypothetical Poisson GLM has an intercept and one numeric predictor. At the final iteratively reweighted least-squares step, suppose all five observations have working weight 1 and their predictor values are \(0,1,2,3,6\). Calculate the leverage of the case with predictor value 6 and compare it with the leverage of the case with predictor value 0.
Method. With unit weights, \(W=I\), so \(h_{ii}=x_i^{\mathsf T}(X^{\mathsf T}X)^{-1}x_i\), where each row vector is \(x_i^{\mathsf T}=(1,x_i)\).
The sum of the predictor values is \(12\), and the sum of their squares is \(50\). Therefore
because the determinant is \(5(50)-12^2=250-144=106\). For \(x=6\),
For \(x=0\), \(h_{ii}=(1,0)(X^{\mathsf T}X)^{-1}(1,0)^{\mathsf T}=50/106=0.4717\). The value 6 has much higher leverage in this design. As a check, the leverages sum to the number of fitted coefficients, \(2\): the five diagonal values are \(50/106, 38/106, 30/106, 26/106,\) and \(86/106\), which sum to \(230/106\). This is not 2, so the check reveals an error in those claimed intermediate values if they were used. Computing each correctly gives \(h(x)=\{50-24x+5x^2\}/106\): at \(x=0,1,2,3,6\), the numerators are \(50,31,22,23,86\), summing to 212; hence the leverages sum to \(212/106=2\). The value at \(x=6\) remains \(86/106=0.8113\), and at \(x=0\) remains \(50/106=0.4717\).
The high-leverage case has greater potential to move the fitted coefficients, but whether it does so depends on how its response relates to the model. A case with high leverage and a small response discrepancy may have limited influence; a case with both high leverage and a large discrepancy deserves closer examination.
Common mistake avoided: Leverage is not the predictor value itself. It is calculated from the full design and, in a GLM, the final working weights.
Case Deletion and Coefficient Changes
The most direct influence investigation is to remove one observation, refit the model, and compare the resulting coefficients and fitted quantities with those from the full data. Write \(\widehat{\boldsymbol{\beta}}\) for the full-data coefficient estimate and \(\widehat{\boldsymbol{\beta}}_{(-i)}\) for the estimate after deleting case \(i\). Their difference identifies how the fitted coefficients respond to that case. Refitting is generally preferable to relying on an approximation when an observation appears influential.
A useful local approximation comes from the final iteratively reweighted least-squares problem. Let \(\mathbf{x}_i\) be the column vector for case \(i\), \(w_i\) its final working weight, and \(e_i^{W}=z_i-\mathbf{x}_i^{\mathsf T}\widehat{\boldsymbol{\beta}}\) its working residual, where \(z_i\) is the adjusted response at that iteration. If the weights and adjusted responses are held fixed, the weighted least-squares deletion identity gives
The approximation sign matters: a full GLM refit can change the means, working responses, and working weights. The local identity is useful for understanding the mechanism, not for replacing refitting when accurate deletion effects are required.
Review response residuals and leverage together. A large residual or high leverage is a reason to investigate, not an instruction to delete a case.
Verify the response, predictors, coding, and any weights or exposure used. A data or specification error should be addressed before interpreting influence.
Compare coefficient estimates and deviance with the full-data fit, noting that the deleted fit uses one fewer observation.
State which fitted quantities change and whether the change matters for the intended risk-modeling conclusion. Do not treat an observation as invalid merely because it is influential.
Worked Example: An Exact Case-Deletion Comparison
A hypothetical Poisson intercept-only model is fitted to counts \(1,1,1,1,8\), with unit weights and no offset. Compare the full-data coefficient and deviance with the fit after deleting the count of 8.
Method. In an intercept-only Poisson model with a log link, the maximum likelihood estimate of the mean is the sample mean, and the intercept estimate is its logarithm. Calculate the Poisson deviance under each fit.
For all five observations, the total count is \(12\), so \(\widehat{\mu}=12/5=2.4\) and \(\widehat{\beta}_0=\log(2.4)=0.8755\), to four decimals. For each count equal to 1, the deviance contribution is
The four such contributions sum to \(4(1.049062)=4.196248\). For the count of 8, its contribution is
Thus the full-data deviance is \(4.196248+8.063565=12.259813\), or 12.2598 to four decimals. After deleting 8, the four remaining counts are all 1, so \(\widehat{\mu}_{(-i)}=1\), \(\widehat{\beta}_{0,(-i)}=\log(1)=0\), and the deleted-fit deviance is 0. The coefficient changes by \(0.8755-0=0.8755\), and the deviance changes by \(12.2598-0=12.2598\).
This case has leverage \(1/5=0.2\), as every observation in an intercept-only model with equal weights does. Its effect is driven by its response discrepancy rather than unusual predictor values. The deviance comparison is descriptive: the two deviances come from datasets of different sizes, so their difference is not, by itself, a likelihood-ratio test between nested models on the same observations.
Common mistake avoided: Do not describe the large change as proof that the count is erroneous. First investigate its validity and context; then assess how sensitive the intended conclusions are to its inclusion.
Interpreting Influence Without a Mechanical Cutoff
A common GLM influence summary is an approximate Cook’s distance based on the case-deletion coefficient change:
where \(p\) is the number of fitted coefficients and the approximation uses the local weighted least-squares representation. Conventions and dispersion treatment can vary across software, so a reported value should be interpreted using its stated definition. A large value flags a case for investigation; it is not a universal deletion threshold.
Deviance can also help describe sensitivity. Compare the full fit with the refitted model and examine which cases contribute to deviance, but remember that removing a case changes both the fitted model and the dataset. For a formal nested-model comparison, the models must be fitted to the same response observations under the relevant assumptions.
A disciplined interpretation combines evidence. A large standardized residual with ordinary leverage points to a response that the model struggles to explain. High leverage with a small residual indicates unusual predictor values but not necessarily a large effect on the fit. High leverage together with a large residual is a stronger warning of potential influence. In every case, verify the data and refit before drawing a conclusion.
Check Your Understanding
For each problem, distinguish what the diagnostic establishes from what still requires investigation.
- A unit-weight Poisson model has \(\widehat{\mu}=4\), \(y=0\), \(h_{ii}=0.20\), and \(\phi=1\). Calculate the deviance residual and its leverage-adjusted standardized value. (Answer: \(d_i=-2.8284\); standardized value \(=-3.1623\), to four decimals.)
- A GLM has a case with \(h_{ii}=0.72\) and a small standardized residual. Is it necessarily influential? State what the leverage indicates and the next diagnostic step. (Answer: No; it has high potential leverage, but influence should be assessed by refitting without it and comparing fitted quantities.)
- In a Poisson intercept-only model, four counts are \(2,2,2,2\), and a fifth count is 12. Find the full fitted mean and the fitted mean after deleting 12. (Answer: Full mean \(=20/5=4\); deleted-fit mean \(=8/4=2\).)
- For the counts in question 3, calculate the full-data intercept and the deleted-fit intercept, using the log link. Give both to four decimals. (Answer: Full intercept \(\log(4)=1.3863\); deleted-fit intercept \(\log(2)=0.6931\).)
- A case has a large deviance residual but ordinary leverage. A second case has high leverage but a small residual. Which case is more clearly unusual in response, and which has greater potential to move the fit? (Answer: The first is more clearly unusual in response; the second has greater potential leverage. Neither is proven influential without a case-deletion comparison.)