Residual and Influence Diagnostics for Binary Models

Why Diagnose a Binary-Response Model?

A fitted logit or probit model assigns each observation a predicted probability of the outcome. Diagnostics compare those predictions with the observed zeros and ones and assess whether particular observations may have an unusually large effect on the fitted coefficients. These are related but distinct questions: a case can be poorly predicted without materially changing the model, or have substantial influence because its covariate pattern gives it high leverage.

For observation $i$, let $Y_i$ be a binary outcome, $Y_i\in\{0,1\}$, and let $x_i$ denote its vector of regressors, including an intercept if the model has one. Assume observations are conditionally independent Bernoulli variables with $P(Y_i=1\mid x_i)=p_i$. A logit model uses $p_i=\Lambda(x_i'\beta)$, where $\Lambda(t)=1/(1+\exp(-t))$; a probit model uses $p_i=\Phi(x_i'\beta)$. Here $\beta$ is estimated by maximum likelihood. Diagnostics are conditional on this specified model and do not establish that its assumptions or causal interpretation are valid.

Definition: A residual summarizes the discrepancy between an observed outcome and its fitted value. Leverage describes how unusual an observation’s regressor pattern is in the model’s weighted design. An observation is influential if its inclusion has a substantial effect on the fitted model or its conclusions.

Deviance and Pearson Residuals

For a Bernoulli observation, the raw discrepancy $Y_i-p_i$ is bounded between $-1$ and $1$, so its size alone does not account for the outcome’s conditional variance. The Pearson residual divides by the model-based conditional standard deviation. The deviance residual instead represents the signed square root of that observation’s contribution to the likelihood-ratio deviance against a saturated model, which fits each observed outcome exactly.

Definition: For an individual binary observation with fitted probability $p_i$, the Pearson residual and deviance residual are $$ r_{Pi}=\frac{Y_i-p_i}{\sqrt{p_i(1-p_i)}}, \qquad r_{Di}=\operatorname{sign}(Y_i-p_i) \sqrt{-2\left[Y_i\log(p_i)+(1-Y_i)\log(1-p_i)\right]}. $$ The sign is positive when $Y_i=1$ and negative when $Y_i=0$. These definitions assume $0<p_i<1$.

The Pearson denominator follows from $\operatorname{Var}(Y_i\mid x_i)=p_i(1-p_i)$. The deviance expression simplifies for individual binary data: when $Y_i=1$, $r_{Di}=\sqrt{-2\log(p_i)}$; when $Y_i=0$, $r_{Di}=-\sqrt{-2\log(1-p_i)}$. Thus, a surprising outcome relative to a probability near zero or one produces a larger residual. Neither residual is exactly normally distributed for an individual Bernoulli observation; their magnitudes are diagnostic signals, not exact normal-test statistics.

Worked Example: Calculating Residuals for Two Outcomes

Suppose a hypothetical fitted model assigns probability $p_i=0.80$ to an event. Compare the residuals if the event occurs and if it does not.

Event occurs, $Y_i=1$. The Pearson residual is $$ r_{Pi}=\frac{1-0.80}{\sqrt{0.80(0.20)}}= \frac{0.20}{\sqrt{0.16}}=\frac{0.20}{0.40}=0.500. $$ The deviance residual is $$ r_{Di}=\sqrt{-2\log(0.80)} =\sqrt{0.4463}=0.6680. $$ Both are positive because the observed outcome is one. The outcome is consistent with a relatively high fitted probability.

Event does not occur, $Y_i=0$. Now $$ r_{Pi}=\frac{0-0.80}{\sqrt{0.80(0.20)}}= \frac{-0.80}{0.40}=-2.000, $$ and $$ r_{Di}=-\sqrt{-2\log(1-0.80)} =-\sqrt{-2\log(0.20)} =-\sqrt{3.2189}=-1.794. $$ The negative signs indicate that the observed zero is below the model’s predicted probability. It is more discrepant than the event in the first case: the model assigned 0.80 probability to an event that did not occur.

Leverage in Logit and Probit Models

In ordinary least squares, leverage is determined by the design matrix alone. In a binary-response model, the local weighted least-squares approximation used in maximum-likelihood estimation also involves weights that depend on fitted probabilities. Let $X$ be the $n\times k$ design matrix, assumed to have full column rank. For a model with link $g$, write $p_i=G(\eta_i)$, where $\eta_i=x_i'\beta$ and $G=g^{-1}$. The Fisher-information weight for observation $i$ is $$ w_i=\frac{\left[dG(\eta_i)/d\eta_i\right]^2}{p_i(1-p_i)}. $$ For logit, $w_i=p_i(1-p_i)$. For probit, $w_i=\phi(\eta_i)^2/[p_i(1-p_i)]$, where $\phi$ is the standard normal density.

Definition: The generalized leverage matrix for the fitted binary model is the weighted projection matrix $$ H=W^{1/2}X(X'WX)^{-1}X'W^{1/2}, $$ where $W$ is diagonal with entries $w_i$. The $i$th diagonal entry $h_i$ is the generalized leverage of observation $i$. Under full rank, $0\leq h_i\leq1$ and $\sum_i h_i=k$.

Leverage is not itself a measure of a poor outcome fit: it depends on the covariates and fitted weights, not on whether $Y_i$ is zero or one. A case with high leverage can be influential if its outcome also conflicts with the pattern supported by the other observations. Conversely, a large residual at a low-leverage case need not change the estimated coefficients much.

Worked Example: Computing Weighted Leverage

Consider a hypothetical logit-model diagnostic calculation with an intercept and one regressor. At the fitted values, three observations have covariates $x=-1,0,1$ and probabilities $p=0.20,0.50,0.80$, respectively. This example calculates leverage from these specified fitted values and design; it does not require interpreting them as a complete dataset’s maximum-likelihood solution.

For logit, the weights are $w_i=p_i(1-p_i)$, so the three weights are $0.16,0.25,0.16$. With design rows $(1,-1),(1,0),(1,1)$, $$ X'WX= \begin{pmatrix} 0.57&0\\ 0&0.32 \end{pmatrix}. $$ The zero off-diagonal entries follow because the two endpoint weights are equal and their covariates sum to zero. For the observation at $x=1$, its design row is $x_i'=(1,1)$, and $$ h_i=w_i x_i'(X'WX)^{-1}x_i =0.16\left(\frac{1}{0.57}+\frac{1}{0.32}\right) =0.7807. $$ For the observation at $x=0$, $$ h_i=0.25\left(\frac{1}{0.57}\right)=0.4386. $$ The third leverage is also $0.7807$. Their sum is $2.000$, equal to $k=2$. The endpoint cases have higher leverage in this small design because their covariate values lie farther from the center.

Influence: Combining Discrepancy and Leverage

Influence concerns how much the fitted model changes when a case is removed. A direct assessment is to refit the model after deleting observation $i$ and compare the resulting coefficients or predictions with those from the full fit. This can be computationally repeated for every case. Cook’s distance offers a useful approximate screening measure based on the local weighted-least-squares representation of the fitted model.

Definition: A commonly used GLM approximation to Cook’s distance for an individual binary observation is $$ D_i\approx\frac{r_{Pi}^{\,2}h_i}{k(1-h_i)^2}, $$ where $r_{Pi}$ is its Pearson residual, $h_i$ its generalized leverage, and $k$ the number of estimated coefficients. Larger values indicate greater potential influence; the expression is a screening approximation, not an exact deletion-based change in every nonlinear model.

This formula explains why neither a large residual nor high leverage alone settles the question. Influence tends to increase when both are substantial, and it can rise sharply as $h_i$ approaches one. The approximation is most informative when the fitted model is not dominated by a few observations and the local quadratic likelihood approximation is adequate. If a case is flagged, refitting with and without it provides a more direct check.

Worked Example: Screening a High-Leverage Case

Suppose a hypothetical fitted logit model has $k=2$ coefficients. One observation has $Y_i=1$, fitted probability $p_i=0.80$, and generalized leverage $h_i=0.7807$, as in the endpoint calculation above. Its Pearson residual is $0.500$. Calculate the approximate Cook’s distance.

Substitute the values into the GLM approximation: $$ D_i\approx\frac{(0.500)^2(0.7807)} {2(1-0.7807)^2} =\frac{0.1952}{2(0.2193)^2} =\frac{0.1952}{0.09619} \approx2.029. $$ This value warrants follow-up because the observation has high leverage, even though its Pearson residual is only $0.500$. The result does not prove that the case is erroneous or that the model is invalid. A useful next step is to check its data and covariates and compare the fitted coefficients and substantive predictions with and without it.

Worked Example: A Large Residual at Lower Leverage

In a hypothetical probit-model diagnostic, an observation has $Y_i=0$, $p_i=0.80$, $h_i=0.10$, and the model estimates $k=3$ coefficients. Its Pearson residual is $$ r_{Pi}=\frac{0-0.80}{\sqrt{0.80(0.20)}}=-2.000. $$ Its approximate Cook’s distance is $$ D_i\approx\frac{(-2.000)^2(0.10)} {3(1-0.10)^2} =\frac{0.400}{2.430}=0.1646. $$ The outcome is poorly predicted relative to its fitted probability, but its lower leverage limits the approximate influence measure. This distinction matters: the case merits inspection as a discrepancy, but the calculation alone gives less evidence that it substantially determines the fitted coefficients than the high-leverage case in the preceding example.

Using Diagnostics Responsibly

A practical review combines residuals, leverage, and influence rather than relying on one statistic. The following sequence is useful:

Inspect residuals.
Find observations with large absolute deviance or Pearson residuals. Check whether the observed outcome was surprising given its fitted probability.

Check leverage.
Determine whether flagged cases have unusual covariate patterns under the model’s weighted design. Leverage alone does not indicate a bad fit.

Assess influence.
Use an approximate measure such as Cook’s distance to prioritize cases, then compare model fits with and without important candidates.

Investigate the cause.
Verify coding and data entry, consider whether the case belongs to the target population, and examine whether conclusions depend on it. Do not delete a valid observation solely because a diagnostic is large.

Rules such as flagging an absolute residual near two, or treating Cook’s distance above a particular value as noteworthy, are heuristics rather than universal decision thresholds. Individual binary residuals are discrete and need not follow a normal reference distribution. In small samples, when fitted probabilities are extreme, or when cases are dependent, simple rules are especially unreliable. The patterns and substantive context matter, and a flagged case should lead to investigation rather than automatic exclusion.

Common Mistake

A common error is to call every large residual “influential” and delete the observation. A residual measures disagreement between the observed outcome and the fitted probability; influence concerns how much the fitted model changes when the observation is removed. The correction is to examine residual size and leverage together, use an influence measure as a screen, and verify the case before changing the analysis. Diagnostics can reveal sensitivity or a possible modeling problem, but they do not by themselves identify its cause.

Key takeaway: Deviance and Pearson residuals measure case-level outcome discrepancy; generalized leverage measures covariate position in the weighted design; influence measures assess sensitivity of the fitted model. They answer different questions and should be interpreted together.

Check Your Understanding

Answer each question using the definitions and distinctions in this tutorial.

If $Y_i=0$ and $p_i=0.90$, what is the sign of both residuals, and why is the outcome surprising under the model?
Why can a case have high leverage but a small residual? What does that imply about its influence?
For logit, calculate the Pearson residual when $Y_i=1$ and $p_i=0.75$.
Why is a large approximate Cook’s distance a reason to investigate rather than automatically delete a case?
In the Cook’s-distance approximation, what happens to the influence measure as leverage approaches one, holding the residual and $k$ fixed?