Influential Observations In Logistic Regression

Why Logistic Regression Needs Its Own Influence Interpretation

In logistic regression, an observation’s influence depends on more than whether its outcome is surprising. The model’s information is weighted by $p_i(1-p_i)$, where $p_i$ is the fitted success probability. That weight is largest at $p_i=1/2$ and small near zero or one. At the same time, a case can be decisive if it is the only observation at a covariate pattern, or if removing it leaves a pattern with only successes or only failures.

These features make it important to distinguish three ideas: response discrepancy, represented by $y_i-p_i$; design leverage, which reflects the case’s covariate position relative to the weighted design; and case-deletion influence, the change in the fitted model when the case is removed. Neither a large residual nor a large leverage value alone guarantees a large coefficient change. In logistic regression, moreover, a finite full-data fit can become a boundary fit after deletion.

Logistic Leverage and a Case-Deletion Approximation

Consider independent Bernoulli responses $Y_i\in\{0,1\}$, with fixed covariate vectors $x_i\in\mathbb R^q$. The logistic model is

P(Y_i=1\mid x_i)=p_i,\qquad \log\!\left(\frac{p_i}{1-p_i}\right)=x_i^\mathsf{T}\beta .

At a finite maximum-likelihood estimate $\hat\beta$, write $\hat p_i$ for the fitted probability, $\hat w_i=\hat p_i(1-\hat p_i)$, and $r_i=y_i-\hat p_i$. The observed information for this model is $ \mathcal I=X^\mathsf{T}WX=\sum_j \hat w_jx_jx_j^\mathsf{T}$, where $W$ is diagonal with entries $\hat w_j$.

Definition: When $X^\mathsf{T}WX$ is nonsingular, the weighted GLM hat matrix is $H=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}$. The logistic leverage of case $i$ is its diagonal entry $h_i=\hat w_i x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i$. It measures the case’s weighted design leverage at the fitted model; it is not, by itself, a measure of response discrepancy.

For a full-rank design, $H$ is a projection matrix. In particular, $0\le h_i\le1$: its diagonal entries are squared lengths of projections of unit coordinate vectors. Also, $\sum_i h_i=\operatorname{tr}(H)=q$. These facts describe the weighted geometry; they do not ensure that a deletion fit exists or that a one-step approximation will be accurate.

Result (one-step case-deletion approximation): Suppose the full-data fit is finite, $X^\mathsf{T}WX$ is nonsingular, and the deleted-data fit is finite and locally close to $\hat\beta$. If $1-h_i>0$, one Newton step for the deleted score, starting at $\hat\beta$, gives $$ \hat\beta_{(-i)}-\hat\beta \ \approx\ -\frac{(X^\mathsf{T}WX)^{-1}x_i\,r_i}{1-h_i}. $$ Here $\hat\beta_{(-i)}$ denotes the deleted-data estimate. The expression is a local approximation, not an identity; it can be poor when deletion changes the fit substantially or moves it toward separation.

To derive it, the full-data score is $U(\beta)=\sum_jx_j(y_j-p_j(\beta))$, and $U(\hat\beta)=0$. The score with case $i$ deleted, evaluated at the full-data estimate, is therefore $U_{(-i)}(\hat\beta)=-x_ir_i$. The deleted-data information evaluated there is $\mathcal I_{(-i)}=\mathcal I-\hat w_ix_ix_i^\mathsf{T}$. Linearizing the deleted score around $\hat\beta$, a Newton correction $\Delta_i$ solves $-x_ir_i-\mathcal I_{(-i)}\Delta_i=0$, so $\Delta_i=-\mathcal I_{(-i)}^{-1}x_ir_i$.

The Sherman–Morrison identity gives $\mathcal I_{(-i)}^{-1}x_i=\mathcal I^{-1}x_i/(1-\hat w_ix_i^\mathsf{T}\mathcal I^{-1}x_i)$, provided the denominator is nonzero. The denominator is $1-h_i$, yielding the displayed approximation. The Taylor step is justified locally when the deleted score is differentiable near $\hat\beta$, its information is nonsingular there, and the deleted root is sufficiently close that higher-order terms can be neglected. If those local conditions fail, the formula should not be treated as a reliable substitute for an actual deletion fit. $\square$

Worked Examples: Reading Deletion Changes in Logistic Models

Worked Example: An Intercept-Only Fit and the One-Step Approximation

Suppose $n=10$ independent Bernoulli observations are fitted with an intercept only, and two outcomes are successes. Then $\hat p=2/10=0.2$, and $\hat\beta=\log(0.2/0.8)=\log(1/4)\approx-1.3863$. The information is $\mathcal I=10(0.2)(0.8)=1.6$, and each case has leverage $h_i=0.16/1.6=0.1$.

First delete a failure, for which $r_i=0-0.2=-0.2$. The one-step approximation is $\Delta_i\approx-(1/1.6)(-0.2)/(1-0.1)=0.1389$. Exactly, nine observations remain with two successes, so $\hat p_{(-i)}=2/9$ and $\hat\beta_{(-i)}=\log((2/9)/(7/9))=\log(2/7)\approx-1.2528$. The exact change is $-1.2528-(-1.3863)=0.1335$, close to the approximation.

Now delete a success, for which $r_i=1-0.2=0.8$. The approximation is $-(1/1.6)(0.8)/0.9\approx-0.5556$. Exactly, one success remains among nine cases, so $\hat\beta_{(-i)}=\log(1/8)\approx-2.0794$, giving a change of about $-0.6931$. The discrepancy illustrates why the formula is a local approximation: the success deletion moves the fitted probability farther, so higher-order effects matter more.

Worked Example: A Sparse Covariate Pattern Controls a Fitted Logit

Consider a logistic model with an intercept and a binary group indicator. Group A has four observations, two successes and two failures. Group B has two observations, one success and one failure. The model has a separate fitted probability for each group: $p_A=\operatorname{logit}^{-1}(\beta_0)$ and $p_B=\operatorname{logit}^{-1}(\beta_0+\beta_1)$.

For a group with $s$ successes among $m$ observations, its log-likelihood contribution is $s\log p+(m-s)\log(1-p)$. When $0<s<m$, setting its derivative $s/p-(m-s)/(1-p)$ to zero gives $p=s/m$. Thus both groups have fitted probability $1/2$, and the full-data estimates are $\hat\beta_0=0$ and $\hat\beta_1=0$.

The design is saturated by the two group indicators. At the fitted probabilities, every observation has weight $1/4$. Within each group, the fitted value is determined by that group’s response proportion; the diagonal leverage for an observation in a group of size $m$ is $1/m$. Hence each Group A case has leverage $1/4$, while each Group B case has leverage $1/2$.

Delete the sole Group B success. Group B then has zero successes in one observation. Its likelihood contribution is $1-p_B$, maximized at the boundary $p_B=0$. In the logistic parameterization, this requires $\beta_0+\beta_1\to-\infty$; there is no finite deleted-data maximum for that group’s logit. The full fit was finite and had a moderate fitted probability, yet deleting one case destroys the finite estimate because that response was one of only two observations defining the sparse pattern.

Worked Example: A Probability Near One and Deletion-Induced Separation

Fit an intercept-only logistic model to ten observations with nine successes and one failure. The full-data estimate is $\hat p=0.9$, $\hat\beta=\log(9)\approx2.1972$, and each case has weight $0.9(0.1)=0.09$. The total information is $10(0.09)=0.9$, so each case has leverage $h_i=0.09/0.9=0.1$.

For the one failure, $r_i=0-0.9=-0.9$. The one-step formula gives $\Delta_i\approx-(1/0.9)(-0.9)/(1-0.1)=1.1111$, a finite correction. But deleting this failure leaves nine successes and no failures. The reduced likelihood is $p^9$, whose supremum is at $p=1$; the corresponding intercept tends to $+\infty$, so there is no finite deleted-data estimate. The finite Newton correction cannot represent this boundary behavior.

How Probabilities Near Zero or One Change the Diagnosis

The logistic weight $\hat w_i=\hat p_i(1-\hat p_i)$ shrinks when a fitted probability approaches either endpoint. That can make a case’s direct contribution to the information small. It does not imply that deleting the case has little effect. A discordant outcome near an endpoint can have a substantial response discrepancy, and the rest of the data may depend on that observation to keep a covariate pattern from becoming all successes or all failures.

The denominator $1-h_i$ in the one-step expression also matters: when leverage is close to one, the local deletion correction can be amplified. But even an apparently modest leverage, as in the intercept-only example, does not rule out an infinite coefficient change if deletion creates a boundary fit. In such situations, the right interpretation is not “the approximation predicts a finite change, therefore influence is finite.” Rather, the approximation’s local assumptions have failed, and the existence and behavior of the deleted fit need direct examination.

Sparse patterns deserve particular attention. If a pattern has both outcomes but very few observations, one response may carry much of the information about the pattern’s logit. Deleting it can leave only one outcome, pushing the pattern-specific estimate toward zero or one. This is a logistic-regression mechanism for near-separation: a small change in the data can require a large coefficient change even when the full-data fit is finite.

Common Mistake

Common mistake: “The fitted probability is near one, so the weight and leverage are small; therefore the case cannot be influential.” The correction is that weight and leverage describe the fitted weighted design, not the complete deletion effect. Check the response discrepancy and the covariate pattern as well. If deleting the case leaves a pattern with only one outcome, the deleted estimate may lie at the boundary or fail to exist as a finite coefficient, regardless of whether a local approximation is modest.

A careful logistic influence assessment therefore asks: Is the case unusual in outcome relative to its fitted probability? Does it occupy a sparse or isolated covariate pattern? Does deleting it substantially change coefficients or fitted probabilities, or create separation? These questions distinguish a diagnostic warning from a demonstrated case-deletion consequence.

Check Your Understanding

Use the definitions and examples above to answer the following questions.

In logistic regression, what information does $h_i$ provide, and why is it not a complete measure of case-deletion influence?
State the one-step approximation for $\hat\beta_{(-i)}-\hat\beta$, and identify the roles of $r_i$ and $1-h_i$.
Why can deleting one observation from a sparse covariate pattern make a finite logistic estimate diverge?
In the intercept-only example with nine successes and one failure, why does deleting the failure yield no finite maximum-likelihood intercept?
Explain why small logistic weight near a fitted probability of zero or one does not, by itself, establish that the observation is unimportant.