From a Logistic Fit to a Case-Deletion Question
Influence analysis asks how much a fitted model depends on particular observations. A large residual alone does not answer that question: an observation can disagree with its fitted value but have little effect on the coefficients, while a case with a more modest residual can matter because of its position in the design. The practical analysis therefore links the design, the fitted model, diagnostics, and a targeted refit.
Consider a small study with a binary outcome and a predictor having three levels, coded \(x=0,1,2\). We use level 0 as the reference and fit a logistic regression with separate indicators for levels 1 and 2. There are ten observations at each level. The numbers of outcomes equal to 1 are respectively 2, 5, and 8. The table gives the grouped counts; each grouped row represents ten independent Bernoulli observations with the same predictor value.
| Predictor level | Number of observations | Number with outcome 1 | Number with outcome 0 |
|---|---|---|---|
| 0 (reference) | 10 | 2 | 8 |
| 1 | 10 | 5 | 5 |
| 2 | 10 | 8 | 2 |
The design is not a numeric linear trend in \(x\): the model allows each non-reference level to have its own log-odds contrast. For an individual observation \(i\), let \(Y_i\) be Bernoulli with success probability \(p_i\), and write \(z_{i1}=1\) when \(x_i=1\) and 0 otherwise, and \(z_{i2}=1\) when \(x_i=2\) and 0 otherwise. The fitted model is
Worked Example: Inspecting the Design and Fitting the Model
Worked Example: Inspecting the Design and Fitting the Model
The three distinct design rows are \((1,0,0)\), \((1,1,0)\), and \((1,0,1)\), each repeated ten times. They are linearly independent, so the design matrix has full column rank three. The response proportions are \(2/10=0.2\), \(5/10=0.5\), and \(8/10=0.8\). All are strictly between zero and one, so each cell has a finite estimated logit.
For a level with \(y\) successes in \(n\) trials, its binomial log-likelihood, apart from a term not involving its probability \(p\), is \(\ell(p)=y\log p+(n-y)\log(1-p)\). Differentiating gives \(\ell'(p)=y/p-(n-y)/(1-p)\), which is zero at \(p=y/n\). The second derivative is negative for \(0<p<1\), so this is the maximum. Thus the fitted risks are \(0.2,0.5,0.8\), and their logits are \(-\log 4,0,\log 4\).
Matching these three logits to the design rows gives \(\hat\beta_0=-\log4=-1.3863\), \(\hat\beta_1=\log4=1.3863\), and \(\hat\beta_2=2\log4=\log16=2.7726\). The odds at level 1 are four times the odds at level 0; at level 2 they are sixteen times the odds at level 0. This is a categorical-level model, so the two contrasts need not be equal.
The likelihood factors into one term for each category because each observation belongs to exactly one category. In category \(g\), the log-likelihood contribution, ignoring a binomial coefficient, is \(y_g\log p_g+(n_g-y_g)\log(1-p_g)\). Its derivative is \(y_g/p_g-(n_g-y_g)/(1-p_g)\), and setting this to zero gives \(y_g(1-p_g)=(n_g-y_g)p_g\), hence \(p_g=y_g/n_g\). The second derivative is \(-y_g/p_g^2-(n_g-y_g)/(1-p_g)^2<0\) for \(0<p_g<1\), establishing a maximum in each category. The indicator parameterization represents these category logits uniquely, so these separate maxima are also the model’s maximum-likelihood fit. \(\square\)
Leverage in This Weighted Design
For logistic regression, the usual local weighted hat matrix is \(H=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}\), where \(X\) is the design matrix and \(W\) is diagonal with entries \(\hat p_i(1-\hat p_i)\). The diagonal \(h_i\) measures a case’s leverage in this weighted design. In the current model, the three indicator design columns separate the categories. The total information for category \(g\) is \(n_g\hat p_g(1-\hat p_g)\), and each case in that category contributes \(\hat p_g(1-\hat p_g)\).
To verify the result, order the rows by category. Then \(X^\mathsf{T}WX\) is diagonal, with category-\(g\) diagonal entry \(n_g w_g\), where \(w_g=\hat p_g(1-\hat p_g)>0\). For an observation in category \(g\), its design vector selects that category’s diagonal entry. Therefore its diagonal hat value is \(w_g/(n_gw_g)=1/n_g\). The positive weight condition permits the inverse, and the stated full-rank model has no other design columns that mix categories. \(\square\)
Here every category has ten observations, so every individual has \(h_i=0.1\), even though the fitted probabilities differ. This is a feature of the saturated category design: the category’s common weight appears in both the individual contribution and the total category information, and cancels in the ratio. It does not mean the three categories contain equal information overall; their total information values differ.
Worked Example: Reading Residuals and Leverage Together
Worked Example: Reading Residuals and Leverage Together
Focus on one observation with \(x=2\) and \(Y=1\). Its fitted probability is \(0.8\), so its raw response residual is \(Y-\hat p=1-0.8=0.2\). Its Bernoulli variance under the fitted model is \(0.8(1-0.8)=0.16\), giving Pearson residual \(0.2/\sqrt{0.16}=0.5\). Its deviance residual is positive and equals \(\sqrt{2\log(1/0.8)}=\sqrt{0.4463}\approx0.668\).
Its leverage is \(0.1\), the same as every other individual in the study. Thus this case is not distinguished by unusually high leverage. Its residual is also not extreme in this small illustration. Still, deletion changes the level-2 success proportion, and hence changes its fitted risk and the corresponding coefficient. Diagnostics help flag cases for examination; the refit determines what actually changes.
Worked Example: Exact Deletion and Its Consequences
Worked Example: Exact Deletion and Its Consequences
Delete the selected successful observation at level 2. That category now has 7 successes among 9 observations, while the other two categories are unchanged. Applying the category-wise likelihood calculation gives \(\hat p_{2(-i)}=7/9\), with odds \((7/9)/(2/9)=7/2\). The reference-category odds remain \(2/8=1/4\), so the exact refitted coefficients are \(\hat\beta_{0(-i)}=-\log4\), \(\hat\beta_{1(-i)}=\log4\), and \(\hat\beta_{2(-i)}=\log\{(7/2)/(1/4)\}=\log14=2.6391\).
The coefficient change is therefore \(\hat\beta_{(-i)}-\hat\beta=(0,0,\log(7/8))^\mathsf{T}\), since \(\log14-\log16=\log(7/8)=-0.1335\). The fitted risk at level 2 falls from \(0.8\) to \(7/9\approx0.7778\); the risks at levels 0 and 1 remain \(0.2\) and \(0.5\). The level-2 odds ratio against level 0 falls from 16 to 14. The direction and broad interpretation of the contrast remain the same, although its estimated magnitude is smaller.
For comparison, a one-step deletion approximation uses the score and information at the full fit. The score contribution of the deleted observation is \(x_i(Y_i-\hat p_i)\), where \(x_i=(1,0,1)^\mathsf{T}\); the level-2 component is \(0.2\). At the full fit, the information matrix is diagonal with entries \(1.6,2.5,1.6\), because the category totals are \(10(0.16),10(0.25),10(0.16)\). Removing the case from the information gives level-2 information \(9(0.16)=1.44\). The first Newton update for the reduced fit is consequently \(\Delta\beta_2\approx-0.2/1.44=-0.1389\), with the other two changes zero. This gives the approximate coefficients \((-1.3863,1.3863,2.6337)\), close to but not identical to the exact \((-1.3863,1.3863,2.6391)\).
One simple coefficient-change summary, using the full-fit information and dividing by the three fitted coefficients, is \(D=(\hat\beta_{(-i)}-\hat\beta)^\mathsf{T}I(\hat\beta_{(-i)}-\hat\beta)/3\). Here it is \(1.6\{\log(7/8)\}^2/3\approx1.6(0.01783)/3\approx0.00951\). This scale-specific summary does not replace inspecting the changed risks and contrasts in context.
A Practical Sequence for Reporting the Analysis
A useful report makes clear how a flagged observation relates to the fitted model, rather than presenting a diagnostic number without interpretation. In this example, design inspection shows three replicated predictor patterns and a full-rank, three-parameter model. The fit assigns risks \(0.2,0.5,0.8\). Leverage is \(0.1\) for every observation, while the selected case has a moderate residual. Exact deletion then reveals a change only in the level-2 contrast, with its fitted risk moving to \(7/9\). The substantive direction persists: the level-2 risk remains above the reference-level risk.
Identify the predictor patterns, parameterization, replication, and whether the model is estimable.
Report fitted probabilities and interpret the coefficients on the log-odds or odds-ratio scale.
Use residual and leverage diagnostics to guide attention, then calculate exact case deletion for a case of interest.
State changes in coefficients and fitted risks, and distinguish a numerical change from a changed substantive conclusion.
Common Mistake: Treating a Diagnostic as the Conclusion
A tempting but incorrect argument is: “The selected case has a nonzero residual, so it must be influential,” or, conversely, “Its leverage is only \(0.1\), so it cannot matter.” A residual describes disagreement between an observation and the full fit; leverage describes its position in the weighted design. Neither alone gives the exact change caused by deletion. The correction is to use diagnostics as screening information, then refit the model without the case and compare quantities relevant to the scientific question.
Another potential error is to treat the numerical codes \(0,1,2\) as implying a linear trend when the model actually uses two indicator variables. In this example, the level-2 coefficient is a contrast against level 0, not a per-unit slope. The model specification determines the interpretation. If the model instead used a single numeric \(x\) term, its restrictions and deletion effects would be different.
Check Your Understanding
Use the model and counts in this worked analysis to answer the following.
- Why does the model use two indicator variables rather than interpreting the codes 0, 1, and 2 as a linear trend?
- Verify the fitted level-1 odds ratio against the reference category from the reported coefficients.
- For an observation at level 0, what is its leverage, and how does the weighted-design calculation produce that value?
- After deleting one successful level-2 observation, why is the revised level-2 probability \(7/9\), not \(7/10\)?
- Does the exact deletion change the direction of the level-2 versus level-0 contrast? Give the revised odds ratio.