Why Delete a Case?
A regression fit can depend strongly on a single observation. This may happen because the case has an unusual predictor value, an unusual response given its predictors, or both. A case-deletion sensitivity analysis compares the fitted model using all observations with the model refitted after removing one case. The point is not to remove a case automatically; it is to determine whether the model-based conclusion depends materially on it.
The comparison should include more than coefficient estimates. Deleting a case changes the residual degrees of freedom and may change the estimated error variance, coefficient standard errors, and prediction uncertainty. A shift in a coefficient may be practically important even if it does not reverse a test conclusion; conversely, a statistically noticeable shift may have little consequence for the decision being considered.
Model and Case-Deletion Formulas
Let \(n\) observations follow the linear model \(\mathbf{Y}=X\boldsymbol{\beta}+\boldsymbol{\epsilon}\), where \(X\) has full column rank and \(p\) columns, including an intercept if the model has one. For the usual standard errors and \(t\)-based inference, assume independent errors with mean zero, common variance \(\sigma^2\), and a normal distribution. The coefficient deletion identities below are algebraic least-squares results; normality is not needed for those identities.
Let \(\mathbf{x}_i^\mathsf{T}\) be row \(i\) of \(X\), let \(e_i=y_i-\mathbf{x}_i^\mathsf{T}\hat{\boldsymbol{\beta}}\) be its full-fit residual, and let \(h_{ii}\) be its leverage, the \(i\)th diagonal element of \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\). The following identities apply when deleting case \(i\) leaves a full-column-rank design, so in particular \(h_{ii}<1\).
A crucial distinction is between the full-fit residual and the deleted-case residual. The full-fit residual remains \(e_i\). The deleted-case residual evaluates the omitted observation against the fit that excluded it:
If \(n>p\), the full and deleted residual variance estimates are \(s^2=\operatorname{SSE}/(n-p)\) and \(s^2_{(i)}=\operatorname{SSE}_{(i)}/(n-p-1)\), respectively. Their coefficient covariance estimates are \(s^2(X^\mathsf{T}X)^{-1}\) and \(s^2_{(i)}(X_{(i)}^\mathsf{T}X_{(i)})^{-1}\), where \(X_{(i)}\) is the design matrix with row \(i\) removed. Thus deleting a case changes both the inverse cross-product matrix and the estimate of error variance.
Why the Update Formulas Hold
Write \(A=X^\mathsf{T}X\). Removing case \(i\) changes the cross-product matrix to \(A-\mathbf{x}_i\mathbf{x}_i^\mathsf{T}\), and changes \(X^\mathsf{T}\mathbf{Y}\) to \(X^\mathsf{T}\mathbf{Y}-\mathbf{x}_i y_i\). The rank-one inverse identity gives
Substituting the deleted cross-products into the least-squares formula yields the coefficient update. To see the residual-sum-of-squares identity, note that the reduction in full-data residual sum of squares attributable to retaining case \(i\), relative to the deleted fit, is \(e_i^2/(1-h_{ii})\). Equivalently, the deleted residual is \(e_{i(i)}=e_i/(1-h_{ii})\), and the decrease is \(e_i e_{i(i)}\). Hence \(\operatorname{SSE}_{(i)}=\operatorname{SSE}-e_i^2/(1-h_{ii})\). These identities also show why a large full residual is not the only warning sign: a residual is amplified in the deleted fit when leverage is high.
Worked Comparisons
Worked Example: Refit, Standard Errors, and a Mean Prediction
A hypothetical pricing analysis uses an intercept and one predictor, so \(p=2\). There are \(n=8\) observations. The full fit has \(\hat{\boldsymbol{\beta}}=(10,2)^\mathsf{T}\), \(\operatorname{SSE}=18\), and \((X^\mathsf{T}X)^{-1}=\begin{pmatrix}17/84&1/84\\1/84&5/84\end{pmatrix}\). For case \(i\), \(\mathbf{x}_i=(1,1)^\mathsf{T}\), \(y_i=14\), and \(e_i=2\). The case is also known to have leverage \(h_{ii}=2/7\). Compare the fits and the standard error for the estimated mean response at predictor value 1.
Method. Apply the case-deletion update, then recalculate the residual variance and the covariance-based standard errors. First, the full fitted value at case \(i\) is \(10+2=12\), so its full-fit residual is \(14-12=2\), as given. The inverse cross-product matrix gives \((X^\mathsf{T}X)^{-1}\mathbf{x}_i=(18/84,6/84)^\mathsf{T}=(3/14,1/14)^\mathsf{T}\). Also, \(1-h_{ii}=1-2/7=5/7\).
The coefficient update is \((3/14,1/14)^\mathsf{T}\times 2/(5/7)=(3/5,1/5)^\mathsf{T}\). Therefore
The full fitted value at the deleted case’s predictor values was 12; the deleted-fit value there is \(9.4+1.8=11.2\). The deleted-case residual is \(14-11.2=2.8\), which checks the formula \(e_{i(i)}=2/(5/7)=2.8\). It is not the full-fit residual, which remains 2.
The residual sums of squares are \(\operatorname{SSE}=18\) and \(\operatorname{SSE}_{(i)}=18-2^2/(5/7)=18-5.6=12.4\). The full residual variance estimate is \(18/(8-2)=3\); the deleted estimate is \(12.4/(8-2-1)=2.48\).
For the deleted fit, the inverse cross-product matrix is \(\begin{pmatrix}17/84&1/84\\1/84&5/84\end{pmatrix} +\begin{pmatrix}3/14\\1/14\end{pmatrix}\begin{pmatrix}3/14&1/14\end{pmatrix}/(5/7) = \begin{pmatrix}4/15&1/30\\1/30&1/15\end{pmatrix}\). Thus, the slope standard error is \(\sqrt{3(5/84)}\approx0.4226\) in the full fit and \(\sqrt{2.48(1/15)}\approx0.4066\) in the deleted fit.
At predictor value 1, use \(\mathbf{x}_0=(1,1)^\mathsf{T}\). The estimated mean is 12 for the full fit and 11.2 for the deleted fit. The full prediction variance factor for the estimated mean is \(\mathbf{x}_0^\mathsf{T}(X^\mathsf{T}X)^{-1}\mathbf{x}_0=17/84+2(1/84)+5/84=2/7\). The deleted factor is \(4/15+2(1/30)+1/15=0.4\). Therefore, the mean-response standard errors are \(\sqrt{3(2/7)}\approx0.9258\) and \(\sqrt{2.48(0.4)}\approx0.9960\), respectively.
Check and trap avoided. Deletion lowers the estimate at this predictor value by 0.8 and slightly increases its mean-response standard error. In the deleted quadratic form, the off-diagonal entry \(1/30\) contributes twice: \(4/15+2(1/30)+1/15=0.4\). Using the wrong off-diagonal entry would give the wrong prediction standard error.
Worked Example: A Large Residual and High Leverage Together
In a hypothetical regression with \(n=12\) observations and \(p=3\) coefficients, one case has \(h_{ii}=0.5\) and full-fit residual \(e_i=3\). The full fit has \(\operatorname{SSE}=30\). Calculate the deleted-case residual, the deleted residual variance estimate, and Cook’s distance for this case.
Method. First calculate the deleted residual and residual variance. The deleted-case residual is \(3/(1-0.5)=6\). The deleted residual sum of squares is \(30-3^2/(1-0.5)=30-18=12\). The full residual degrees of freedom are \(12-3=9\), giving \(s^2=30/9=10/3\). After deletion, the degrees of freedom are \(12-1-3=8\), giving \(s^2_{(i)}=12/8=1.5\).
For a case in ordinary least squares, Cook’s distance is \(D_i=e_i^2h_{ii}/\{p s^2(1-h_{ii})^2\}\). Substitution gives \(D_i=9(0.5)/\{3(10/3)(0.5)^2\}=4.5/2.5=1.8\). As an additional check on the two diagnostic signals, the internally studentized residual is \(3/\sqrt{(10/3)(0.5)}\approx2.3238\), whereas the externally studentized deleted residual is \(6/\sqrt{1.5(0.5)}\approx6.9282\).
Interpretation and trap avoided. This case has both substantial leverage and a nonzero residual, and its deletion deserves investigation. The deleted residual is 6; it is not the full-fit residual, which is 3. A diagnostic value flags sensitivity for examination, but by itself does not justify removing a valid observation.
Worked Example: Does Deletion Change a Prediction Conclusion?
A hypothetical severity model has \(n=10\), \(p=2\), full coefficients \((100,5)^\mathsf{T}\), and \(\operatorname{SSE}=36\). Its inverse cross-product matrix is \(\begin{pmatrix}0.25&0\\0&0.10\end{pmatrix}\). A case has \(\mathbf{x}_i=(1,1)^\mathsf{T}\) and full-fit residual \(e_i=1.3\). Compare the slope estimate and the estimated mean at predictor value 3 before and after deleting the case.
Method. The leverage is \(h_{ii}=0.25+0.10=0.35\). Since \((X^\mathsf{T}X)^{-1}\mathbf{x}_i=(0.25,0.10)^\mathsf{T}\) and \(e_i/(1-h_{ii})=1.3/0.65=2\), the coefficient change is \((0.50,0.20)^\mathsf{T}\). The deleted coefficients are \((99.5,4.8)^\mathsf{T}\).
The full residual variance estimate is \(36/(10-2)=4.5\). Deletion reduces the sum of squares by \(1.3^2/0.65=2.6\), so \(\operatorname{SSE}_{(i)}=33.4\) and \(s^2_{(i)}=33.4/(10-1-2)=33.4/7\approx4.7714\). The full slope standard error is \(\sqrt{4.5(0.10)}\approx0.6708\). The deleted inverse cross-product matrix has slope diagonal entry \(0.10+0.10^2/0.65\approx0.1154\), so its slope standard error is \(\sqrt{(33.4/7)(0.1154)}\approx0.7420\).
At predictor value 3, the full estimated mean is \(100+5(3)=115\); after deletion it is \(99.5+4.8(3)=113.9\). For the full fit, the mean-response variance factor is \(0.25+3^2(0.10)=1.15\), giving standard error \(\sqrt{4.5(1.15)}\approx2.2750\). For the deleted fit, the inverse cross-product matrix is approximately \(\begin{pmatrix}0.3462&0.0385\\0.0385&0.1154\end{pmatrix}\). Its factor at \((1,3)^\mathsf{T}\) is \(0.3462+2(3)(0.0385)+9(0.1154)\approx1.6154\), giving standard error \(\sqrt{(33.4/7)(1.6154)}\approx2.7763\).
Check and trap avoided. Deletion lowers the estimated slope from 5 to 4.8 and the estimated mean at predictor value 3 from 115 to 113.9, while increasing its standard error from about 2.2750 to 2.7763. The prediction variance factor must use the full quadratic form, including twice the off-diagonal contribution; it is not just the sum of the diagonal terms.
How to Judge Sensitivity
A useful deletion analysis compares like with like: the same response scale, predictor set, and target prediction, fitted once with all cases and once with the case omitted. When evaluating a coefficient, compare its estimate and standard error, or the relevant contrast and its uncertainty. When evaluating a prediction, compare the estimated mean and its standard error; for an individual future response, prediction uncertainty also includes the error variance. Do not treat a mean-response standard error as an individual prediction standard error.
Leverage describes how unusual a case’s predictor position is relative to the design. The full-fit residual describes its response discrepancy after fitting all cases. A case can have high leverage but a small residual and little effect, or a large residual but modest leverage. Deletion sensitivity reflects their interaction and the surrounding design. Consequently, no single residual, leverage, or Cook’s distance cutoff proves that an observation should be discarded. Check the data and its context, then state whether the substantive model conclusion changes.
Check Your Understanding
For each problem, calculate the requested deletion quantity before checking the answer.
- A regression has \(n=15\), \(p=3\), \(e_i=2.5\), \(h_{ii}=0.2\), and full \(\operatorname{SSE}=40\). Find the deleted-case residual and \(\operatorname{SSE}_{(i)}\). (Answer: \(e_{i(i)}=3.125\), \(\operatorname{SSE}_{(i)}=32.1875\).)
- A full fit has \(\hat{\boldsymbol{\beta}}=(3,2)^\mathsf{T}\). For a case, \((X^\mathsf{T}X)^{-1}\mathbf{x}_i=(0.4,-0.1)^\mathsf{T}\), \(e_i=1.5\), and \(h_{ii}=0.25\). Find the deleted coefficient estimate. (Answer: \(\hat{\boldsymbol{\beta}}_{(i)}=(2.2,2.2)^\mathsf{T}\).)
- A model has \(n=9\), \(p=2\), full \(\operatorname{SSE}=28\), \(e_i=2\), and \(h_{ii}=0.5\). Calculate the full and deleted residual variance estimates. (Answer: \(s^2=4\) and \(s^2_{(i)}=20/6\approx3.3333\).)
- For a target predictor vector, the full estimated mean is 50 with mean-response standard error 2.0; after deletion, the estimated mean is 48.5 with standard error 2.5. State the changes and whether the estimate’s movement is larger than either standard error. (Answer: The mean estimate decreases by 1.5; this is smaller than both standard errors, 2.0 and 2.5.)