Case-Deletion Residuals and Leave-One-Out Fitting

Why Refit Without the Case?

An ordinary residual measures the difference between an observed response and the fitted value from a model that included that observation. That fitted value may be pulled toward the observation itself. A leave-one-out prediction instead asks: what would the fitted model have predicted for this case if the case had not been used to estimate the model? Comparing the two quantities helps reveal when an observation’s own inclusion substantially affects its apparent prediction error.

This tutorial develops the exact relationship between those errors for ordinary least squares. The result uses the observation’s leverage, the diagonal entry of the hat matrix, as well as its ordinary residual. The algebra is exact; no normality assumption is needed for the identity. Statistical assumptions matter when attaching inferential meaning to residuals, but not for this calculation.

Definition: In a linear regression fitted by ordinary least squares, the ordinary residual for case $i$ is $e_i=y_i-\hat{y}_i$. The case-deletion residual, also called the leave-one-out prediction error, is $d_i=y_i-\hat{y}_{(i)}$, where $\hat{y}_{(i)}$ is the prediction at case $i$’s predictor values from a model fitted after omitting case $i$. The sign convention is observed minus predicted for both residuals.

Let $X$ be the $n$ by $p$ design matrix, including a column of ones when the model has an intercept, and let $\mathbf y$ be the response vector. Assume $X$ has full column rank, so the ordinary least-squares estimate is $\hat{\boldsymbol\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\mathbf y$. Write $\mathbf x_i^\mathsf{T}$ for row $i$ of $X$. The hat matrix is $H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$, and the leverage of case $i$ is $h_{ii}$, the $i$th diagonal element of $H$.

The deleted model must itself be estimable: after removing case $i$, the remaining design matrix must still have full column rank. Under this condition, $h_{ii}<1$, so the denominator in the relationship below is not zero. If $h_{ii}=1$, deleting the case makes the coefficients unidentifiable in at least one direction, and the leave-one-out formula does not apply.

The Deleted-Residual Relationship

For a full-rank ordinary least-squares model with an estimable deleted fit, the case-deletion residual is

d_i=y_i-\hat{y}_{(i)}=\frac{e_i}{1-h_{ii}}.

This formula shows why an ordinary residual alone does not tell the whole story. For the same $e_i$, a case with greater leverage has a smaller $1-h_{ii}$, and therefore a larger absolute leave-one-out prediction error. The deleted residual retains the response’s measurement units; it is not a standardized residual.

Derivation: Let $A=X^\mathsf{T}X$, and let $\hat{\boldsymbol\beta}_{(i)}$ denote the coefficient estimate after deleting case $i$. The full-data least-squares normal equations give $X^\mathsf{T}\mathbf y=A\hat{\boldsymbol\beta}$. The deleted normal equations are therefore $$ (A-\mathbf x_i\mathbf x_i^\mathsf{T})\hat{\boldsymbol\beta}_{(i)} =A\hat{\boldsymbol\beta}-\mathbf x_i y_i. $$ Subtracting the deleted estimate from the full estimate and using $e_i=y_i-\mathbf x_i^\mathsf{T}\hat{\boldsymbol\beta}$ gives $$ (A-\mathbf x_i\mathbf x_i^\mathsf{T})(\hat{\boldsymbol\beta}-\hat{\boldsymbol\beta}_{(i)}) =\mathbf x_i e_i. $$ The rank-one inverse identity yields $$ (A-\mathbf x_i\mathbf x_i^\mathsf{T})^{-1}\mathbf x_i =\frac{A^{-1}\mathbf x_i}{1-\mathbf x_i^\mathsf{T}A^{-1}\mathbf x_i} =\frac{A^{-1}\mathbf x_i}{1-h_{ii}}. $$ Consequently, $$ \mathbf x_i^\mathsf{T}(\hat{\boldsymbol\beta}-\hat{\boldsymbol\beta}_{(i)}) =\frac{\mathbf x_i^\mathsf{T}A^{-1}\mathbf x_i\,e_i}{1-h_{ii}} =\frac{h_{ii}e_i}{1-h_{ii}}. $$ Since $\hat{y}_i=\mathbf x_i^\mathsf{T}\hat{\boldsymbol\beta}$, this says $\hat{y}_i-\hat{y}_{(i)}=h_{ii}e_i/(1-h_{ii})$. Finally, $$ y_i-\hat{y}_{(i)} =(y_i-\hat{y}_i)+(\hat{y}_i-\hat{y}_{(i)}) =e_i+\frac{h_{ii}e_i}{1-h_{ii}} =\frac{e_i}{1-h_{ii}}. $$

The last step also gives a useful interpretation: the deleted-fit prediction for case $i$ moves away from the full-fit prediction by $h_{ii}e_i/(1-h_{ii})$. If $e_i$ is positive, the full-data fitted value lies below the observed response, and omitting the case moves its prediction further down; the deleted residual is positive and larger than the ordinary residual when $0\leq h_{ii}<1$. The signs reverse when $e_i$ is negative.

Worked Example: Checking a Deleted Residual by Refitting

A hypothetical insurer models a claim-severity index $y$ with a simple linear regression on a rating score $x$. The four observations are $(0,2),(1,3),(2,5)$; a fourth point is not needed, so the fitted data set here consists of these three observations. Find the ordinary and deleted residuals for the observation $(1,3)$, and check the deleted residual by fitting the model without it.

Method. Fit the full regression, calculate the case’s leverage and ordinary residual, then apply the deletion formula. Finally, refit on the other two observations. This check avoids assuming that the ordinary residual equals the leave-one-out prediction error.

For the three observations, $\bar{x}=(0+1+2)/3=1$, $\bar{y}=(2+3+5)/3=10/3$, and $S_{xx}=(-1)^2+0^2+1^2=2$. The slope is \[ \hat{\beta}_1=\frac{(-1)(2-10/3)+0(3-10/3)+1(5-10/3)}{2} =\frac{4/3+0+5/3}{2}=\frac{3}{2}. \] The intercept is $\hat{\beta}_0=10/3-(3/2)(1)=11/6$. At $x=1$, the fitted value is $11/6+3/2=20/6=10/3$, so the ordinary residual is $e_i=3-10/3=-1/3$.

For a simple regression with an intercept, $h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}$. Here $n=3$, $x_i=\bar{x}=1$, and $S_{xx}=2$, so $h_{ii}=1/3+0=1/3$. Thus \[ d_i=\frac{-1/3}{1-1/3}=\frac{-1/3}{2/3}=-\frac{1}{2}. \]

To check directly, omit $(1,3)$. The remaining points $(0,2)$ and $(2,5)$ define a line with slope $(5-2)/(2-0)=3/2$ and intercept $2$. Its prediction at $x=1$ is $2+3/2=7/2$, so the deleted residual is $3-7/2=-1/2$, matching the formula. The model is estimable after deletion because the two remaining $x$-values differ.

Leverage Changes the Meaning of a Residual

A residual is calculated after fitting the model to the case as well as to the other observations. The case can influence its own fitted value, particularly when its predictor values are unusual relative to the design. This is why the deleted residual can be much larger in magnitude than the ordinary residual. It is a prediction error from a fit that did not use the response being evaluated.

For a fixed design and a case with $h_{ii}$ close to one, the multiplier $1/(1-h_{ii})$ can be large. This does not guarantee that $d_i$ is large: if $e_i=0$, then $d_i=0$. It does mean that a nonzero ordinary residual can correspond to a substantially larger deleted residual. Conversely, a modest leverage leaves the ordinary and deleted residuals closer together.

Worked Example: Equal Ordinary Residuals, Different Leverages

A hypothetical portfolio analysis uses a simple regression with an intercept and four rating scores $x=-2,-1,1,2$. The observed responses are $y=7,7,11,15$, respectively. Compare the deleted residuals for the cases at $x=-2$ and $x=-1$.

Method. First verify the full fitted line and residuals. Then calculate leverage from the predictor values and apply $d_i=e_i/(1-h_{ii})$. This comparison avoids ranking cases by ordinary residual magnitude alone.

The predictor mean is $\bar{x}=0$, and the response mean is $\bar{y}=(7+7+11+15)/4=10$. The slope numerator is \[ (-2)(7-10)+(-1)(7-10)+(1)(11-10)+(2)(15-10)=6+3+1+10=20. \] The denominator is $S_{xx}=4+1+1+4=10$, so the slope is $20/10=2$, and the intercept is $10-2(0)=10$. The fitted values, in order, are $6,8,12,14$, giving residuals $1,-1,-1,1$. In particular, the two cases being compared have ordinary residuals $1$ and $-1$, equal in absolute value.

For each case, $h_{ii}=1/4+x_i^2/10$. At $x=-2$, $h_{ii}=1/4+4/10=0.65$, so \[ d_i=\frac{1}{1-0.65}=\frac{1}{0.35}=\frac{20}{7}\approx 2.8571. \] At $x=-1$, $h_{ii}=1/4+1/10=0.35$, so \[ d_i=\frac{-1}{1-0.35}=\frac{-1}{0.65}=-\frac{20}{13}\approx -1.5385. \] The endpoint case has the larger absolute deleted residual despite having the same absolute ordinary residual. The deleted fit for $x=-2$ is estimable: the remaining predictor values $-1,1,2$ are not all identical. As a direct check, their response mean is $11$, their predictor mean is $2/3$, and their slope is $12/(14/3)=18/7$. The fitted line has intercept $65/7$, predicts $29/7$ at $x=-2$, and gives $7-29/7=20/7$, as above.

Using Deleted Residuals for Case Screening

A large deleted residual flags a case whose observed response is poorly predicted by a model that did not use that response. It is a useful screening quantity, especially when a case’s ordinary residual may understate its prediction error because of leverage. Screening is not the same as deciding that a record is erroneous or should be removed. A large value calls for investigation of the case, the data, and the adequacy of the model; it does not establish a cause.

The formula is an algebraic identity for ordinary least squares with a fixed design and a valid deleted fit. It is not, by itself, a probability statement or a universal cutoff. The raw deleted residual is measured in response units, so its size depends on the response scale. If a standardized or studentized diagnostic is needed, that requires additional scaling and assumptions; the deleted residual alone should not be interpreted as a standard-normal score.

There is also an important boundary condition. When $h_{ii}$ approaches one, the factor $1/(1-h_{ii})$ becomes very large. This can make the deleted residual numerically sensitive, and at $h_{ii}=1$ the deletion fit is not estimable. Before using the formula, check that the model can still be fitted after deleting the case. For weighted regression or a model refitted with a different specification, do not apply this ordinary least-squares identity without establishing the corresponding result for that procedure.

Worked Example: Interpreting a Large Multiplier

A hypothetical regression of a monthly service-cost index reports an ordinary residual of $e_i=0.6$ index points and leverage $h_{ii}=0.80$ for one case. Assume the regression is ordinary least squares, has full column rank, and remains estimable when this case is deleted. Calculate the deleted residual and explain what it does and does not imply.

Method. Apply the exact deleted-residual identity, then interpret the answer in response units. This avoids treating a large multiplier as proof that the case is invalid.

The denominator is $1-h_{ii}=1-0.80=0.20$. Therefore \[ d_i=\frac{0.6}{0.20}=3.0 \] index points. The ordinary fitted value is $0.6$ points below the observation, while the deleted-fit prediction is $3.0$ points below it. The difference between the two predictions is $3.0-0.6=2.4$ points, which also equals $h_{ii}e_i/(1-h_{ii})=(0.80)(0.6)/0.20=2.4$. These calculations agree.

The case merits review because its leave-one-out prediction error is much larger than its ordinary residual. The calculation does not establish that the observation is erroneous, nor does it give a probability that the case is unusual. The conclusion is limited to its prediction error under the stated fitted model.

Common Exam Trap

A frequent error is to multiply the ordinary residual by $1-h_{ii}$ instead of dividing by it. For example, if $e_i=1.2$ and $h_{ii}=0.50$, the incorrect calculation $1.2(1-0.50)=0.6$ makes the deleted error smaller. The correct calculation is \[ d_i=\frac{1.2}{1-0.50}=\frac{1.2}{0.50}=2.4. \] The deleted residual is larger here because the full-data fit partially uses the case to fit itself. A second trap is to equate a large deleted residual with automatic case deletion. The diagnostic identifies a potentially important prediction discrepancy; the analyst must investigate context and model adequacy before drawing a conclusion.

Key takeaway: For an estimable leave-one-out ordinary least-squares fit, $d_i=e_i/(1-h_{ii})$. The ordinary residual supplies the direction and initial size of the prediction error; leverage determines how much larger that error becomes when the case is excluded. Use the result to screen and investigate, not as an automatic deletion rule.

Check Your Understanding

For each problem, assume the stated ordinary least-squares model remains estimable after deleting the case. Use the observed-minus-predicted sign convention.

A case has ordinary residual $e_i=1.5$ response units and leverage $h_{ii}=0.25$. Calculate its deleted residual and state its sign. (Answer: $2.0000$ response units, positive.)
A case has ordinary residual $e_i=-1.2$ and leverage $h_{ii}=0.60$. Calculate its deleted residual and the difference $\hat{y}_i-\hat{y}_{(i)}$ between the full-fit and deleted-fit predictions. (Answer: $d_i=-3.0000$; $\hat{y}_i-\hat{y}_{(i)}=-1.8000$.)
Two cases have $(e_i,h_{ii})=(1.5,0.25)$ and $(1.0,0.60)$. Calculate both deleted residuals and identify which has the larger absolute leave-one-out error. (Answer: $2.0000$ and $2.5000$, respectively; the second case.)
A case has deleted residual $d_i=-2.4$ and leverage $h_{ii}=0.20$. Recover its ordinary residual, then state whether its observed response is above or below its deleted-fit prediction. (Answer: $e_i=-1.9200$; the response is below its deleted-fit prediction.)
A case has $e_i=0.10$ and $h_{ii}=0.95$, with a valid deleted fit. Calculate its deleted residual and explain why the size alone does not prove that the case is invalid. (Answer: $d_i=2.0000$; it is a large leave-one-out discrepancy but requires investigation and is not proof of an invalid record.)

What You'll Learn