Case Deletion and Changes in a Least-Squares Fit

Why Compare Fits With and Without One Case?

A least-squares line can respond noticeably to one observation, especially when that observation has an unusual response, a distinctive predictor value, or both. Comparing the original fit with a fit recomputed after omitting that case is a direct way to describe this sensitivity. The comparison asks: How much would the estimated relationship change if this one observation were not included?

A case-deletion comparison is a diagnostic, not a rule for removing data. A changed fit does not establish that an observation is erroneous, and an observation should not be discarded merely because its removal makes the results more convenient. First check the record, measurement process, and analysis question; then report the sensitivity transparently.

Define the Two Fits Before Comparing Them

Consider a regression model with \(n\) observations and \(p\) regression coefficients, including an intercept if the model has one. Write the predictor vector for observation \(i\) as \(\mathbf{x}_i\), the response as \(y_i\), and the full-sample least-squares coefficient vector as \(\hat{\boldsymbol{\beta}}\). Its fitted value is \(\hat{y}_i=\mathbf{x}_i'\hat{\boldsymbol{\beta}}\), and its residual is \(e_i=y_i-\hat{y}_i\).

Definition: The case-deletion fit for observation \(i\) is the least-squares fit obtained by removing observation \(i\), while keeping the response, predictors, model form, and coding unchanged. Denote its coefficient vector by \(\hat{\boldsymbol{\beta}}_{(i)}\). A coefficient change can be reported as \(\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)}\), provided that this subtraction order is stated and used consistently.

For the usual least-squares regression model, the conditional mean of the response is represented by the chosen linear predictor, and the errors have mean zero. Independence and constant error variance are common assumptions for standard inference. The case-deletion calculations below are algebraic properties of least squares: they require a full-rank design matrix for the fit being calculated, but do not require normally distributed errors. Comparisons are meaningful only when the same model specification is used for both fits.

In matrix notation, let \(X\) be the design matrix whose rows are \(\mathbf{x}_i'\), and let \(\mathbf{y}\) be the response vector. When \(X\) has full column rank, \(\hat{\boldsymbol{\beta}}=(X'X)^{-1}X'\mathbf{y}\). Define the leverage of observation \(i\) as \(h_{ii}=\mathbf{x}_i'(X'X)^{-1}\mathbf{x}_i\). If deleting the observation leaves the design matrix full rank, then \(h_{ii}<1\).

Worked Example: Comparing the Full and Deleted Lines

Suppose a hypothetical training exercise records study time \(x\), in hours, and a practice-test score \(y\), in points, for five participants. The data are \((0,1), (1,2.4), (2,3.8), (3,5.2), (4,9.6)\). Fit a simple linear regression with an intercept, then compare it with the line after deleting the last case.

The full-sample means are \(\bar{x}=2\) hours and \(\bar{y}=4.4\) points. The sum of squared deviations in \(x\) is \(S_{xx}=\sum (x_i-\bar{x})^2=10\) hours squared. The cross-product sum is \(S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})=20\) hour-points. Thus

\hat{\beta}_1=\frac{S_{xy}}{S_{xx}}=\frac{20}{10}=2 \qquad \hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=4.4-(2)(2)=0.4.

The full-sample fitted line is \(\hat{y}=0.4+2x\): each additional study hour is associated with an estimated 2-point increase in practice-test score over the fitted line. The intercept is the fitted score at zero hours, in points.

Now remove \((4,9.6)\). The remaining four points lie exactly on \(y=1+1.4x\): for example, at \(x=3\), \(1+1.4(3)=5.2\). The deleted-sample intercept is 1 point and the slope is 1.4 points per hour. With the comparison defined as full minus deleted, the intercept change is \(0.4-1=-0.6\) points and the slope change is \(2-1.4=0.6\) points per hour. In particular, the slope change is \(0.6\), not \(-0.4\).

Why Deletion Changes the Coefficients

The case-deletion formula makes the roles of the full-sample residual and leverage explicit. It also gives an exact result for ordinary least squares, rather than an approximation.

Theorem: If \(X\) has full column rank and deleting observation \(i\) leaves the design matrix full rank, then

\hat{\boldsymbol{\beta}}_{(i)} = \hat{\boldsymbol{\beta}} - \frac{(X'X)^{-1}\mathbf{x}_i e_i}{1-h_{ii}}.

Thus the change, defined as full minus deleted, is \((X'X)^{-1}\mathbf{x}_i e_i/(1-h_{ii})\). A larger full-sample residual can increase the change, while leverage affects both the direction through \(\mathbf{x}_i\) and the size through \(h_{ii}\) and the denominator. The expression is a vector: its components describe the changes to the individual coefficients, each in that coefficient’s units.

To justify the result, set \(A=X'X\). The full-sample normal equations give \(A\hat{\boldsymbol{\beta}}=X'\mathbf{y}\). After deleting case \(i\), the cross-product matrix and response cross-product are \(A-\mathbf{x}_i\mathbf{x}_i'\) and \(X'\mathbf{y}-\mathbf{x}_iy_i\). Substitute the claimed deleted estimate into the left side of the deleted normal equations:

\begin{aligned} (A-\mathbf{x}_i\mathbf{x}_i')\hat{\boldsymbol{\beta}}_{(i)} &=A\hat{\boldsymbol{\beta}} -\frac{\mathbf{x}_i e_i}{1-h_{ii}} -\mathbf{x}_i\left(\hat{y}_i-\frac{h_{ii}e_i}{1-h_{ii}}\right)\\ &=X'\mathbf{y}-\mathbf{x}_i(\hat{y}_i+e_i)\\ &=X'\mathbf{y}-\mathbf{x}_iy_i. \end{aligned}

Here \(\mathbf{x}_i'\hat{\boldsymbol{\beta}}=\hat{y}_i\), \(\mathbf{x}_i'(X'X)^{-1}\mathbf{x}_i=h_{ii}\), and \(\hat{y}_i+e_i=y_i\). The substituted estimate therefore satisfies the deleted normal equations. Full rank makes their least-squares solution unique, establishing the formula.

A related result describes the prediction at the deleted observation’s predictor values. Its residual in the deleted fit is

y_i-\mathbf{x}_i'\hat{\boldsymbol{\beta}}_{(i)} =\frac{e_i}{1-h_{ii}}.

This is the leave-one-out residual: it measures how far the response lies from a fit that did not use that response to estimate its coefficients. It is not the same as the full-sample residual \(e_i\).

Worked Example: Using Residual and Leverage to Explain the Change

Use the five hypothetical study-time observations from the previous example. For the last case, \(x_i=4\), the full-sample fitted score is \(0.4+2(4)=8.4\) points, so its full-sample residual is \(e_i=9.6-8.4=1.2\) points. In simple linear regression with an intercept, the leverage is \(h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}\). Therefore,

h_{ii}=\frac{1}{5}+\frac{(4-2)^2}{10}=0.2+0.4=0.6.

For a simple linear regression, the slope component of the full-minus-deleted change is

\hat{\beta}_1-\hat{\beta}_{1(i)} = \frac{(x_i-\bar{x})e_i}{S_{xx}(1-h_{ii})} = \frac{(4-2)(1.2)}{10(1-0.6)} = \frac{2.4}{4} = 0.6 \text{ points per hour}.

This agrees with the direct comparison: the full slope is 2 and the deleted slope is 1.4 points per hour. The full-sample residual is 1.2 points, but the deleted-fit residual is \(1.2/(1-0.6)=3\) points. Indeed, the deleted line predicts \(1+1.4(4)=6.6\) points at four hours, and \(9.6-6.6=3\). The full fit partly follows the unusual response, so its residual for that case is smaller.

Coefficient Changes and Fitted-Value Changes

A coefficient comparison alone does not show how much fitted values change in a particular part of the predictor range. For any predictor vector \(\mathbf{x}_*\), the difference between the full and deleted fitted values is

\hat{y}(\mathbf{x}_*)-\hat{y}_{(i)}(\mathbf{x}_*) = \mathbf{x}_*'(\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}_{(i)}).

The change therefore depends on both the coefficient changes and the predictor values at which the fits are compared. Two lines may differ little near one value of \(x\), but more at another. A slope change is measured per unit of the predictor; it should not be mistaken for a fixed change in predicted response.

Worked Example: Comparing Predictions Across Study Times

For the hypothetical data above, compare the full line \(\hat{y}=0.4+2x\) with the deleted line \(\hat{y}_{(i)}=1+1.4x\). Both fitted scores are measured in points, and \(x\) is measured in hours.

At zero hours, the full fit gives \(0.4+2(0)=0.4\) points, while the deleted fit gives \(1+1.4(0)=1\) point. The full-minus-deleted fitted-value change is \(0.4-1=-0.6\) points.

At two hours, the fits give \(0.4+2(2)=4.4\) points and \(1+1.4(2)=3.8\) points. The change is \(4.4-3.8=0.6\) points. At four hours, they give 8.4 and 6.6 points, so the change is \(8.4-6.6=1.8\) points. The difference between the fitted lines varies with study time; it is not just the intercept change or the slope change alone.

At the deleted case’s own study time of four hours, the observed score is 9.6 points. Its full-sample residual is \(9.6-8.4=1.2\) points, whereas its deleted-fit residual is \(9.6-6.6=3\) points. The larger deleted-fit residual shows how much the full fit had been pulled toward that response.

Check the Scope and Limits of the Comparison

A case-deletion analysis is most informative when its comparison is carefully defined. State which observation was omitted, retain the same model specification, and report changes with their signs and units. If deleting a case makes predictors redundant or otherwise causes rank deficiency, the original coefficient-by-coefficient comparison may no longer be available: the deleted model cannot uniquely estimate all the same coefficients.

The algebra explains sensitivity, not data quality or scientific importance. A large coefficient change may reflect a genuine but unusual observation, a recording problem, or a model that is sensitive in that part of the predictor space. A small change does not prove that a case is correct or unimportant for every purpose. Conclusions should be based on investigation of the observation and the analysis goals, not on a preferred direction of change.

Wrong: “Removing the case improves the fit, so it should be removed.” A better-looking fit alone is not evidence that the observation is invalid.
Correction: Verify the measurement and report the sensitivity. Exclude a case only for a defensible reason, and explain the choice.
Wrong: “The full-sample residual is the case’s residual after deletion.” The full-sample fit used the case; the deleted fit did not.
Correction: Distinguish \(e_i\) from the leave-one-out residual \(e_i/(1-h_{ii})\).
Wrong: “A slope change of 0.6 means every fitted score changes by 0.6 points.” A slope change is in points per hour; the fitted-value difference depends on the predictor value.

Key takeaway: Case deletion compares two fits of the same model, one using all observations and one omitting a specified case. Coefficient and fitted-value changes describe sensitivity; they do not, on their own, justify deleting the observation.

Check Your Understanding

Answer these questions using the definitions and results above.

In a case-deletion comparison, what must remain unchanged besides the included observations for coefficient changes to be interpretable?
For a specified case, explain the roles of its full-sample residual and leverage in the coefficient update formula.
Why can the residual for a case in the deleted fit differ from its full-sample residual?
If two fitted lines have different slopes, why might their fitted-value difference be small at one predictor value and larger at another?
A case deletion substantially changes a slope estimate. What does this show, and what does it not establish about whether the case should be removed?