Case-Deletion Checks for Regression Influence

Why Refit a Regression Line?

A regression diagnostic can flag an observation as influential: the fitted regression may change noticeably when that observation is included. A case-deletion check makes that concern concrete. Fit the line using all observations, remove the flagged case, fit the line again, and compare the results. The point is not to decide automatically that the case should be excluded. It is to find out whether the substantive description or prediction depends heavily on it.

This check follows naturally from the earlier material on Cook's Distance for Influential Observations. Cook's Distance can identify cases whose removal may matter; case deletion directly examines what the refitted line looks like. The comparison should be made in the context of the predictor and response, with particular attention to the slope and to predictions at relevant predictor values.

Refitting Simple Linear Regression

For paired observations \((x_i,y_i)\), the least-squares regression line is \(\hat{y}=b_0+b_1x\). It chooses the intercept \(b_0\) and slope \(b_1\) to minimize the sum of squared residuals. Provided the predictor values are not all identical, its coefficients can be calculated using

where \(S_{xx}=\sum (x_i-\bar{x})^2\) and \(S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})\). The intercept is the fitted response at \(x=0\); that value may not be practically meaningful if zero is outside the relevant range of the predictor.

If observation \(i\) is removed, the remaining sample size is \(n-1\). Calculate a new predictor mean, response mean, \(S_{xx}\), and \(S_{xy}\) using only the retained cases. The resulting coefficients can be denoted \(b_{0(-i)}\) and \(b_{1(-i)}\), giving the deleted-case line \(\hat{y}_{(-i)}=b_{0(-i)}+b_{1(-i)}x\). The observations used in the refit are the same in each calculation except for the specified deletion.

Why the Centered Sums Change

A short derivation helps explain why deleting a case requires recalculating the regression rather than simply removing its residual from a summary. Let \(X=\sum x_i\), \(Y=\sum y_i\), \(X_2=\sum x_i^2\), and \(XY=\sum x_iy_i\), all calculated over the full data. Expanding the definitions of the centered sums gives

In the second expression, \(XY\) on the right denotes the sum of products \(\sum x_iy_i\); the product in the numerator is the product of the separate totals \(X\) and \(Y\). Equivalently, to avoid that visual ambiguity, write the formulas with the totals named explicitly:

For deletion of case \(i\), each raw total must first have that case's contribution removed, and the divisor becomes \(n-1\). For example, the new sum of products is \(\sum x_iy_i-x_iy_i\), where the first term is the full-data total and the second is the deleted case's product. The new correction term uses the new totals and \(n-1\). Thus deletion changes both the raw totals and the means used for centering. Since the slope is a ratio of the resulting centered sums, its direction or size can change substantially.

Worked Examples

How to Interpret the Comparison

A case-deletion check is most useful when the quantities compared connect to a real question. A slope change matters when the estimated rate of response per predictor unit is central. A prediction change matters when a specific predictor value is used for planning or decision-making. If the intercept corresponds to an unrealistic value such as zero hours of practice when no one in the data is near zero, its numerical change may be less informative than changes in predictions within the observed range.

The check is descriptive: it reports how the fitted results change in this data set. It does not, by itself, determine why the case is unusual or whether it should be removed. Verify the record and the case's eligibility using the study's rules. If the observation is valid and belongs to the population being studied, an influential result may be important information about how much the analysis relies on that case. Report the comparison transparently rather than presenting only the more convenient line.

Check that the regression remains calculable after deletion: at least two observations must remain, and their predictor values must not all be identical. Also examine the data and the fit in context. As in the earlier material on residual plots, a line that changes after deletion is not the only possible model concern; a deletion comparison cannot show by itself that a linear model is appropriate. Nor does one deletion check assess every possible source of error or explain why a case affects the fit.

Common Mistakes

• Using an incorrect sum of products. The deleted case contributes \(x_iy_i\), so removing it changes \(\sum x_iy_i\) by that amount. Check the raw totals against the listed data before computing the slope.
• Changing only the numerator or only the denominator. Deletion changes the means as well as the sums of squares and products. Recalculate both \(S_{xx}\) and \(S_{xy}\) for the retained observations.
• Calling the flagged case an error without checking. A diagnostic flag signals a case worth examining; it does not prove that its recorded values are wrong or that it should be excluded.
• Comparing slopes without comparing relevant predictions. A slope may change little while predictions shift, or a slope may change while a prediction at a particular value remains similar. Choose comparisons that address the question.
• Treating sensitivity as causation. A changed regression line describes dependence of the fitted association on an observation. It does not show that the predictor causes the response.