How Influential Cases Change Regression Inference

Why Influence Can Change a Regression Conclusion

A regression diagnostic can identify an observation whose presence materially affects a fitted model. That effect is not limited to the fitted line: removing the observation and refitting can also change coefficient standard errors, confidence intervals, and hypothesis-test conclusions. A reported conclusion therefore needs to specify which data and which fitted model it describes.

The comparison is between two fits: the full fit, using all observations, and the <strongcase-deleted fit, refitted after excluding one observation. These are distinct analyses. The deleted fit has a different coefficient estimate, design matrix, residual variance estimate, and residual degrees of freedom. Its standard errors and tests must be recomputed from that fit.

Definition: An observation is influential for regression inference when excluding it and refitting materially changes an inferential result, such as a coefficient estimate, its standard error, a confidence interval, or a test conclusion. Influence describes the effect on the fitted analysis; by itself, it does not establish that the observation is erroneous or should be removed.

What Changes When an Observation Is Deleted?

Consider the linear regression model \(Y=X\beta+\varepsilon\), where \(X\) has \(n\) rows and \(p\) columns, including a column of ones when the model has an intercept. Under the usual fixed-design assumptions for classical regression inference, errors have mean zero, common variance \(\sigma^2\), and are independent; normal errors give the exact \(t\)-based procedures described here. The least-squares estimate is \(\hat{\beta}=(X'X)^{-1}X'Y\), when \(X'X\) is invertible. The estimated error variance and coefficient covariance matrix are

\hat{\sigma}^2=\frac{\mathrm{SSE}}{n-p}, \qquad \widehat{\operatorname{Cov}}(\hat{\beta})= \hat{\sigma}^2(X'X)^{-1}.

For coefficient \(j\), the estimated standard error is the square root of the \(j\)th diagonal entry of this covariance matrix. A two-sided \(100(1-\alpha)\%\) confidence interval is \(\hat{\beta}_j\pm t_{1-\alpha/2,n-p}\operatorname{SE}(\hat{\beta}_j)\). A test of \(H_0:\beta_j=0\) uses \(t=\hat{\beta}_j/\operatorname{SE}(\hat{\beta}_j)\), with \(n-p\) residual degrees of freedom under the model assumptions.

After deleting one observation, the refitted data contain \(n-1\) rows. Provided the refitted design matrix still has full column rank, its standard error uses the deleted fit’s own residual sum of squares, design matrix, and degrees of freedom, \(n-1-p\). Both the numerator and uncertainty in a test statistic can change. The degrees of freedom also decrease by one. There is no general rule that deletion must increase or decrease a standard error.

Why the Coefficient Estimate Changes

The exact case-deletion identity makes the role of the deleted observation visible. Let \(x_i\) be its row of predictors, \(e_i=y_i-x_i'\hat{\beta}\) its residual in the full fit, and \(h_{ii}=x_i'(X'X)^{-1}x_i\) its leverage. When the deleted design matrix remains full rank and \(h_{ii}<1\), the estimate after deleting observation \(i\) is

\hat{\beta}_{(i)} = \hat{\beta} - \frac{(X'X)^{-1}x_i e_i}{1-h_{ii}}.

Here is a short derivation. The full normal equations are \(X'X\hat{\beta}=X'Y\). Removing row \(i\) changes the left-hand matrix to \(X'X-x_ix_i'\) and the right-hand vector to \(X'Y-x_iy_i\). Write the deleted estimate as \(\hat{\beta}+\Delta\). Substituting it into the deleted normal equations and using \(X'X\hat{\beta}=X'Y\) gives

(X'X-x_ix_i')\Delta=-x_i e_i.

Solving this rank-one update yields \(\Delta=-(X'X)^{-1}x_i e_i/(1-h_{ii})\), the stated identity. The formula explains why a large residual or high leverage can contribute to a substantial coefficient change. It does not, however, determine whether deletion is appropriate: that requires checking the observation, the data collection, and the modeling context.

Worked Example: A Slope Test Changes After Refitting

A hypothetical analyst models centered annual claim-cost change, in thousands of dollars, using a rating score \(x\). Five observations are \((-2,-0.5),(-1,0),(0,4),(1,4),(2,7.5)\). A sixth observation is \((4,3)\). Compare the full and case-deleted 5% two-sided tests of a zero slope, with an intercept in both fits.

Method. Compute each fit from its own centered sums and residual sum of squares, then use its own standard error and degrees of freedom. This avoids the mistake of pairing the deleted slope with the full-fit uncertainty.

For the first five observations, \(\bar{x}=0\) and \(\bar{y}=3\). The centered predictor sum of squares is

S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=10.

The centered cross-product is \(S_{xy}=(-2)(-3.5)+(-1)(-3)+0(1)+1(1)+2(4.5)=20\). Thus the deleted-fit slope is \(20/10=2\), and its intercept is \(3-2(0)=3\). Its fitted values are \(-1,1,3,5,7\), so the residuals are \(0.5,-1,1,-1,0.5\). Therefore \(\mathrm{SSE}_{(i)}=0.25+1+1+1+0.25=3.5\), and \(\mathrm{MSE}_{(i)}=3.5/(5-2)=7/6\). The slope standard error and test statistic are

\operatorname{SE}(\hat{\beta}_{1(i)}) =\sqrt{\frac{7/6}{10}} =\sqrt{\frac{7}{60}} \approx 0.342, \qquad t_{(i)}=\frac{2}{\sqrt{7/60}}\approx 5.855.

The deleted fit has \(5-2=3\) degrees of freedom. Its 95% critical value is approximately \(3.182\), so the test rejects a zero slope. Its interval, to three decimals, is \(2\pm3.182\sqrt{7/60}=[0.913,3.087]\).

Now include \((4,3)\). With six observations, \(\bar{x}=4/6=2/3\) and \(\bar{y}=18/6=3\). Adding one observation to the five-point sample gives

S_{xx,\mathrm{full}}=10+\frac{5}{6}(4-0)^2=\frac{70}{3}, \qquad S_{xy,\mathrm{full}}=20+\frac{5}{6}(4-0)(3-3)=20.

Thus the full-fit slope is \(20/(70/3)=6/7\), and its intercept is \(3-(6/7)(2/3)=17/7\). The added point’s residual relative to the deleted fit is \(3-(3+2(4))=-8\). The full-fit SSE can be obtained by adding the contribution of that point while accounting for refitting:

\mathrm{SSE}_{\mathrm{full}} = 3.5+\frac{5}{6}(-8)^2 = \frac{341}{6}.

The full fit has \(6-2=4\) residual degrees of freedom, so \(\mathrm{MSE}_{\mathrm{full}}=(341/6)/4=341/24\). Since \(S_{xx,\mathrm{full}}=70/3\), its slope standard error and test statistic are

\operatorname{SE}(\hat{\beta}_{1,\mathrm{full}}) = \sqrt{\frac{341/24}{70/3}} = \sqrt{\frac{341}{560}} \approx 0.780, \qquad t_{\mathrm{full}} = \frac{6/7}{\sqrt{341/560}} \approx 1.098.

The full fit’s 95% critical value is approximately \(2.776\), so it does not reject a zero slope. Its 95% interval is \((6/7)\pm2.776\sqrt{341/560}=[-1.309,3.023]\), to three decimals. The interval includes zero, in agreement with the test.

The single added observation changes the slope from \(2\) to \(6/7\), increases the slope standard error, and changes the test conclusion in this example. The arithmetic establishes sensitivity, not a reason to discard the observation; its validity and relevance must be assessed separately.

Intervals and Uncertainty Can Move Either Way

A changed coefficient does not by itself reveal how the interval will move. The standard error depends on both the residual variance estimate and the predictor configuration through the inverse design cross-product matrix. Removing a case can make the remaining responses fit more closely, but it can also reduce predictor spread or alter relationships among predictors. Those changes can offset one another.

The following examples use hypothetical regression output summaries. In each, the stated degrees of freedom are the ones from the indicated fit, and the displayed standard errors are used in the calculations. The summaries are sufficient for the requested coefficient inferences; they are not a claim about any real portfolio or company.

Worked Example: A Slope Loses Significance After Deletion

In a hypothetical multiple regression with an intercept and three predictors, a rating coefficient has full-fit estimate \(0.42\), standard error \(0.18\), and \(14\) residual degrees of freedom. After excluding a flagged observation and refitting, its estimate is \(0.38\), standard error \(0.22\), and \(13\) residual degrees of freedom. Compare two-sided 5% tests and 95% intervals for a zero coefficient.

Method. Recompute the statistic and interval using each fit’s own standard error and critical value. This avoids assuming that the full and deleted fits share an uncertainty estimate.

For the full fit, \(t=0.42/0.18=2.333\). With \(14\) degrees of freedom, the 95% critical value is \(2.145\), so the test rejects zero. The interval is \(0.42\pm2.145(0.18)=0.42\pm0.3861\), or \([0.034,0.806]\) to three decimals.

For the deleted fit, \(t=0.38/0.22=1.727\). With \(13\) degrees of freedom, the critical value is \(2.160\), so the test does not reject zero. Its interval is \(0.38\pm2.160(0.22)=0.38\pm0.4752\), or \([-0.095,0.855]\) to three decimals. Both the estimate and its uncertainty changed; the deleted-fit interval includes zero.

Worked Example: Deletion Can Reduce a Standard Error

A hypothetical five-coefficient regression has \(35\) observations. For one coefficient, the full-fit estimate is \(0.60\), its standard error is \(0.24\), and the residual degrees of freedom are \(30\). After deleting one observation and refitting, the estimate is \(0.72\), its standard error is \(0.20\), and the residual degrees of freedom are \(29\). Calculate the two test statistics and 95% intervals.

Method. Treat the two supplied summaries as separate fits and use the relevant \(t\) critical value for each. The common mistake avoided here is assuming that deletion must inflate uncertainty.

For the full fit, \(t=0.60/0.24=2.500\). The 95% critical value for \(30\) degrees of freedom is \(2.042\), so the test rejects zero. The interval is \(0.60\pm2.042(0.24)=0.60\pm0.49008=[0.110,1.090]\), to three decimals.

For the deleted fit, \(t=0.72/0.20=3.600\). The critical value for \(29\) degrees of freedom is \(2.045\), so it also rejects zero. Its interval is \(0.72\pm2.045(0.20)=0.72\pm0.409=[0.311,1.129]\). In this example deletion raises the estimate and reduces its standard error. This is possible, but it is not a general direction of change.

Common Exam Trap: Mixing Results From Different Fits

Do not combine a coefficient from one fit with a standard error or degrees of freedom from another. For example, in the first worked example, using the deleted slope \(2\) with the full-fit standard error gives the incorrect statistic \(2/0.780=2.564\). Comparing it with the full-fit critical value \(2.776\) would incorrectly suggest no rejection. The correct deleted-fit calculation is \(2/\sqrt{7/60}=5.855\), compared with the deleted-fit critical value \(3.182\); it rejects zero.

The error is not merely a rounding issue: the mixed calculation uses the wrong estimated uncertainty and the wrong reference distribution for the deleted coefficient. Every case-deletion inference should come from a complete refit, including the coefficient, residual variance, covariance matrix, and residual degrees of freedom.

Do not treat influence as proof of error. Check the observation’s source, units, and coding, and consider whether it represents a valid part of the target population.
Do not report only the preferred conclusion. When the substantive interpretation is sensitive to a valid case, describe that sensitivity rather than silently choosing one fit.
Do not infer that a nonsignificant result proves no effect. A test conclusion depends on estimate, uncertainty, sample size, and assumptions.

Key takeaway: Deleting an influential case changes the fitted model, so inference must be recomputed from the refit. Compare estimates, standard errors, intervals, and degrees of freedom, then investigate whether exclusion is substantively justified. A diagnostic identifies sensitivity; it does not prescribe deletion.

Check Your Understanding

For each pair of fits, use the supplied summaries and the stated two-sided 5% critical values to compare the coefficient test and interval. Round interval endpoints to three decimals.

A full fit has coefficient \(0.90\), standard error \(0.30\), and \(10\) degrees of freedom; its deleted refit has coefficient \(0.60\), standard error \(0.25\), and \(9\) degrees of freedom. Use critical values \(2.228\) and \(2.262\), respectively. Give both test conclusions and intervals. (Answer: Full \(t=3.000\), interval \([0.232,1.568]\), reject zero; deleted \(t=2.400\), interval \([0.035,1.165]\), reject zero.)
A full fit has coefficient \(0.30\), standard error \(0.16\), and \(12\) degrees of freedom; its deleted refit has coefficient \(0.25\), standard error \(0.15\), and \(11\) degrees of freedom. Use critical values \(2.179\) and \(2.201\). Does either fit reject a zero coefficient? (Answer: Full \(t=1.875<2.179\); deleted \(t=1.667<2.201\); neither rejects zero.)
A full fit has coefficient \(0.55\), standard error \(0.20\), and \(16\) degrees of freedom; its deleted refit has coefficient \(0.48\), standard error \(0.18\), and \(15\) degrees of freedom. Use critical values \(2.120\) and \(2.131\). Compare the 95% intervals and test conclusions. (Answer: Full interval \([0.126,0.974]\), \(t=2.750\), reject zero; deleted interval \([0.096,0.864]\), \(t=2.667\), reject zero.)
A full fit has coefficient \(1.10\), standard error \(0.40\), and \(8\) degrees of freedom; after deletion, the coefficient is \(0.70\), standard error \(0.35\), and \(7\) degrees of freedom. Use critical values \(2.306\) and \(2.365\). State the test conclusion for each fit. (Answer: Full \(t=2.750>2.306\), reject zero; deleted \(t=2.000<2.365\), do not reject zero.)

What You'll Learn