Influence in Least-Squares Regression

What Does It Mean for an Observation to Be Influential?

A fitted regression line depends on the observations used to calculate it. Some observations have little effect on the result; others can noticeably change the estimated line if they are removed. In regression, influence refers to this effect on the fitted model, not simply to whether an observation looks unusual.

To make that idea precise, compare the fit using all observations with a fit made after omitting one observation at a time. The comparison might focus on a slope, an intercept, or a prediction at a specified predictor value. An observation can have an appreciable effect on one of these quantities and a small effect on another.

Definition: An observation is influential with respect to a specified model quantity when removing that observation changes the fitted quantity appreciably. The quantity may be a regression coefficient or a prediction at a stated predictor value. Influence is therefore assessed by comparing the full-data fit with the fit that omits the observation; what counts as appreciable depends on the context and the quantity being examined.

The Model and the Comparison

For simple linear regression, let \((x_i,y_i)\) denote observation \(i\), where \(x_i\) is its predictor value and \(y_i\) its outcome. The fitted least-squares line is

\hat{y}_i=b_0+b_1x_i

Here \(b_0\) is the fitted intercept and \(b_1\) is the fitted slope. Least squares chooses the coefficients to minimize the sum of squared residuals, where the residual for observation \(i\) is \(e_i=y_i-\hat{y}_i\). For a simple linear regression with an intercept, the fitted slope and intercept can be calculated as

b_1=\frac{\sum_i(x_i-\bar{x})(y_i-\bar{y})}{\sum_i(x_i-\bar{x})^2}, \qquad b_0=\bar{y}-b_1\bar{x}.

To assess the effect of observation \(i\), fit the line again without that observation. Write the resulting coefficients as \(b_{0(-i)}\) and \(b_{1(-i)}\). For a prediction at a specified value \(x^\ast\), the full-data prediction is \(b_0+b_1x^\ast\), and the prediction without observation \(i\) is \(b_{0(-i)}+b_{1(-i)}x^\ast\). Differences between these results describe the observation’s influence on the chosen quantity.

This is a comparison of fitted results, not a hypothesis test. Calculating it does not require normally distributed errors or constant error variance. The data must, however, be suitable for fitting the regression being considered, and the model must be estimable both with and without the observation for the comparison to be meaningful. For simple linear regression, the remaining predictor values must include at least two distinct values. Whether the fitted line is an appropriate description of the mean relationship is a separate modeling question.

Worked Example: A Predictor Value Far from the Rest Can Pull the Line

Suppose four hypothetical observations are \((1,1),(2,2),(3,3),(4,4)\). A fifth observation, \((10,0)\), is added. Compare the fitted line with the fit that omits this fifth observation.

With the first four observations alone, every point satisfies \(y=x\), so the least-squares line is \(\hat{y}=x\), with \(b_0=0\) and \(b_1=1\).

For all five observations, \(\bar{x}=20/5=4\), \(\bar{y}=10/5=2\), \(\sum x_i^2=130\), and \(\sum x_iy_i=30\). Therefore,

\sum_i(x_i-\bar{x})^2=130-5(4^2)=50, \qquad \sum_i(x_i-\bar{x})(y_i-\bar{y})=30-5(4)(2)=-10.

The fitted slope and intercept are

b_1=\frac{-10}{50}=-0.2, \qquad b_0=2-(-0.2)(4)=2.8.

Including \((10,0)\) changes the slope from \(1\) to \(-0.2\) and the intercept from \(0\) to \(2.8\). At \(x=4\), the prediction changes from \(4\) to \(2.8-0.2(4)=2\), a decrease of \(2\) outcome units. In this example, the observation substantially affects both the fitted coefficients and a prediction within the range of the first four predictor values.

The predictor value 10 is also far from the others. That feature gives the observation room to affect the line’s slope, but being far from the other predictor values does not, by itself, prove that it will be influential. Its outcome value matters too.

Influence Is Not the Same as Unusualness

An observation may look unusual in different ways. Its predictor value may be far from the other predictor values, or its outcome may be far from the outcome predicted by the fitted line. These features can help explain an observation’s behavior, but neither one is the definition of influence. Influence is about the change in the fit when the observation is removed.

For example, a point with an unusual predictor value can lie exactly on the same line as all the other observations. If removing it leaves the same line estimable, it may have no effect on the fitted coefficients. Conversely, an observation with a large residual may change some fitted quantities only modestly, especially when many observations contribute to the fit. The size of an observed outcome alone is not a substitute for comparing the fits.

Worked Example: A Large Residual Does Not Determine the Change in the Slope

Consider 11 hypothetical observations with predictor values \(-5,-4,\ldots,4,5\). For each nonzero \(x\), let \(y=x\); at \(x=0\), let \(y=10\). Compare the full-data fit with the fit that omits \((0,10)\).

For the full data, \(\bar{x}=0\) and \(\bar{y}=10/11\). Also, \(\sum x_i^2=110\) and \(\sum x_iy_i=110\), since the observation at \(x=0\) contributes zero to both sums. Thus,

b_1=\frac{110}{110}=1, \qquad b_0=\frac{10}{11}-1(0)=\frac{10}{11}\approx 0.9091.

The fitted value at \(x=0\) is \(10/11\), so the residual for \((0,10)\) is

e=10-\frac{10}{11}=\frac{100}{11}\approx 9.091.

That is a large vertical difference between the observed outcome and the fitted value. Yet if \((0,10)\) is removed, the remaining observations all satisfy \(y=x\). Their fitted line has slope \(1\) and intercept \(0\). Removing the observation changes the slope by \(0\) and the intercept by about \(-0.9091\). At \(x=5\), the prediction changes from \(10/11+5\approx 5.909\) to \(5\), a decrease of about \(0.9091\) outcome units.

The observation has a large residual, but the change in slope is zero, and the prediction change at \(x=5\) is much smaller than the residual. This illustrates why residual size and influence answer different questions. It does not mean the intercept or prediction change is always negligible; their importance depends on the context and units.

Why Omitting a Point Can Change the Fit

The reason deletion can matter follows directly from the least-squares objective. For proposed coefficients \(a_0\) and \(a_1\), the full-data sum of squared residuals is

Q(a_0,a_1)=\sum_j\left[y_j-(a_0+a_1x_j)\right]^2.

If observation \(i\) is removed, its squared-error contribution is removed from the objective:

Q_{(-i)}(a_0,a_1) = Q(a_0,a_1)-\left[y_i-(a_0+a_1x_i)\right]^2.

The coefficients are chosen to minimize the objective. Changing the objective can therefore change the minimizing coefficients and the predictions made from them. If an observation has residual zero at the full-data fit, its squared-error term has zero derivative there, so removing it does not change the objective’s first derivative at those coefficients. When the remaining data still give a unique least-squares fit, the coefficients are then unchanged. This is one reason an unusual predictor value need not be influential: if it lies exactly on the fitted line, it can leave the fitted coefficients unchanged when omitted.

Worked Example: Influence Depends on the Quantity Examined

Consider five hypothetical observations: \((0,1),(1,2),(2,2),(3,4),(4,5)\). First fit all five, then compare the effect of omitting the middle observation \((2,2)\) with the effect of omitting the endpoint \((4,5)\).

For the full data, \(\bar{x}=2\), \(\bar{y}=14/5=2.8\), \(\sum x_i^2=30\), and \(\sum x_iy_i=38\). Hence,

\sum_i(x_i-\bar{x})^2=30-5(2^2)=10, \qquad \sum_i(x_i-\bar{x})(y_i-\bar{y})=38-5(2)(2.8)=10.

It follows that the full-data fitted line has \(b_1=10/10=1\) and \(b_0=2.8-1(2)=0.8\). Its prediction at \(x=4\) is \(0.8+4=4.8\).

After omitting \((2,2)\), the remaining points are \((0,1),(1,2),(3,4),(4,5)\), all of which lie on \(\hat{y}=1+x\). The slope stays at \(1\), the intercept changes from \(0.8\) to \(1\), and the prediction at \(x=4\) changes from \(4.8\) to \(5\), an increase of \(0.2\).

After omitting \((4,5)\), the remaining observations have \(n=4\), \(\bar{x}=1.5\), \(\bar{y}=2.25\), \(\sum x_i^2=14\), and \(\sum x_iy_i=18\). Therefore,

\sum_i(x_i-\bar{x})^2=14-4(1.5^2)=5, \qquad \sum_i(x_i-\bar{x})(y_i-\bar{y})=18-4(1.5)(2.25)=4.5.

The resulting slope is \(4.5/5=0.9\), and the intercept is \(2.25-0.9(1.5)=0.9\). The prediction at \(x=4\) is now \(0.9+0.9(4)=4.5\), a decrease of \(0.3\) from the full-data prediction.

The middle observation’s full-data residual is \(2-(0.8+1(2))=-0.8\); the endpoint’s residual is \(5-(0.8+1(4))=0.2\). Omitting the middle point changes the intercept but not the slope. Omitting the endpoint changes both coefficients and gives a somewhat larger change in the prediction at \(x=4\). Thus, the assessment depends on which coefficient or prediction matters, not just on which residual is larger.

Using Influence Carefully

Influence comparisons matter because conclusions about a fitted relationship can depend on particular observations. If omitting one case changes the slope substantially, the description of how the outcome changes with the predictor may be sensitive to that case. If it changes a prediction at a value relevant to the application, that prediction deserves careful attention. Reporting the full fit without examining such sensitivity can hide how much it depends on individual data points.

A deletion comparison is a diagnostic, not an automatic instruction to discard an observation. First check whether the value was recorded correctly and whether the observation belongs to the population and setting being analyzed. If the observation is valid, removing it simply because it changes the result can misrepresent the data. Instead, report the sensitivity when relevant and consider whether the chosen regression model adequately describes the pattern.

Common mistake: “The point has an extreme outcome, so it must be influential.” An extreme outcome may have a large residual, but influence is determined by how fitted coefficients or predictions change when the point is omitted.
Common mistake: “The predictor value is unusual, so the point must be influential.” An unusual predictor value can affect the fit, but its outcome’s position relative to the line matters too. A point lying on the fitted line can leave the coefficients unchanged when removed, provided the remaining fit is unique.
Common mistake: “Influential means incorrect, so delete it.” Influence describes sensitivity of the fit, not data quality. Verify the observation and explain the sensitivity rather than removing a valid case without justification.
Common mistake: “One influence judgment applies to every result.” A point can affect a slope, intercept, and particular prediction by different amounts. State what quantity is being compared and interpret the change in context and units.

Key takeaway: Influence is a change-in-fit idea. Compare the coefficients or predictions from the full data with those from a fit omitting the observation, then interpret the difference for the specific quantity that matters.

Check Your Understanding

For each question, focus on the difference between a point’s appearance and its effect on a fitted result.

In simple linear regression, what comparison gives a direct assessment of one observation’s influence on the slope?
Can an observation have a large residual but produce no change in the fitted slope when omitted? Explain how the two quantities differ.
Why might a predictor value far from the others have the potential to affect a fitted line without necessarily changing it?
Suppose omitting a point leaves the slope unchanged but shifts the intercept. Is it correct to say the point has no influence? Explain.
Why is influence, by itself, not a valid reason to delete an observation?