Influence Calculations in Simple Regression

Why Work Through Influence Calculations?

Influence diagnostics combine information about how unusual a response is and how unusual its predictor value is. Working through the calculations in a small simple regression makes that combination concrete: a case can have high leverage without a large residual, or a large residual without high leverage. Neither feature alone tells the whole story.

This tutorial uses one hypothetical dataset throughout. The calculations show what each diagnostic measures and how removing a case can change a fitted line. They are tools for investigating sensitivity, not automatic rules for deleting observations.

Model and Notation

Suppose \(n\) cases have predictor values \(x_i\) and responses \(y_i\). The simple linear regression model is

For the usual regression calculations, assume the errors are independent, have mean zero and constant variance \(\sigma^2\), and the mean response is linear in \(x_i\). Normal errors are additionally needed for exact small-sample \(t\)-based inference, but the diagnostic quantities below can be calculated without treating a diagnostic flag as a formal hypothesis test. The predictor values are treated as fixed for these calculations.

Write \(\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_i\) for the fitted response and \(e_i=y_i-\hat{y}_i\) for the residual. In simple regression with an intercept, the residual degrees of freedom are \(n-2\), and the residual mean square is \(s^2=\sum e_i^2/(n-2)\). The \(s\) here estimates the error standard deviation.

The leverage values in a simple regression with an intercept sum to 2, corresponding to the two fitted coefficients, \(\hat{\beta}_0\) and \(\hat{\beta}_1\). A case far from \(\bar{x}\) has greater leverage than one near \(\bar{x}\). Leverage indicates potential to affect the fitted line; it does not by itself show that a case actually changes the fit substantially.

A Hypothetical Dataset and Its Leverages

Consider six hypothetical observations. The response is measured in arbitrary units. The values are constructed for illustration and are not from a real study.

Studentized Residuals and Cook’s Distance

A raw residual is measured in the response units, so its size is difficult to compare across settings. An internally studentized residual scales it by an estimate of its standard deviation, taking account of its leverage. For case \(i\), it is

The factor \(\sqrt{1-h_{ii}}\) reflects that residuals at high-leverage cases tend to have smaller variance under the model. The internally studentized residual uses \(s\) calculated from the full dataset, including case \(i\).

An externally studentized deleted residual instead estimates the error scale after omitting case \(i\). Let \(s_{(i)}\) be the residual standard deviation from that reduced fit, with \(n-3\) residual degrees of freedom. Then

The two residual measures are related but not interchangeable: the external version excludes the case when estimating the error scale. Both express a residual relative to its expected scale, rather than in the original response units.

Cook’s distance summarizes how much the fitted values change when case \(i\) is deleted. For simple regression with two fitted coefficients, one common equivalent calculation is

Equivalently, it is the average squared change in fitted values over the \(n\) cases, divided by \(2s^2\). A larger value indicates greater change in the fitted line relative to the estimated residual variation. Its size is a diagnostic summary, not a probability or a direct test that the case is erroneous.

Coefficient Changes After Deletion

A direct way to understand influence is to compare the full-data coefficients with the coefficients obtained after removing a case. Denote the reduced-fit coefficients by \(\hat{\beta}_{0(i)}\) and \(\hat{\beta}_{1(i)}\). For a simple regression with an intercept, the slope difference has the identity

The intercept difference can be obtained from the same deletion update:

These expressions make the ingredients visible. A large residual can increase the coefficient changes, as can a predictor value far from its mean. The denominator \(1-h_{ii}\) also shows why deletion calculations become especially sensitive as leverage approaches 1.

Reading the Calculations Carefully

The previous example highlights why a deleted residual scale needs care: when the remaining observations fit a line perfectly, \(s_{(i)}=0\), so the externally studentized deleted residual cannot be calculated as a finite value. A formula can be evaluated only when its ingredients are defined. This example also illustrates that an internally studentized residual and a coefficient change answer different questions, even when calculated for the same observation.

In practice, calculate values from the same full-data fit and keep track of which scale is being used. Leverage depends only on \(x\); residuals depend on \(x\) and \(y\); Cook’s distance combines residual size and leverage; coefficient changes report how a particular fitted coefficient shifts after deletion. Report the predictor and response units when describing the fitted coefficients, and state that a diagnostic concerns the model fit rather than establishing a cause for the observed response.

Common Mistake

Wrong reasoning: “Case 6 has high leverage, so it must be influential and should be removed.” High leverage alone measures unusualness in the predictor, not the case’s actual effect on the fitted line. A high-leverage observation can lie close to the fitted pattern and make little difference, while a case with a less extreme predictor value can have a substantial residual.

Correction: Consider leverage together with residual size and a measure of fit change, such as Cook’s distance or the coefficient changes. Then investigate the observation’s context and the purpose and assumptions of the model before deciding how to proceed. A flagged diagnostic is a prompt for scrutiny, not a deletion instruction.