Response Outliers, Leverage, and Influence

Three Different Ways a Case Can Be Unusual

A regression data set can contain observations that stand out for different reasons. One case may have an unusual response compared with the model’s prediction at its predictor values. Another may have an unusual combination of predictor values, even if its response is close to the fitted model. A third may substantially alter the fitted regression when it is included. These features are related, but they are not interchangeable: they describe, respectively, an unusual response, an unusual predictor profile, and an effect on the fitted model.

Keeping the distinctions clear matters in applied work. A case that is far out in predictor space is not automatically a response outlier, and a large residual does not by itself tell us how much the fitted model changes. The labels are prompts for investigation, not reasons on their own to discard data.

For the calculations here, consider the usual least-squares regression model with \(n\) observations, an intercept, and \(m\) predictors. Write \(y_i\) for the response and \(\mathbf{x}_i\) for the predictor values of case \(i\). The model assumes a linear conditional mean, independent errors with mean zero and constant variance, and a correctly recorded set of observations. Normal errors are not required for the descriptive leverage calculations or for fitting the least-squares line. The examples treat the predictor values as fixed for the calculations.

Response Outliers and Predictor Profiles

The fitted value for case \(i\), written \(\hat{y}_i\), is the response predicted by the fitted model at that case’s predictor values. The residual is \(e_i=y_i-\hat{y}_i\). A response outlier has a response that departs unusually from the pattern represented by the regression, so a large residual can be a sign of one. The word “unusually” is contextual: a residual’s size must be judged in relation to the response scale and the rest of the data. A residual is not, by itself, a proof that a record is erroneous or that a case should be removed.

Leverage is about predictor values, not the response. In a model with an intercept, the leverage of case \(i\), denoted \(h_{ii}\), is the \(i\)th diagonal entry of the hat matrix \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), where \(X\) is the design matrix containing the intercept and predictors. This definition assumes that the columns of \(X\) are linearly independent, so the inverse exists. Leverage describes how unusual a case’s predictor profile is relative to the design of the observed data; it does not directly measure how far the response is from its fitted value.

For a simple linear regression with one predictor \(x\) and an intercept, leverage has a useful direct formula. Let \(\bar{x}\) be the sample mean of the predictor and \(S_{xx}=\sum_{j=1}^n(x_j-\bar{x})^2\). Then

This formula shows why predictor values far from \(\bar{x}\) tend to have high leverage. In a multiple-predictor model, the same idea applies to the whole predictor profile: unusual combinations can have high leverage even when no single predictor value looks extreme on its own.

Average Leverage and Practical Guidelines

The diagonal leverage values have a useful average. If the model has an intercept and \(m\) predictors, the design matrix has \(m+1\) columns. The sum of the leverages equals the number of columns, so the average leverage is \((m+1)/n\). This provides a reference for identifying cases whose leverages are comparatively large.

The result follows from the definition of the hat matrix. The sum of its diagonal entries is its trace, and the trace of a product can be rearranged:

Dividing this sum by \(n\) gives the average, \((m+1)/n\). In practice, values greater than \(2(m+1)/n\), or sometimes \(3(m+1)/n\), are used as screening guidelines for high leverage. These are heuristics, not universal boundaries: a value above a guideline calls for attention, not an automatic classification as a problem case. The number of predictors, sample size, and the structure of the predictor data all matter.

Influence: What Changes When a Case Matters?

Influence concerns the effect of a case on the fitted model. A direct conceptual check is to compare the model fit with and without the case, asking whether coefficients or predictions change in a practically important way. A common summary for this purpose is Cook’s distance, \(D_i\). In a least-squares model with an intercept, \(m\) predictors, and residual mean square \(MSE\), it is

Cook’s distance combines two ingredients: the size of the residual and the leverage. A large residual can contribute to influence, as can high leverage; their combination can make a case influential even when neither feature alone tells the whole story. The statistic is a diagnostic summary, not a probability, and there is no single cutoff that is appropriate in every application.

Why the Distinction Matters

A case can occupy different positions on the three dimensions. A case with an ordinary predictor profile and a large response departure may be a response outlier without having high leverage. A case far from the center of predictor space may have high leverage but lie close to the fitted response pattern. A case may be influential when its residual and leverage combine to shift the fitted line or important predictions. None of these labels, by itself, identifies an error.

This distinction also helps keep investigation proportional. Check whether measurements and records are correct, consider whether the model represents the setting adequately, and examine whether conclusions depend on the case. If a case is valid and important to the population or range of interest, deleting it simply because it changes the fit can make the analysis less informative. Influence is a reason to understand sensitivity, not a license for automatic removal.