Leverage and Influence in Multiple Regression

Why Residual Size Is Not the Whole Story

A multiple regression observation can stand out in two different ways. Its response may be far from the value predicted by the fitted model, giving it a large residual. Or its combination of predictor values may be unusual, even if its response lies close to the fitted regression surface. The first feature concerns the response after accounting for the predictors; the second concerns the observation’s position among the predictors.

These features matter because observations in unusual predictor locations can have considerable potential to change a fitted regression. Whether they actually do so depends also on their responses. Leverage measures how unusual an observation’s predictor values are, while influence describes how much the fitted results change when that observation is removed. A high-leverage point need not be influential, and an observation with a large residual need not have high leverage.

The Regression Setup and Leverage

For $n$ observations, write the multiple linear regression model in matrix form as $Y=X\beta+\varepsilon$. Here $Y$ is the vector of responses, $X$ is the design matrix containing the predictor values (and typically a column of ones for the intercept), and $\beta$ is the vector of regression coefficients. Let $p$ be the number of estimated coefficients, including the intercept. The usual least-squares fit is $\hat{Y}=HY$, where $H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$, assuming the columns of $X$ are linearly independent.

Definition: The leverage of observation $i$ is the diagonal element $h_{ii}$ of the hat matrix $H$. It depends on the design matrix $X$, so it reflects the observation’s predictor values, not its response. With an intercept and a full-rank design, $0\le h_{ii}\le1$, and the average leverage is $p/n$.

The ordinary residual is $e_i=y_i-\hat{y}_i$. A large residual signals that the observed response differs substantially from its fitted value, but raw residuals do not account for the fact that residuals have different variances at different predictor locations. Under the usual linear model with independent errors of mean zero and common variance $\sigma^2$, the variance of $e_i$, conditional on $X$, is $\sigma^2(1-h_{ii})$. Thus, a residual at a high-leverage point tends to have a smaller variance than one at an average-leverage point.

The regression model and these diagnostic measures are most useful when the model form and data recording are being treated seriously: the response is quantitative, the mean response is represented adequately by the predictors and chosen terms, and the errors are independent with constant variance. Normal errors are additionally used for exact small-sample inference based on studentized residuals. Diagnostics can help reveal problems with these assumptions; a diagnostic flag alone does not prove which assumption, if any, has failed.

Worked Example: A Large Residual or High Leverage?

Suppose a hypothetical regression with $n=40$ observations and $p=4$ coefficients has residual mean square $MSE=9$, so $s=\sqrt{9}=3$. Two observations have the following fitted diagnostics: observation A has residual $e_A=9$ and leverage $h_{AA}=0.06$; observation B has residual $e_B=2$ and leverage $h_{BB}=0.30$. Compare their internally studentized residuals and leverages.

The internally studentized residual is $r_i=e_i/[s\sqrt{1-h_{ii}}]$. For A,

r_A=\frac{9}{3\sqrt{1-0.06}}=\frac{9}{3\sqrt{0.94}}\approx 3.094.

A has a large residual relative to its estimated residual standard deviation. Its leverage, $0.06$, is below the average $p/n=4/40=0.10$. This is a response outlier candidate, not a high-leverage observation.

For B,

r_B=\frac{2}{3\sqrt{1-0.30}}=\frac{2}{3\sqrt{0.70}}\approx 0.7968.

B’s response is not especially unusual relative to the fitted value, but its leverage of $0.30$ is three times the average. It is a high-leverage observation, though this comparison alone does not establish that it changes the fitted coefficients substantially.

Studentized Residuals and Deleted Fits

The internally studentized residual $r_i$ scales the ordinary residual using the full-model estimate $s=\sqrt{MSE}$, where $MSE=SSE/(n-p)$ and $SSE=\sum_i e_i^2$. An externally studentized residual, also called a deleted studentized residual, uses a residual variance estimate calculated after removing observation $i$. Write $s_{(i)}^2$ for that deleted-fit estimate.

Definition: The externally studentized residual for observation $i$ is $$ t_i=\frac{e_i}{s_{(i)}\sqrt{1-h_{ii}}}, \qquad s_{(i)}^2=\frac{SSE-e_i^2/(1-h_{ii})}{n-p-1}. $$ When the regression errors are independent and normally distributed with mean zero and constant variance, $t_i$ has a $t$ distribution with $n-p-1$ degrees of freedom under the model.

The deleted-fit formula uses the fact that the residual for observation $i$ from a model fitted without $i$ is $e_i/(1-h_{ii})$. One way to see why is to apply the one-observation update for least squares: fitting the full model changes the deleted-fit coefficients by a term proportional to $x_i$, the design row for observation $i$. In particular, $\hat{\beta}=\hat{\beta}_{(i)}+(X^\mathsf{T}X)^{-1}x_i^\mathsf{T}e_i/(1-h_{ii})$. Multiplying by $x_i$ gives $\hat{y}_i-\hat{y}_{i(i)}=h_{ii}e_i/(1-h_{ii})$, so $y_i-\hat{y}_{i(i)}=e_i/(1-h_{ii})$. Squaring this deleted residual gives $e_i^2/(1-h_{ii})^2$. The least-squares deletion identity gives $SSE_{(i)}=SSE-e_i^2/(1-h_{ii})$, so the amount removed from $SSE$ when observation $i$ is left out is $e_i^2/(1-h_{ii})$, which is $(1-h_{ii})$ times the squared deleted residual.

Studentization helps compare residuals across observations, but it does not measure the full change in every fitted coefficient or fitted value. Influence measures address that separate question by comparing the original fit with a fit that omits the observation.

Worked Example: Calculating Studentization and Cook’s Distance

Consider hypothetical diagnostic output from a regression with $n=6$ observations and $p=2$ coefficients. For observation $i$, suppose $e_i=1$, $h_{ii}=11/21\approx0.5238$, and the full-model $MSE=3$. The deleted-fit mean square is $s_{(i)}^2=3.300$. These summaries are internally consistent: the full $SSE$ is $3(6-2)=12$, and the deleted $SSE$ is $12-1^2/(1-11/21)=9.9$, which divided by $6-2-1=3$ gives $3.300$.

First, the internally studentized residual is

r_i=\frac{1}{\sqrt{3}\sqrt{1-11/21}} =\frac{1}{\sqrt{3}\sqrt{10/21}} \approx 0.8367.

The externally studentized residual is

t_i=\frac{1}{\sqrt{3.300}\sqrt{10/21}}\approx 0.7977.

Next, Cook’s distance measures the total change in fitted values when $i$ is deleted, relative to the full-model residual scale. Its formula is $D_i=e_i^2h_{ii}/[p\,MSE(1-h_{ii})^2]$. Therefore,

D_i=\frac{1^2}{2(3)} \frac{11/21}{(10/21)^2} =\frac{1}{6}(2.31)=0.385.

DFFITS measures the change in the fitted value for observation $i$, in units of its deleted-fit standard error. It can be computed as $t_i\sqrt{h_{ii}/(1-h_{ii})}$, giving

\mathrm{DFFITS}_i =0.7977\sqrt{\frac{11}{10}} \approx 0.8367.

These measures give different views of the case: its leverage is above the average $p/n=2/6$, while its studentized residual is less than one in magnitude. Cook’s distance and DFFITS do not, by themselves, call for deleting it. The calculations identify a case to inspect, not a verdict about its validity or importance.

Influence Measures for Coefficients and Fitted Values

Cook’s distance summarizes how much the fitted regression surface changes overall. If $\hat{y}_{j(i)}$ is the fitted value for observation $j$ from the model that excludes observation $i$, then

D_i=\frac{\sum_{j=1}^{n}(\hat{y}_j-\hat{y}_{j(i)})^2}{p\,MSE} =\frac{e_i^2}{p\,MSE}\frac{h_{ii}}{(1-h_{ii})^2}.

The second expression shows why neither a large residual nor high leverage alone determines Cook’s distance: both $e_i^2$ and $h_{ii}$ contribute. DFFITS focuses on the change in the fitted value for the observation being examined. DFBETAS instead focuses on the change in an individual coefficient. For coefficient $j$, one common definition is

\mathrm{DFBETAS}_{ij} =\frac{\hat{\beta}_j-\hat{\beta}_{j(i)}}{s_{(i)}\sqrt{c_{jj}}},

where $\hat{\beta}_{j(i)}$ is the coefficient from the fit with observation $i$ removed, and $c_{jj}$ is the $j$th diagonal element of $(X^\mathsf{T}X)^{-1}$ for the full fit. DFBETAS is expressed in standard-error units; its sign indicates the direction of the coefficient change under this convention. Each measure answers a different question, so examining more than one can be informative.

Worked Example: A Case That Changes One Coefficient

A hypothetical regression uses $n=20$ observations and $p=4$ coefficients. Software reports that omitting observation $i$ changes one predictor’s coefficient from $\hat{\beta}_j=2.40$ to $\hat{\beta}_{j(i)}=1.70$. For this coefficient, suppose $s_{(i)}=0.50$ and $\sqrt{c_{jj}}=0.20$. The coefficient change is $2.40-1.70=0.70$, and its standardizing denominator is $0.50(0.20)=0.10$. Thus,

\mathrm{DFBETAS}_{ij} =\frac{2.40-1.70}{0.50(0.20)} =\frac{0.70}{0.10}=7.00.

This is a large change in the positive direction under the stated DFBETAS convention: the coefficient estimated with the observation included is higher than the coefficient estimated without it. Suppose the same output gives $D_i=0.62$. The rough Cook’s distance guideline $4/(n-p)$ is $4/(20-4)=0.25$, so $0.62$ merits investigation. Neither value establishes that the observation is erroneous or that the coefficient without it is preferable. The analyst should verify the recorded values, examine the predictor combination and response, and report how conclusions depend on including the case.

Using Guidelines Carefully

Common screening guidelines include flagging leverage above $2p/n$ or $3p/n$, and examining observations with $|r_i|$ or $|t_i|$ around 2 or 3 and above. A frequently used Cook’s distance screen is $4/(n-p)$; a rough DFFITS screen is $2\sqrt{p/n}$. These are practical prompts for review, not universal cutoffs, hypothesis tests, or rules for deleting observations. Their usefulness depends on the sample size, number of predictors, model purpose, and the pattern of diagnostics.

If a case is flagged, first verify its data and units. Then examine which predictors make it unusual and whether the fitted relationship is plausible in that region. Compare the original results with a fit excluding the observation as a sensitivity analysis, and explain any important differences. Exclusion requires a substantive reason, such as a confirmed recording error or a clearly defined analysis rule; removing a valid observation simply because it affects the result can distort the analysis.

Common Mistake

Wrong reasoning: “This observation has high leverage, so it must be an outlier and should be removed.”
Correction: Leverage concerns the predictor values, while a response outlier has a large residual relative to its fitted value. Influence concerns the change in the fit after deletion. Check residual and influence diagnostics together, investigate the observation in context, and do not treat a screening threshold as a deletion rule.

The distinction matters because one observation can be unusual in one respect but not another. A point with an ordinary residual can still deserve attention if its predictor combination is rare and it changes important fitted results. Conversely, a large residual at an ordinary predictor location may have limited influence on the coefficients. Diagnostics organize this investigation; they do not replace judgment about the data and model.

Key takeaway: Residual size concerns how far the response is from its fitted value; leverage concerns how unusual the predictors are; influence concerns how much the fitted model changes. Studentized residuals, Cook’s distance, DFFITS, and DFBETAS help examine these distinct features.

Check Your Understanding

Try these questions without looking back.

In a regression with $n=50$ observations and $p=5$ coefficients, what is the average leverage? What does leverage measure?
An observation has high leverage but an externally studentized residual close to zero. What can you conclude, and what can you not conclude?
Why does an externally studentized residual use a variance estimate calculated after deleting the observation?
What does Cook’s distance summarize, and why can neither a large residual nor high leverage alone determine its value?
A case exceeds a common Cook’s distance guideline. Name two sensible steps to take before deciding whether to exclude it.