High-Leverage Observations and OLS Sensitivity

Why One Observation Can Matter

An observation may lie far from the others in predictor space yet sit close to the fitted regression line. Such an observation has high leverage but may have a small residual. Because its predictor values help determine the fitted line, removing it can still change the estimated coefficients substantially. Conversely, an observation can have an unusual outcome and a large residual but modest leverage, so it may have a smaller effect on the fitted coefficients.

The distinction matters when interpreting a regression: residual size alone does not measure how sensitive the coefficient estimates are to individual observations. Deletion and perturbation calculations make that sensitivity explicit. They describe what the fitted OLS coefficients would do under a specified change to the data; they do not by themselves establish that an observation is erroneous or that a regression relationship is causal.

Definition: In a linear regression with design matrix $X$ of full column rank, the OLS estimate is $\hat{\beta}=(X'X)^{-1}X'y$. For observation $i$, let $x_i'$ be row $i$ of $X$, $e_i=y_i-x_i'\hat{\beta}$ its full-sample residual, and $h_{ii}=x_i'(X'X)^{-1}x_i$ its leverage. Leverage depends on the predictors, including any intercept column; the residual depends on both the outcome and the fitted model.

The hat matrix is $H=X(X'X)^{-1}X'$, so $h_{ii}$ is its $i$th diagonal element. Leverage values are between 0 and 1 and sum to the number of columns in $X$, say $k$. Thus their average is $k/n$. A value notably larger than that average can flag an observation for inspection, but there is no universal cutoff that establishes a problem. High leverage is not the same thing as an unusual outcome: it is a property of the predictor configuration.

Deleting One Observation

Write $\hat{\beta}_{(-i)}$ for the OLS coefficient vector computed after deleting observation $i$. The following identity gives the exact coefficient change when the remaining design matrix has full column rank. If $h_{ii}=1$, deleting the observation makes the design matrix rank deficient, so this deletion formula does not apply.

Theorem: If $X$ has full column rank and $h_{ii}<1$, then $$ \hat{\beta}_{(-i)}-\hat{\beta} =-\frac{(X'X)^{-1}x_i e_i}{1-h_{ii}}. $$ The formula is a vector: its entries give the change in each estimated coefficient when observation $i$ is removed.

The numerator combines two features. The residual $e_i$ measures how far the observed outcome is from its full-sample fitted value. The vector $(X'X)^{-1}x_i$ describes how the observation’s predictor values align with the coefficient directions. The denominator $1-h_{ii}$ can be small for a high-leverage observation, magnifying the deletion change. A small residual can therefore coexist with a large change in coefficients.

Why the Deletion Formula Holds

Let $A=X'X$. After deleting row $i$, the cross-product matrix is $A-x_ix_i'$, and the outcome cross-product vector is $X'y-x_i y_i$. Consider the proposed deleted estimate $\hat{\beta}-A^{-1}x_i e_i/(1-h_{ii})$. Multiplying it by the deleted cross-product matrix gives

\begin{aligned} (A-x_ix_i')\left(\hat{\beta}-\frac{A^{-1}x_i e_i}{1-h_{ii}}\right) &=A\hat{\beta}-x_i x_i'\hat{\beta} -\frac{x_i e_i}{1-h_{ii}} +\frac{x_i h_{ii}e_i}{1-h_{ii}}\\ &=X'y-x_i x_i'\hat{\beta}-x_i e_i\\ &=X'y-x_i y_i. \end{aligned}

The first equality uses $x_i'A^{-1}x_i=h_{ii}$; the last uses $e_i=y_i-x_i'\hat{\beta}$. Thus the proposed vector solves the normal equations for the data with observation $i$ deleted. Since $1-h_{ii}>0$ and the deleted design is full rank, this solution is the deleted-sample OLS estimate.

For simple linear regression with an intercept and one predictor, let $\bar{x}$ be the sample mean of the predictor and let $S_{xx}=\sum_j(x_j-\bar{x})^2$. Then

h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{S_{xx}}.

The slope component of the deletion formula is $- (x_i-\bar{x})e_i/\{S_{xx}(1-h_{ii})\}$. This makes clear that, in simple regression, leverage depends on how far $x_i$ is from the predictor mean, while the residual indicates how unusual $y_i$ is relative to the fitted line.

Worked Example: High Leverage With a Small Residual

Consider the hypothetical observations $(x_i,y_i)=(0,1),(1,3),(2,5),(10,7)$, fit by simple linear regression with an intercept. The design has full rank. For the full sample, $n=4$, $\bar{x}=3.25$, $\bar{y}=4$, $S_{xx}=62.75$, and $\sum_i(x_i-\bar{x})(y_i-\bar{y})=31$. Therefore the slope and intercept are

\hat{\beta}_1=\frac{31}{62.75}\approx0.494024,\qquad \hat{\beta}_0=4-3.25(0.494024)\approx2.394422.

For the observation at $x_i=10$, the fitted value is $2.394422+10(0.494024)\approx7.334661$, so its residual is $e_i=7-7.334661\approx-0.334661$. Its leverage is

h_{ii}=\frac{1}{4}+\frac{(10-3.25)^2}{62.75} =\frac{245}{251}\approx0.976096.

This leverage is near 1, while the full-sample residual is small in the outcome’s units. Deleting the observation leaves the three points $(0,1),(1,3),(2,5)$, which lie exactly on the line $\hat{y}=1+2x$. The slope changes from approximately $0.494024$ to $2$, a change of approximately $1.505976$; the intercept changes from approximately $2.394422$ to $1$. The deletion formula gives the slope change directly:

-\frac{(10-3.25)(-0.334661)}{62.75(1-0.976096)} \approx1.505976.

The observation’s predictor value is far from the others, so it has strong influence on the fitted slope despite its small full-sample residual. In the deleted fit, its outcome is 14 units below the prediction at $x=10$, illustrating why the full-sample residual can understate the discrepancy revealed by deletion.

Changing an Outcome Instead of Deleting It

Deletion is not the only useful sensitivity calculation. Suppose the predictor values stay fixed, but the outcome for observation $i$ is changed from $y_i$ to $y_i+\delta$. The OLS coefficient vector changes exactly by

Result: For an outcome change of $\delta$ at observation $i$, with $X$ unchanged, $$ \hat{\beta}^{\,*}-\hat{\beta}=(X'X)^{-1}x_i\delta. $$

This follows because the outcome cross-product vector $X'y$ changes to $X'y+x_i\delta$, while $X'X$ stays fixed. Unlike deletion, the perturbation formula has no $1-h_{ii}$ denominator. In simple regression, changing $y_i$ by $\delta$ changes the slope by $(x_i-\bar{x})\delta/S_{xx}$. The intercept changes by $\delta\{1/n-\bar{x}(x_i-\bar{x})/S_{xx}\}$. These expressions show how predictor location determines which coefficients are affected.

Worked Example: A One-Unit Outcome Perturbation

Use the four observations from the preceding example and increase the outcome at $x=10$ by one unit, from 7 to 8. Here $\delta=1$, $x_i-\bar{x}=6.75$, and $S_{xx}=62.75$. The slope change is

\Delta\hat{\beta}_1=\frac{6.75(1)}{62.75}\approx0.107570.

The intercept change is

\Delta\hat{\beta}_0 =\frac{1}{4}-\frac{(3.25)(6.75)}{62.75} =0.25-\frac{21.9375}{62.75} \approx0.25-0.349602 =-0.099602.

Thus the perturbed fit has slope approximately $0.601594$ and intercept approximately $2.294820$. The values are obtained by adding the changes to the original coefficients $0.494024$ and $2.394422$, respectively. The one-unit outcome change moves the slope because the observation is far from the predictor mean; its intercept change has the opposite sign because of the observation’s position relative to both the predictor mean and the intercept term.

Worked Example: The Same Outcome Change at Different Predictor Values

Consider the hypothetical data $(-2,-3),(-1,-1),(0,1),(1,3),(2,5)$, fit with an intercept. They lie on $\hat{y}=1+2x$. Here $\bar{x}=0$, $S_{xx}=10$, and each leverage is $1/5+x_i^2/10$. The observation at $x=0$ has leverage $0.2$; the observation at $x=2$ has leverage $0.6$.

Increase the outcome of one observation by three units. At $x=0$, the slope change is $0(3)/10=0$, and the intercept change is $3/5=0.6$. The new coefficients are intercept $1.6$ and slope $2$. At $x=2$, the slope change is $2(3)/10=0.6$, and the intercept change is again $3/5=0.6$. The new coefficients are intercept $1.6$ and slope $2.6$. The outcomes at both points began exactly on the fitted line, so both initial residuals were zero. Nevertheless, the same outcome perturbation changes the slope more at $x=2$, the higher-leverage point.

This example isolates predictor location: the changed outcome has the same size in both cases, but a point far from $\bar{x}$ has more capacity to rotate the fitted line. Leverage is not itself a coefficient change; the perturbation’s effect also depends on the direction and size of the change.

How to Use Sensitivity Calculations

A disciplined assessment separates three questions: how unusual are the predictor values, how far is the outcome from the fitted value, and how much do the coefficients change under a stated deletion or perturbation? Leverage addresses the first question, the full-sample residual addresses the second, and the sensitivity formulas address the third. None can be substituted for the others.

Deletion is a diagnostic comparison, not an instruction to remove observations. A high-leverage point may be a valid and important part of the population. Before changing an analysis, check data entry, units, coding, and the scientific context, and report the sensitivity transparently. Coefficient changes describe the fitted sample relationship; they do not by themselves diagnose model misspecification or quantify sampling uncertainty.

Common Mistakes

Equating a small residual with low influence. The reasoning “the point is close to the line, so deleting it cannot matter” ignores the factor $1/(1-h_{ii})$ and the predictor direction in the deletion formula. Check both leverage and the coefficient change.
Calling high leverage an outcome outlier. Leverage is determined by the predictors; residual size concerns the outcome relative to the fitted model. A point can have high leverage and a small residual, or low leverage and a large residual.
Assuming a large deletion change proves the observation is invalid. The calculation establishes sensitivity, not a data error. Verify the observation and explain how conclusions depend on its inclusion.
Applying the deletion formula when $h_{ii}=1$. In that case $1-h_{ii}=0$, and deleting the observation leaves insufficient predictor rank for the stated coefficient vector. The formula requires $h_{ii}<1$.

Key takeaway: A high-leverage observation can help determine the fitted line and consequently have a small residual. Use leverage, residuals, and explicit deletion or perturbation calculations together to describe OLS sensitivity accurately.

Check Your Understanding

Answer each question using the definitions and formulas above.

In a regression with an intercept and two predictors, what determines an observation’s leverage, and what is the average diagonal leverage when the design matrix has full column rank?
Why can deleting a high-leverage observation produce a substantial coefficient change even when its full-sample residual is small?
For simple regression with $n=5$, $\bar{x}=0$, and $S_{xx}=10$, calculate the leverage of an observation at $x=2$.
If an outcome at $x_i$ is increased by $\delta$, state the general formula for the resulting OLS coefficient change when the design matrix is fixed.
Why does a large sensitivity calculation not, by itself, justify removing an observation or making a causal claim?