Leverage and Unusual Predictor Values

Why Predictor Values Matter

A regression line is estimated from the observed pairs of predictor and response values. Most observations may have predictor values clustered together, while one observation lies far from that cluster. Because the line must account for all the data, an observation at an unusual predictor value has the potential to affect the fitted line substantially. Leverage measures this unusualness in the predictor values.

Leverage is not a measure of how far an observed response is from the fitted line. That is a different question, addressed by residuals and studentized residuals. A point can have high leverage and a small residual, low leverage and a large residual, or both. This tutorial focuses on leverage in simple linear regression with an intercept.

Definition: For observation $i$ in a simple linear regression with an intercept, the leverage, or hat value, is $$ h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{S_{xx}}, \qquad S_{xx}=\sum_{j=1}^{n}(x_j-\bar{x})^2. $$ It measures how unusual $x_i$ is relative to the observed predictor values. A hat value is dimensionless and depends only on the predictor values and the sample size, not on the response values.

The subscript $ii$ comes from the “hat matrix,” which maps the observed responses to their least-squares fitted values. You do not need to construct that matrix to use the formula above. The formula shows directly that observations farther from $\bar{x}$, relative to the overall spread $S_{xx}$, have larger leverage.

Where the Hat-Value Formula Comes From

The derivation also explains why leverage is about predictor unusualness rather than response unusualness. In simple linear regression with an intercept, the fitted value for observation $i$ is $\hat{y}_i=\bar{x}+b_1(x_i-\bar{x})$, where $b_1$ is the least-squares slope. The slope is $b_1=\sum_{j=1}^{n}(x_j-\bar{x})(y_j-\bar{y})/S_{xx}$. Substituting this slope and writing the sample mean response as a sum gives:

\begin{aligned} \hat{y}_i &=\bar{y}+(x_i-\bar{x})b_1\\ &=\frac{1}{n}\sum_{j=1}^{n}y_j +\frac{x_i-\bar{x}}{S_{xx}} \sum_{j=1}^{n}(x_j-\bar{x})y_j\\ &=\sum_{j=1}^{n} \left[ \frac{1}{n}+ \frac{(x_i-\bar{x})(x_j-\bar{x})}{S_{xx}} \right]y_j. \end{aligned}

Thus each fitted value is a weighted combination of the observed responses. The weight on $y_i$ in $\hat{y}_i$ is the diagonal weight $\frac{1}{n}+\frac{(x_i-\bar{x})^2}{S_{xx}}$, which is $h_{ii}$. The formula uses only $x_i$, $\bar{x}$, and $S_{xx}$; changing the responses while keeping the predictor values fixed cannot change any leverage.

A useful check follows by adding the leverage values across all observations:

\sum_{i=1}^{n}h_{ii} =\sum_{i=1}^{n}\left(\frac{1}{n}+ \frac{(x_i-\bar{x})^2}{S_{xx}}\right) =1+\frac{S_{xx}}{S_{xx}}=2.

The total is two because a simple regression line with an intercept estimates two coefficients: an intercept and a slope. This does not mean each observation has leverage $2/n$. Rather, the average leverage is $2/n$, and points far from the predictor mean receive more than average leverage.

Worked Example: Finding Leverage for an Extreme Predictor Value

Suppose five hypothetical observations have predictor values $1, 2, 3, 4,$ and $10$. Find the leverage of the observation with $x_i=10$, and compare it with the leverage of the observation with $x_i=3$.

First calculate the predictor mean and sum of squared deviations:

\bar{x}=\frac{1+2+3+4+10}{5}=4, \qquad S_{xx}=(-3)^2+(-2)^2+(-1)^2+0^2+6^2=50.

For $x_i=10$, the squared distance from the mean is $36$:

h_{ii}=\frac{1}{5}+\frac{(10-4)^2}{50} =0.2+\frac{36}{50} =0.920.

For $x_i=3$, the squared distance from the mean is $1$:

h_{ii}=\frac{1}{5}+\frac{(3-4)^2}{50} =0.2+\frac{1}{50} =0.220.

The observation at $x=10$ has much higher leverage because its predictor value is far from the center of the other predictor values. Across all five observations, the leverage values are $0.380, 0.280, 0.220, 0.200,$ and $0.920$, which sum to $2.000$, as expected.

Using Hat Values as a Screening Tool

A hat value is useful for identifying observations whose predictor values deserve closer attention. A common screening guideline calls an observation potentially high-leverage when its hat value exceeds twice the average leverage. In simple linear regression, the average is $2/n$, so this guideline becomes $h_{ii}>4/n$. Some analysts use a more conservative comparison of $h_{ii}>6/n$. These are rules of thumb, not formal hypothesis tests or universal boundaries.

The guidelines are most useful as prompts to inspect the data and the fitted model. A threshold does not establish that an observation is erroneous, that the model is inappropriate, or that the point has actually changed the fitted line substantially. The cutoff may also be uninformative in a small sample: when $n=5$, the $4/n$ guideline equals $0.8$, while the average leverage is already $0.4$.

Worked Example: Applying a Screening Guideline

In the five-observation data set from the previous example, use the $4/n$ guideline to flag potential high-leverage observations.

Here $n=5$, so the screening value is

\frac{4}{n}=\frac{4}{5}=0.800.

The hat values were $0.380, 0.280, 0.220, 0.200,$ and $0.920$. Only the observation with $x=10$ exceeds $0.800$. It is therefore flagged by this particular guideline. The correct conclusion is that its predictor value warrants attention because it is unusual in this data set—not that it must be deleted or that it has been proven to distort the regression.

The guideline should also be interpreted in light of the data. With only five observations, one extreme predictor value can account for a substantial share of the spread in $x$, making the fitted line especially dependent on a very small data set. The cutoff summarizes a diagnostic concern; it does not remove the need to examine the observations and context.

High Leverage Is Not the Same as a Response Outlier

A response outlier is identified by how unusual the observed response is relative to the fitted value, using its residual and, when appropriate, a studentized residual. Leverage instead uses the predictor values alone. An observation with an unusual $x_i$ can lie close to the fitted line and therefore have a small residual. Conversely, a point with a typical $x_i$ can be far above or below the line and have a large residual without having high leverage.

Leverage indicates potential to affect the fitted line. A distant predictor value gives that observation a special position in determining the slope. Whether it actually has a substantial effect also depends on its response value and on the rest of the data. A separate diagnostic, Cook’s distance, combines information about residual size and leverage to assess the effect of an observation on the fitted values. The hat value alone does not make that assessment.

Worked Example: High Leverage Can Occur With a Small Residual

Consider hypothetical data with predictor-response pairs $(1,2), (2,4), (3,5), (4,7), (10,8)$. The predictor values are those from the first example, so the observation at $x=10$ has leverage $0.920$. To see how a high-leverage observation can still lie near the fitted line, calculate the least-squares line for all five pairs.

The response mean is $\bar{y}=26/5=5.2$. The cross-product sum is

\begin{aligned} S_{xy} &=\sum_{j=1}^{5}(x_j-\bar{x})(y_j-\bar{y})\\ &=(-3)(-3.2)+(-2)(-1.2)+(-1)(-0.2)+(0)(1.8)+(6)(2.8)\\ &=9.6+2.4+0.2+0+16.8=29.0. \end{aligned}

Since $S_{xx}=50$, the fitted slope is $b_1=29/50=0.580$, and the intercept is $b_0=\bar{y}-b_1\bar{x}=5.2-(0.580)(4)=2.88$. Thus the fitted value at $x=10$ is

\hat{y}=2.88+(0.580)(10)=8.68, \qquad e= y-\hat{y}=8.00-8.68=-0.680.

The point has high leverage but a residual of only $-0.680$ response units. This illustrates why leverage and response unusualness must not be treated as interchangeable. The point’s predictor value is far from the others, but its response is fairly close to the fitted value in this data set.

What to Check in Practice

When an observation has a large hat value, first confirm that its predictor value has been recorded correctly and that it belongs to the population or setting being studied. Then examine the scatterplot and the fitted relationship in context. A high-leverage observation may be a legitimate and important case, not a data error. If it lies in a region with little or no other data, conclusions about the regression there may rely heavily on that observation.

Do not remove an observation solely because its hat value crosses a screening threshold. Any decision to correct, exclude, or retain a data point needs a substantive justification, such as a documented recording error or a clear mismatch with the target population. Report what was done and consider how conclusions depend on the choice. A high-leverage point can alert you to sensitivity, but evaluating influence requires additional information.

Common Mistakes

Calling high leverage a large residual. Leverage is calculated from predictor values; a residual measures the difference between an observed and fitted response. Check both diagnostics separately.
Assuming a large hat value proves influence. It indicates unusual predictor position and potential to affect the line. It does not, by itself, quantify the actual effect on fitted values.
Deleting a point because it exceeds a rule of thumb. Screening thresholds are prompts for investigation, not automatic deletion rules. Check the data and context before deciding how to proceed.
Thinking leverage depends on the response values. The hat values are unchanged if the $y$-values change while the $x$-values stay fixed.

Key takeaway: In simple linear regression with an intercept, $h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}$. Large values identify observations with unusual predictor positions. They flag potential to affect the fitted line, not proof of a large residual or actual influence.

Check Your Understanding

Use the definitions and reasoning in this tutorial to answer the following questions.

For a simple linear regression with an intercept, what information is needed to calculate $h_{ii}$, and what information is not needed?
Why does an observation far from $\bar{x}$ tend to have a larger hat value?
In a sample of 20 observations, calculate the $4/n$ screening value. What should you conclude if an observation exceeds it?
Can an observation have high leverage and a small residual? Explain why.
The hat values from a simple linear regression with an intercept sum to $2$. What does their average equal for a sample of size $n$?