Leverage Values in Simple Linear Regression

Why Predictor Location Matters

In simple linear regression, observations can differ in two important ways: their predictor values may be unusually far from the rest, or their responses may be poorly described by the fitted line. Leverage measures the first feature, not the second. It is determined by the predictor values alone, so it can be calculated before the responses are examined.

This distinction matters when reviewing a regression fit. An observation far from the center of the observed predictor values has greater capacity to affect the fitted line, but its leverage alone does not tell us whether its response is unusual. Leverage is aligned with the SOA VEE Mathematical Statistics syllabus as one part of regression diagnostics.

Definition and Model

Suppose $n$ observations are fitted by ordinary least squares to a simple linear regression model with an intercept:

Y_i=\beta_0+\beta_1x_i+\varepsilon_i,\qquad i=1,\ldots,n.

Here $x_i$ is the observed predictor value, $Y_i$ is the response, and $\varepsilon_i$ is the error. For the usual regression interpretation, the errors have mean zero, common variance $\sigma^2$, and are uncorrelated. Normality is needed for some exact inferential results, but not to calculate leverage. The leverage formula is an algebraic property of the predictor values and the least-squares fit. Assume throughout that the model includes an intercept and that the predictor values are not all equal.

Definition: The leverage $h_{ii}$ of observation $i$ is the $i$th diagonal entry of the hat matrix: the matrix that maps the response vector to the fitted-value vector. In simple linear regression with an intercept, it is $$ h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{S_{xx}}, \qquad \bar{x}=\frac{1}{n}\sum_{j=1}^{n}x_j,\quad S_{xx}=\sum_{j=1}^{n}(x_j-\bar{x})^2. $$

The formula shows the two ingredients directly. Every observation has a baseline leverage of $1/n$. The additional amount depends on the squared distance between its predictor and the sample mean, scaled by the overall spread $S_{xx}$. Thus, an observation is relatively far from the predictor center when its distance is large compared with the spread of the whole predictor sample.

Leverage is dimensionless. Changing the predictor’s units—for example, converting years to months—does not change it, because both the squared distance and $S_{xx}$ are multiplied by the same squared conversion factor.

Why the Formula Holds

The fitted value at $x_i$ is $\hat{Y}_i=\hat{\beta}_0+\hat{\beta}_1x_i$. With an intercept, the least-squares estimates can be written as $\hat{\beta}_1=\sum_j(x_j-\bar{x})Y_j/S_{xx}$ and $\hat{\beta}_0=\bar{Y}-\hat{\beta}_1\bar{x}$. Substituting these expressions into the fitted value gives:

\begin{aligned} \hat{Y}_i &=\bar{Y}+(x_i-\bar{x})\hat{\beta}_1\\ &=\frac{1}{n}\sum_{j=1}^{n}Y_j+ \frac{x_i-\bar{x}}{S_{xx}}\sum_{j=1}^{n}(x_j-\bar{x})Y_j\\ &=\sum_{j=1}^{n} \left[\frac{1}{n}+ \frac{(x_i-\bar{x})(x_j-\bar{x})}{S_{xx}}\right]Y_j. \end{aligned}

The coefficient multiplying $Y_i$ in this expression is the diagonal entry of the mapping from observed responses to fitted responses. Setting $j=i$ gives $h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}$, as stated. This also explains the interpretation: leverage describes how the fitted value at an observation’s predictor location is weighted in the least-squares fit.

Useful Properties for Checking Calculations

For a nonconstant predictor and a model with an intercept, leverage values satisfy several checks:

$h_{ii}\geq 1/n$, since the squared-distance term cannot be negative. Equality holds when $x_i=\bar{x}$.
$h_{ii}\leq 1$. A calculated value greater than 1 signals inconsistent inputs or an arithmetic error.
The leverage values sum to 2: $$\sum_{i=1}^{n}h_{ii}=2.$$

For the sum, add the formula over all observations. The baseline terms sum to $n(1/n)=1$, and the distance terms sum to $S_{xx}/S_{xx}=1$. Therefore the total is $1+1=2$. This total corresponds to the two fitted regression coefficients, the intercept and slope.

These results depend on the model specification. If the predictor is constant, then $S_{xx}=0$, the slope cannot be estimated, and the formula does not apply. If the fitted model has no intercept, its leverage formula is different; do not use the with-intercept formula automatically.

Worked Example: Computing All Leverages in a Small Design

A hypothetical study relates a device’s weekly operating hours $x$ to a measured response. The five observed predictor values are $2,4,6,8,10$ hours. An intercept is included in the ordinary least-squares fit. Calculate every leverage value and check the result. No error distribution is needed for this calculation.

Method. Find $\bar{x}$ and $S_{xx}$, then substitute each predictor value into the leverage formula. The response values are not needed.

First, $\bar{x}=(2+4+6+8+10)/5=30/5=6$ hours. The sum of squared deviations is $S_{xx}=(-4)^2+(-2)^2+0^2+2^2+4^2=16+4+0+4+16=40$ hours squared.

For $x=2$, $h_{ii}=1/5+(2-6)^2/40=0.2+16/40=0.6$. For $x=4$, $h_{ii}=0.2+4/40=0.3$. For $x=6$, $h_{ii}=0.2+0/40=0.2$. By the same substitutions, $x=8$ has leverage $0.2+4/40=0.3$, and $x=10$ has leverage $0.2+16/40=0.6$.

Check. The values sum to $0.6+0.3+0.2+0.3+0.6=2.0$, as required. Each is at least $1/5=0.2$. The two predictor values farthest from the mean have the greatest leverage. A common mistake is to use $x_i^2$ rather than $(x_i-\bar{x})^2$; that would incorrectly make leverage depend on the arbitrary choice of zero on the predictor scale.

Worked Example: A Predictor Far from the Center

A hypothetical insurer studies a response associated with annual exposure, recorded in thousands of exposure units. The six predictor values are $2,3,4,5,6,10$. An intercept is included. Calculate the leverage for the observation with predictor 10 and for the observation with predictor 4, then verify the total leverage.

Method. Use the same with-intercept formula for each observation; the response values and any error distribution are irrelevant to these calculations.

The mean predictor is $\bar{x}=(2+3+4+5+6+10)/6=30/6=5$. The sum of squared deviations is $S_{xx}=(-3)^2+(-2)^2+(-1)^2+0^2+1^2+5^2=9+4+1+0+1+25=40$.

For the observation at $x=10$, $h_{ii}=1/6+(10-5)^2/40=1/6+25/40=1/6+5/8=19/24$, approximately $0.7917$ to four decimal places. For $x=4$, $h_{ii}=1/6+(4-5)^2/40=1/6+1/40=23/120$, approximately $0.1917$ to four decimal places.

To check the total, the six leverage values are $47/120,\,4/15,\,23/120,\,1/6,\,23/120,$ and $19/24$. Their sum is $(47+32+23+20+23+95)/120=240/120=2$.

The observation at 10 has much greater leverage because it is farther from the sample mean relative to the predictor spread. This does not establish that its response is poorly fit. A common mistake is to interpret a large leverage value as proof of a large residual; leverage uses only the $x$-values, whereas a residual also depends on the observed response.

Leverage Is Not Residual Size

A residual is $e_i=Y_i-\hat{Y}_i$, the difference between the observed response and its fitted value. Unlike leverage, it depends on the responses as well as the predictor values. An observation can have a high leverage value and a small residual, or a low leverage value and a large residual. The following calculation makes the distinction explicit.

Worked Example: High Leverage With a Small Residual

A hypothetical laboratory records five responses at predictor values $x=0,1,2,3,4$, with observed responses $Y=10,12,18,16,18$. Fit a line with an intercept, calculate the endpoint and center leverages, and compare their residuals.

Method. Calculate the least-squares line and residuals from the responses, then calculate leverage separately from the predictor values.

Here $\bar{x}=2$, $S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=10$, and $\bar{Y}=(10+12+18+16+18)/5=74/5=14.8$. The slope is \[ \hat{\beta}_1=\frac{(-2)(10-14.8)+(-1)(12-14.8)+0(18-14.8)+1(16-14.8)+2(18-14.8)}{10} =\frac{9.6+2.8+0+1.2+6.4}{10}=2. \] The intercept is $\hat{\beta}_0=\bar{Y}-\hat{\beta}_1\bar{x}=14.8-2(2)=10.8$, so the fitted line is $\hat{Y}=10.8+2x$.

At $x=4$, the fitted response is $18.8$ and the residual is $18-18.8=-0.8$. Its leverage is $1/5+(4-2)^2/10=0.2+0.4=0.6$. At the center $x=2$, the fitted response is $14.8$, so its residual is $18-14.8=3.2$; its leverage is $1/5=0.2$.

Check. The endpoint has three times the center observation’s leverage, but its residual has much smaller absolute size: $0.8<3.2$. The endpoint is unusual in predictor location, while the center response is farther from its fitted value. The common mistake is to call the center observation “high leverage” because its residual is large. That confuses response discrepancy with predictor location.

Interpreting Leverage Carefully

A high leverage value identifies an observation whose predictor location is relatively unusual within the observed design. It is a warning to inspect the observation, not a verdict that the observation is erroneous or that the fitted model is invalid. A point near $\bar{x}$ has leverage near its minimum $1/n$; points far from $\bar{x}$, especially relative to a narrow predictor spread, have larger leverage.

Leverage is also not a complete measure of how much an observation affects the fitted regression. That assessment requires considering the response residual as well as predictor location. In particular, a point can be far out in predictor space yet lie close to the fitted line. Conversely, an observation near the predictor center can have a substantial residual without having high leverage.

One rough screening convention for simple regression with an intercept is to flag $h_{ii}>2p/n=4/n$, where $p=2$ is the number of fitted coefficients. This is only a rule of thumb, not a formal test or a universal cutoff. It depends on sample size and design; the formula and the context should guide interpretation.

Key takeaway: In simple linear regression with an intercept, leverage is $1/n+(x_i-\bar{x})^2/S_{xx}$. It measures how unusual an observation’s predictor location is relative to the design. It does not measure the size of that observation’s residual.

Check Your Understanding

For each problem, assume ordinary least squares with an intercept unless stated otherwise. Use the leverage formula and its checks; answers are given for verification.

A regression uses predictor values $0,2,4,6$. Find $\bar{x}$, $S_{xx}$, and the leverage of the observation at $x=6$. (Answer: $\bar{x}=3,\ S_{xx}=20,\ h_{ii}=0.7000$.)
A five-observation design has predictor values $1,3,4,8,9$. Calculate the leverage for $x=1$, to four decimal places. (Answer: $\bar{x}=5,\ S_{xx}=46,\ h_{ii}=0.5478$.)
A proposed design summary reports $n=6$, $\bar{x}=10$, and $S_{xx}=30$, with one included observation at $x=16$. Use the formula to assess whether those summaries can describe a valid with-intercept design. (Answer: They cannot: the formula gives $h_{ii}=1/6+36/30=1.3667>1$, which is impossible for a leverage value.)
In a five-observation design, the predictor values are $1,2,3,4,10$. Calculate the leverage at $x=10$ and $x=3$. Which is farther from the predictor center, and what can the leverage comparison establish about their residuals? (Answer: $\bar{x}=4,\ S_{xx}=50;\ h_{10}=0.9200,\ h_3=0.2200$. The $x=10$ observation is farther from the center and has greater leverage; this alone does not establish which residual is larger.)
A regression includes an intercept and has $n=8$ observations. Seven leverage values sum to $1.72$. Find the eighth leverage value, then state whether it meets the rough $4/n$ screening convention. (Answer: The eighth value is $2-1.72=0.28$; $4/n=0.50$, so it does not exceed that rough screening cutoff.)

What You'll Learn