Leverage and the Diagonal of the Hat Matrix

Why Leverage Is a Property of the Regressors

In ordinary least squares, the fitted values are linear functions of the observed outcomes. The matrix that maps outcomes to fitted values is called the hat matrix. Its diagonal elements describe how each observation’s own outcome enters its fitted value. These diagonal elements are called leverage.

Leverage is determined by the design: the regressor values and the model’s included columns. It does not depend on the observed outcomes. In simple linear regression with an intercept, an observation’s leverage is larger when its regressor value lies farther from the sample mean, relative to the spread of all regressor values. The relationship can be derived exactly.

Definition: Let $X$ be an $n\times p$ regressor matrix with full column rank, including any intercept column. The hat matrix is $H=X(X'X)^{-1}X'$. The leverage of observation $i$ is the diagonal element $h_{ii}$ of $H$. It is the $i$th diagonal entry, not the entire $i$th row or column.

The hat matrix projects the outcome vector onto the column space of $X$, the set of vectors that can be represented by the model’s regressors. Thus the vector of fitted values is $\hat{y}=Hy$. The formula for $H$ requires that $X$ have full column rank, so its columns are linearly independent and $X'X$ is invertible. The results below concern this full-rank case.

Deriving the Distance Formula

Consider a simple regression with an intercept and one quantitative regressor $x$. Suppose there are $n$ observations and the regressor is not constant, so the intercept and $x$ columns are linearly independent. Write $\bar{x}$ for the sample mean of the regressor and define $z_i=x_i-\bar{x}$. Centering $x$ does not change the model’s column space: $x$ is the sum of the centered regressor $z$ and a constant vector. The vectors of ones and $z$ are orthogonal because $\sum_{i=1}^n z_i=0$.

The projection onto the constant vector has matrix $\mathbf{1}\mathbf{1}'/n$. The projection onto the centered regressor has matrix $zz'/(z'z)$. Since these two vectors are orthogonal and span the same space as the original intercept and regressor columns, the full projection is the sum of these two projections.

H=\frac{\mathbf{1}\mathbf{1}'}{n}+\frac{zz'}{\sum_{j=1}^n z_j^2}

Taking the $i$th diagonal element gives the leverage formula. The first term contributes $1/n$, while the second contributes the squared centered regressor value divided by the total centered sum of squares.

Result: In simple linear regression with an intercept and nonconstant regressor $x$, $$ h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{\sum_{j=1}^n (x_j-\bar{x})^2}. $$ The denominator is the regressor’s centered sum of squares, often written $S_{xx}$. The formula makes leverage depend on how far $x_i$ is from the sample center relative to the overall spread of the regressor values.

The first term, $1/n$, is a baseline shared by every observation. The second term is nonnegative and grows with the squared distance from $\bar{x}$. Distance alone does not determine leverage: the same absolute distance produces less leverage when the sample’s total regressor spread is larger. Rescaling the regressor from, for example, meters to centimeters does not change leverage, because both the squared deviation and $S_{xx}$ are multiplied by the same factor.

Worked Example: Calculating Leverage for Five Regressor Values

Suppose a hypothetical simple regression includes an intercept and a regressor measured in units, with values $1, 2, 3, 4,$ and $10$. Find the leverage of the observation with $x_i=10$, and check the leverage values for all five observations.

The sample size is $n=5$, and the regressor mean is $$ \bar{x}=\frac{1+2+3+4+10}{5}=4. $$ The centered sum of squares is $$ S_{xx}=(1-4)^2+(2-4)^2+(3-4)^2+(4-4)^2+(10-4)^2 =9+4+1+0+36=50. $$ For the observation with $x_i=10$, the formula gives $$ h_{ii}=\frac{1}{5}+\frac{(10-4)^2}{50} =0.2+\frac{36}{50} =0.92. $$

Applying the same calculation to each value gives $0.38, 0.28, 0.22, 0.20,$ and $0.92$, respectively. Their sum is $2.00$, as expected for a simple regression with two estimated coefficients: an intercept and a slope. The observation at $x=10$ has the greatest leverage because it is farthest from the regressor mean relative to the total spread.

Range and Average Leverage

The distance formula shows that simple-regression leverages are at least $1/n$, but a general property of the hat matrix gives the upper bound and applies beyond simple regression. The matrix $H$ is symmetric and idempotent: $H'=H$ and $H^2=H$. These properties hold because it is an orthogonal projection onto the column space of $X$.

Let $e_i$ be the vector with a one in position $i$ and zeros elsewhere. Then $h_{ii}=e_i'He_i$. By symmetry and idempotence, this diagonal element is the squared length of the projection of $e_i$ onto the model’s column space. The projected vector and the remaining component are orthogonal, so the squared length of the projection cannot exceed the squared length of $e_i$, which is one.

h_{ii}=e_i'He_i=e_i'H'He_i=\lVert He_i\rVert^2, \qquad \lVert e_i\rVert^2=\lVert He_i\rVert^2+\lVert (I-H)e_i\rVert^2.

Both squared lengths in the second equality are nonnegative. Therefore $0\leq \lVert He_i\rVert^2\leq 1$, which establishes the general range. In the simple-regression setting with an intercept, the distance formula sharpens the lower bound to $h_{ii}\geq 1/n$.

Theorem: For a full-column-rank design matrix with $p$ columns, every leverage satisfies $0\leq h_{ii}\leq 1$, and the average leverage is $p/n$. In simple linear regression with an intercept, $p=2$, so the average leverage is $2/n$.

To obtain the average, sum the diagonal elements. The sum of the diagonal elements is the trace, and the trace of a projection matrix equals its rank. Because $X$ has full column rank, that rank is $p$. Thus the total leverage is $p$, and dividing by the $n$ observations gives the average $p/n$.

\sum_{i=1}^{n}h_{ii}=\operatorname{tr}(H)=p, \qquad \frac{1}{n}\sum_{i=1}^{n}h_{ii}=\frac{p}{n}.

This is a design-wide accounting identity: if some observations have leverage above the average, others must have leverage below it. It does not say that every observation has leverage $p/n$. In simple regression, the distance formula shows exactly how the values vary around the shared baseline.

Worked Example: Symmetric Values Around the Mean

Suppose a hypothetical simple regression has an intercept and regressor values $2, 4, 6,$ and $8$, measured in a common unit. Calculate all four leverages.

Here $n=4$ and $$ \bar{x}=\frac{2+4+6+8}{4}=5, \qquad S_{xx}=(-3)^2+(-1)^2+1^2+3^2=20. $$ The two observations at the ends of the regressor range have leverage $$ h_{ii}=\frac{1}{4}+\frac{(2-5)^2}{20} =0.25+\frac{9}{20}=0.70, $$ and the same calculation applies at $x_i=8$. For $x_i=4$ and $x_i=6$, $$ h_{ii}=\frac{1}{4}+\frac{1^2}{20}=0.30. $$

The leverages are $0.70, 0.30, 0.30,$ and $0.70$. Their average is $0.50$, equal to $p/n=2/4$. The values are symmetric because the regressor observations are symmetrically positioned around their mean.

Worked Example: A Leverage of One

Consider hypothetical regressor values $0, 0, 0,$ and $6$ in a simple regression with an intercept. These values are not constant, so the design has full column rank. Calculate the leverage at $x_i=6$ and verify the bounds and average.

The mean and centered sum of squares are $$ \bar{x}=\frac{0+0+0+6}{4}=1.5, \qquad S_{xx}=3(0-1.5)^2+(6-1.5)^2=3(2.25)+20.25=27. $$ At $x_i=6$, $$ h_{ii}=\frac{1}{4}+\frac{(6-1.5)^2}{27} =0.25+\frac{20.25}{27}=1. $$ Each of the three observations at $x_i=0$ has $$ h_{ii}=\frac{1}{4}+\frac{(0-1.5)^2}{27} =0.25+\frac{2.25}{27} =\frac{1}{3}. $$

All four values lie between zero and one, and their sum is $1+3(1/3)=2$, giving average $2/4=0.50$. This example shows that the upper bound can be attained even when the design matrix is full rank. It does not imply that an observation with leverage one must have an unusual outcome; leverage is calculated from the regressors alone.

What Leverage Does—and Does Not—Describe

Leverage summarizes the geometric position of an observation’s regressor values within the design. In simple regression with an intercept, values near $\bar{x}$ have leverage close to the baseline $1/n$, while values farther away receive a larger squared-distance contribution. The denominator $S_{xx}$ puts that distance in context by accounting for the spread of the full regressor sample.

Leverage is not an outcome, a residual, or a direct measure of how much the estimated slope changes when an observation is removed. Those questions involve information beyond the hat-matrix diagonal and are separate from calculating leverage itself. Likewise, a large value of $h_{ii}$ is not, by itself, evidence that an observation is erroneous or that the regression specification is wrong. It describes where the observation lies in regressor space.

Common Mistakes

Using distance without accounting for overall spread. The wrong reasoning is that a fixed deviation from $\bar{x}$ always means the same leverage. The formula divides the squared deviation by $S_{xx}$, so the same distance contributes less when the regressor values are more dispersed.
Forgetting the intercept contribution. In simple regression with an intercept, leverage is not just $(x_i-\bar{x})^2/S_{xx}$. The full expression includes $1/n$, which is why the leverages sum to two rather than one.
Claiming the values are all equal to their average. The identity $\sum_i h_{ii}=p$ fixes the average, not each individual value. Regressor positions determine how the total is distributed.
Interpreting leverage as a feature of the outcome. The wrong reasoning is that an unusual response $y_i$ automatically creates high leverage. The correction is that $h_{ii}$ depends on $X$, not on $y$.
Applying the simple-regression formula to a different design. The distance formula above assumes an intercept and one regressor. For a different set of regressors, calculate the diagonal from the corresponding hat matrix rather than substituting one variable into the simple-regression formula.

Key takeaway: With an intercept and one nonconstant regressor, $h_{ii}=1/n+(x_i-\bar{x})^2/S_{xx}$. More generally, leverage values are between zero and one, and their average is the number of fitted coefficients divided by the sample size.

Check Your Understanding

Try these questions using the definitions and results above.

In a simple regression with an intercept, why does centering the regressor leave the model’s column space unchanged?
A sample has $n=10$, one intercept, and one slope. What is the average leverage? What is the sum of the leverages?
In the simple-regression distance formula, what happens to an observation’s leverage contribution if its squared deviation from the mean stays fixed while $S_{xx}$ increases?
Explain why the hat-matrix diagonal elements cannot be negative or exceed one.
Can the leverage values all equal $p/n$? Explain what the average-leverage identity does and does not require.