Hat Values and Leverage

What Hat Values Measure

In ordinary least-squares (OLS) regression, each fitted value is a weighted combination of the observed responses. The weights are determined by the predictor values, not by the observed response values. The diagonal weight for observation \(i\) is its hat value, also called its leverage. Hat values help describe how unusual an observation's predictor information is relative to the rest of the design.

This is a property of the predictor data and the chosen model, such as whether an intercept or additional predictors are included. It does not depend on whether the observed response is close to its fitted value. Consequently, a high hat value is not, by itself, evidence that an observation has a large residual or an influential effect on the fitted regression.

The name comes from the mapping \(y\mapsto\hat{y}\): the matrix puts a “hat” on the observed response vector. The off-diagonal entries also contribute to fitted values, but this tutorial focuses on the diagonal entries. The algebraic results below require a full-rank design for the displayed inverse. They do not require normally distributed errors or equal error variances: they follow from the OLS design matrix itself.

Range and Average of the Hat Values

For a full-rank design with \(m\) columns, every hat value is between zero and one, inclusive. Also, all \(n\) hat values sum to \(m\), so their average is \(m/n\). These facts give useful checks on hand calculations and a basic reference for understanding how the leverage is distributed across observations.

Here is a short derivation. The matrix \(H\) is symmetric because \((X'X)^{-1}\) is symmetric. It is also idempotent, meaning \(H^2=H\): multiplying out gives \(X(X'X)^{-1}(X'X)(X'X)^{-1}X'=H\). Let \(e_i\) be the vector with a one in position \(i\) and zeros elsewhere. Symmetry and idempotence give \(h_{ii}=e_i'He_i=(He_i)'(He_i)\), which is a squared length and hence nonnegative. The matrix \(I-H\) is also symmetric and idempotent, so \(1-h_{ii}=e_i'(I-H)e_i=((I-H)e_i)'((I-H)e_i)\ge0\). Therefore \(h_{ii}\le1\).

For the sum, the sum of diagonal entries is the trace. Using the identity that the trace of a product can be cyclically rearranged,

Dividing this total by the \(n\) observations gives the average \(m/n\). In a model with an intercept and \(k\) predictors, the design matrix has \(k+1\) columns when they are linearly independent, so its average leverage is \((k+1)/n\). The average is a property of the model's design, not an individual-observation cutoff for declaring a case unusual.

Simple Linear Regression: A Direct Formula

In simple linear regression with an intercept and one quantitative predictor \(x\), the leverage can be calculated without constructing the full hat matrix. Define \(\bar{x}\) as the sample mean of the predictor values and \(S_{xx}=\sum_{j=1}^{n}(x_j-\bar{x})^2\). Provided the predictor is not constant, the formula is

To see why, the OLS fitted value at \(x_i\) is \(\hat{y}_i=\bar{y}+(x_i-\bar{x})\sum_{j=1}^{n}(x_j-\bar{x})(y_j-\bar{y})/S_{xx}\). Since \(\bar{y}=(1/n)\sum_j y_j\), the coefficient of \(y_i\) in the first term is \(1/n\). In the second term, the coefficient of \(y_i\) is \((x_i-\bar{x})^2/S_{xx}\); subtracting \(\bar{y}\) does not change that coefficient because \(\sum_j(x_j-\bar{x})=0\). But \(\hat{y}_i=\sum_j h_{ij}y_j\), so the coefficient of \(y_i\) is precisely \(h_{ii}\). Adding the two contributions yields the formula.

The formula shows that observations at the predictor mean have leverage \(1/n\), while observations farther from the mean have larger leverage. It also makes clear that the response values \(y_j\) do not enter the calculation.

Calculating Leverage in Other Designs

With more than one predictor, the simple-regression shortcut does not apply in general. For a row vector \(x_i'\) containing the predictor values for observation \(i\), including a one if there is an intercept, the diagonal entry can be written

This expression follows by taking the \(i\)th diagonal entry of \(X(X'X)^{-1}X'\). It provides a direct route to calculation when the design columns are manageable.

How to Use Hat Values Carefully

Leverage describes the predictor position of an observation relative to the design used to fit the model. Because \(H\) is determined by \(X\), changing a response while holding all predictors fixed leaves every hat value unchanged. Conversely, changing the predictor data or the model's set of predictors can change the hat values, even if the responses remain the same.

A frequent mistake is to reason, “This observation has a high hat value, so it must be an outlier or must strongly influence the fitted regression.” That conclusion does not follow. A hat value is not a measure of residual size: it contains no response information. Nor does leverage alone establish how much a fitted model would change if an observation were removed. Those questions require other diagnostics. The safe conclusion is narrower: the observation has relatively high leverage within this particular predictor design.

Hat values are useful as a screening description, not as a stand-alone verdict. The range and average offer checks: a computed value outside \([0,1]\), or a sum that differs from the number of independent design columns beyond rounding, signals a calculation or setup problem. But the average \(m/n\) is not a universal threshold for identifying cases that deserve concern. Interpret any particular value in light of the model and the predictor data.