The Hat Matrix and Regression Leverage

Why the Hat Matrix Matters

In ordinary least squares (OLS), fitted values are calculated from the observed responses and the predictor values. The hat matrix makes that relationship explicit: it is the matrix that maps the response vector to the fitted-value vector. Its diagonal entries, called leverages, describe how unusual each observation’s predictor values are relative to the full design.

Leverage concerns the predictors, not whether an observed response is unusually high or low. That distinction matters in regression diagnostics: an observation can have unusual predictors and still lie close to the fitted regression surface. The hat matrix helps identify this predictor-space feature, but leverage alone does not establish that an observation is erroneous or influential.

Deriving the Mapping from Responses to Fitted Values

Suppose the regression model is \(y=X\beta+\varepsilon\), where \(y\) is an \(n\)-vector of observed responses, \(X\) is the \(n\times p\) design matrix, and \(\beta\) is a \(p\)-vector of coefficients. The rows of \(X\) contain the predictor values; when the model includes an intercept, each row also contains a 1. Assume \(X\) has full column rank, so the OLS coefficient estimate is uniquely defined. The matrix identity below is an algebraic result of OLS; normality or constant error variance is not required to compute \(H\).

OLS chooses \(\hat{\beta}\) to minimize the residual sum of squares, \((y-X\beta)^\mathsf{T}(y-X\beta)\). Differentiating with respect to \(\beta\) and setting the derivative to zero gives the normal equations. Full column rank allows us to solve them:

The fitted values are \(X\hat{\beta}\). Substituting the expression for \(\hat{\beta}\) shows that fitted values are a matrix transformation of the observed responses:

Thus, the name “hat matrix” reflects its role in putting a hat on \(y\): \(H\) transforms \(y\) into \(\hat{y}\). This formula does not apply as written if \(X\) is rank deficient, because \((X^\mathsf{T}X)^{-1}\) then does not exist. In this tutorial, the full-rank OLS setting is assumed throughout.

What the Diagonal Entries Measure

The \(i\)th fitted value is a weighted sum of all the observed responses:

The diagonal entry \(h_{ii}\) is the weight attached to observation \(i\)’s own response in its fitted value. It is also a measure of how far the observation’s predictor row extends in the geometry of the design. A larger \(h_{ii}\) indicates more unusual predictor coordinates relative to the other rows of \(X\), not necessarily an unusual response.

The geometry can be seen from the properties of \(H\). It is symmetric, \(H^\mathsf{T}=H\), and idempotent, \(H^2=H\). Symmetry follows directly from the formula for \(H\). For idempotence, multiply \(H\) by itself and use \((X^\mathsf{T}X)^{-1}X^\mathsf{T}X=I\):

These properties show that \(H\) is the matrix for projection onto the column space of \(X\), the space of response vectors that the regression model can fit. In particular, they explain why diagonal entries are between 0 and 1. Since \(H\) is symmetric and idempotent,

The right-hand side is nonnegative. Also, \(H\) is an orthogonal projection, which cannot increase vector length. Projecting the \(i\)th coordinate unit vector therefore gives a vector of squared length at most 1; that squared length is \(h_{ii}\). Hence \(0\le h_{ii}\le1\). This bound is an algebraic property, not a rule for classifying observations as high or low leverage.

Leverage in Regression with an Intercept

For a simple linear regression with an intercept, leverage has a convenient form in terms of the predictor values. Let \(\bar{x}\) be their mean and \(S_{xx}=\sum_{j=1}^{n}(x_j-\bar{x})^2\), with \(S_{xx}>0\). The diagonal entry for observation \(i\) is:

The first term is shared by all observations. The second increases as \(x_i\) moves farther from the predictor mean, measured relative to the overall spread of the predictor values. This is the one-predictor version of a broader idea: in multiple regression with an intercept, leverage reflects how far the observation’s centered predictor vector lies from the center, accounting for the scaling and relationships among predictors.

To see the multiple-predictor expression, let \(Z\) be the design matrix with the intercept column removed and each predictor column centered. If the centered predictor columns are linearly independent, then:

where \(z_i\) is the centered predictor vector for observation \(i\). The quadratic term accounts for predictor scales and correlations together; looking at one predictor at a time can miss an unusual combination of otherwise moderate values.

Leverage Depends on the Design, Not the Responses

The formula \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) contains \(X\), but not \(y\). Consequently, changing the responses while keeping the predictor values and model specification fixed leaves every leverage unchanged. Conversely, adding or changing predictor values can alter the leverage of existing observations as well as that of the new observation.

Interpretation and a Common Exam Trap

A common mistake is to use response values in a leverage calculation, perhaps by treating a response as though it were a predictor or by assuming that a large residual implies high leverage. For example, in simple regression with an intercept, the correct formula uses \(x_i\), \(\bar{x}\), and \(S_{xx}\); replacing \(x_i\) with \(y_i\) does not calculate leverage. The response values enter the fitted values and residuals, but they do not enter \(H\).

The converse is important too: high leverage does not automatically mean an observation has a large residual. Leverage describes the position of the predictor row in the design. Whether the observed response agrees with the fitted relationship is a separate question. A leverage value by itself does not prove a response anomaly, indicate an error, or justify deleting an observation.

For the full-rank model, another useful check is that the sum of the diagonal entries equals the rank of \(X\), which is \(p\):

This follows from the trace of \(X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), which equals the trace of the \(p\times p\) identity matrix. It is a check on a computed set of leverages, not a method for deciding whether any individual value is concerning. The practical interpretation always depends on the predictor design and the purpose of the model.