The Hat Matrix for Fitted Values

From Least-Squares Estimates to Fitted Values

In multiple regression, the ordinary least-squares (OLS) estimate describes how the response is fitted using the columns of a design matrix. A related object, the hat matrix, acts directly on the observed response vector and returns the fitted response vector. Its usefulness is not just computational: its algebraic properties identify fitted values as a projection onto the space spanned by the design matrix.

This tutorial assumes the multiple regression model and design-matrix notation, along with the full-column-rank condition used for the unique OLS estimate. The focus is the linear transformation from observed responses to fitted responses—not coefficient interpretation or residual analysis. The hat-matrix construction is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: Let $X$ be an $n \times p$ design matrix with linearly independent columns, and let $y$ be the $n \times 1$ observed response vector. The hat matrix is $$ H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}. $$ It maps the observed responses to the fitted responses: $\hat{y}=Hy$. The full-column-rank condition ensures that $X^\mathsf{T}X$ is invertible.

The term “hat” refers to the fact that multiplying by $H$ places a hat over the response vector: $y$ becomes $\hat{y}$. For a model with an intercept, the first column of $X$ is typically a column of ones; the formula itself applies whether or not an intercept is included, provided $X$ has full column rank.

Deriving the Hat Matrix

The OLS coefficient estimate is obtained from the normal equations. Under the full-column-rank condition, it is $\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}y$. The fitted response vector is defined by applying the design matrix to that coefficient estimate. Substitution gives the hat-matrix formula.

Start with the OLS estimate.
$\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}y$, an $p \times 1$ vector of estimated coefficients.

Define the fitted responses.
The $n \times 1$ fitted vector is $\hat{y}=X\hat{\beta}$.

Substitute the coefficient estimate.
$\hat{y}=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}y$.

Identify the response-to-fit map.
The matrix multiplying $y$ is $H$, so $\hat{y}=Hy$.

The dimensions help distinguish the hat matrix from the coefficient estimator. The matrix $X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$ is $n \times n$, so it maps an $n \times 1$ response vector to another $n \times 1$ vector. In contrast, $(X^\mathsf{T}X)^{-1}X^\mathsf{T}$ is $p \times n$, and maps the response vector to the $p \times 1$ coefficient estimate.

Worked Example: Fitting an Intercept-Only Model

A hypothetical analyst observes responses $y=(2,5,8)^\mathsf{T}$ and fits an intercept-only model. Construct $H$, calculate the fitted values, and check the result against the fitted intercept. The method is to form the design matrix and apply the hat-matrix formula; this also avoids treating the coefficient-estimation matrix as though it were $H$.

Step 1: State the design matrix and check its rank. An intercept-only model has one design column, consisting of ones:

X=\begin{pmatrix}1\\1\\1\end{pmatrix}, \qquad X^\mathsf{T}X=(1\quad 1\quad 1) \begin{pmatrix}1\\1\\1\end{pmatrix}=3.

The single column is nonzero, so $X$ has full column rank and $X^\mathsf{T}X$ is invertible.

Step 2: Calculate $H$.

H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T} =\begin{pmatrix}1\\1\\1\end{pmatrix}\frac{1}{3}(1\quad 1\quad 1) =\frac{1}{3}\begin{pmatrix}1&1&1\\1&1&1\\1&1&1\end{pmatrix}.

Step 3: Apply $H$ to the observed responses.

\hat{y}=Hy =\frac{1}{3}\begin{pmatrix}1&1&1\\1&1&1\\1&1&1\end{pmatrix} \begin{pmatrix}2\\5\\8\end{pmatrix} =\frac{1}{3}\begin{pmatrix}2+5+8\\2+5+8\\2+5+8\end{pmatrix} =\begin{pmatrix}5\\5\\5\end{pmatrix}.

Check. The OLS estimate of the intercept is the mean, $(2+5+8)/3=5$, so all three fitted responses equal 5, as the matrix calculation gives.

Symmetry and Idempotence

Two central properties follow directly from the formula. A matrix $H$ is symmetric if $H^\mathsf{T}=H$. It is idempotent if $H^2=H$. For a full-column-rank design matrix, the hat matrix has both properties.

Theorem: If $X$ has full column rank and $H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$, then $H^\mathsf{T}=H$ and $H^2=H$. Symmetry follows from transposing the product; idempotence follows from cancelling the middle factor $X^\mathsf{T}X$ against its inverse.

Here is the algebra for each property. The matrix $X^\mathsf{T}X$ is symmetric, and its inverse is symmetric as well: for an invertible matrix $A$, $(A^{-1})^\mathsf{T}=(A^\mathsf{T})^{-1}$. Transposing the product for $H$ therefore gives $H$ itself. For idempotence, group the adjacent factors $X^\mathsf{T}X$ and use $(X^\mathsf{T}X)^{-1}(X^\mathsf{T}X)=I_p$.

\begin{aligned} H^\mathsf{T} &=\left[X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\right]^\mathsf{T}\\ &=X\left[(X^\mathsf{T}X)^{-1}\right]^\mathsf{T}X^\mathsf{T}\\ &=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}=H, \end{aligned} \qquad \begin{aligned} H^2 &=X(X^\mathsf{T}X)^{-1}X^\mathsf{T} X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\\ &=X(X^\mathsf{T}X)^{-1}(X^\mathsf{T}X) (X^\mathsf{T}X)^{-1}X^\mathsf{T}\\ &=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}=H. \end{aligned}

Idempotence means that once a vector is fitted by $H$, applying the same fitted-value transformation again does not change it: $H\hat{y}=H(Hy)=H^2y=Hy=\hat{y}$. This property concerns the matrix transformation, not a claim that the original responses equal their fitted values.

Worked Example: Verifying the Properties for a Two-Predictor Design

Consider the hypothetical design matrix and response vector $X=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}$ and $y=(3,0,6)^\mathsf{T}$. Calculate the fitted values and verify symmetry and idempotence. The method is direct matrix multiplication, followed by a second application of $H$; this checks the defining properties rather than assuming them from a numerical result.

Step 1: Check full column rank and calculate the inverse. The two columns of $X$ are linearly independent, and

X^\mathsf{T}X= \begin{pmatrix}2&1\\1&2\end{pmatrix}, \qquad (X^\mathsf{T}X)^{-1} =\frac{1}{3}\begin{pmatrix}2&-1\\-1&2\end{pmatrix}.

Step 2: Form the hat matrix. Since $X^\mathsf{T}=\begin{pmatrix}1&0&1\\0&1&1\end{pmatrix}$, multiplication gives

H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T} =\frac{1}{3}\begin{pmatrix}2&-1&1\\-1&2&1\\1&1&2\end{pmatrix}.

Step 3: Calculate fitted values.

\hat{y}=Hy =\frac{1}{3}\begin{pmatrix}2&-1&1\\-1&2&1\\1&1&2\end{pmatrix} \begin{pmatrix}3\\0\\6\end{pmatrix} =\frac{1}{3}\begin{pmatrix}6+0+6\\-3+0+6\\3+0+12\end{pmatrix} =\begin{pmatrix}4\\1\\5\end{pmatrix}.

Step 4: Check symmetry. The displayed matrix has matching entries on opposite sides of its main diagonal, so $H^\mathsf{T}=H$.

Step 5: Apply $H$ to the fitted vector.

H\hat{y} =\frac{1}{3}\begin{pmatrix}2&-1&1\\-1&2&1\\1&1&2\end{pmatrix} \begin{pmatrix}4\\1\\5\end{pmatrix} =\frac{1}{3}\begin{pmatrix}8-1+5\\-4+2+5\\4+1+10\end{pmatrix} =\begin{pmatrix}4\\1\\5\end{pmatrix} =\hat{y}.

This confirms that the fitted vector is unchanged by a second application of $H$, consistent with $H^2=H$. The calculation also checks the original fitted values directly.

Projection Meaning and an Important Distinction

Every fitted vector has the form $X\hat{\beta}$, so it lies in the column space of $X$, the set of all linear combinations of the design-matrix columns. The hat matrix projects the observed response vector onto this space. Symmetry and idempotence are the characteristic algebraic properties of this orthogonal projection.

This perspective gives a useful interpretation of idempotence: a vector already in the column space is already a possible fitted vector, so projecting it again leaves it unchanged. In particular, $Hy$ is in the column space because $Hy=X[(X^\mathsf{T}X)^{-1}X^\mathsf{T}y]$. Applying $H$ to it again therefore returns the same vector.

Do not confuse $H$ with the coefficient-estimation matrix. The latter produces $\hat{\beta}$, while $H$ produces $\hat{y}$. Their dimensions and outputs are different, even though the hat-matrix formula is obtained by combining them.

Worked Example: Avoiding a Coefficient-versus-Fit Error

Use $X=\begin{pmatrix}1&-1\\1&0\\1&1\end{pmatrix}$ and $y=(1,2,4)^\mathsf{T}$. Find the fitted vector and explain why the coefficient-estimation matrix alone is not the hat matrix. This example checks both the result and the dimensions, avoiding the common error of stopping after calculating $\hat{\beta}$.

Step 1: Calculate the coefficient-estimation matrix. The columns are orthogonal, giving

X^\mathsf{T}X=\begin{pmatrix}3&0\\0&2\end{pmatrix}, \qquad (X^\mathsf{T}X)^{-1}X^\mathsf{T} =\begin{pmatrix} 1/3&1/3&1/3\\ -1/2&0&1/2 \end{pmatrix}.

This is a $2 \times 3$ matrix, so multiplying it by the $3 \times 1$ response gives the $2 \times 1$ coefficient vector:

\hat{\beta} =\begin{pmatrix} 1/3&1/3&1/3\\ -1/2&0&1/2 \end{pmatrix} \begin{pmatrix}1\\2\\4\end{pmatrix} =\begin{pmatrix}7/3\\3/2\end{pmatrix}.

Step 2: Multiply by $X$ to obtain the fitted responses.

\hat{y}=X\hat{\beta} =\begin{pmatrix}1&-1\\1&0\\1&1\end{pmatrix} \begin{pmatrix}7/3\\3/2\end{pmatrix} =\begin{pmatrix}7/3-3/2\\7/3\\7/3+3/2\end{pmatrix} =\begin{pmatrix}5/6\\7/3\\23/6\end{pmatrix}.

Step 3: Check by forming $H$. The hat matrix for this design is

H=\frac{1}{3}\begin{pmatrix}1&1&1\\1&1&1\\1&1&1\end{pmatrix} +\frac{1}{2}\begin{pmatrix}1&0&-1\\0&0&0\\-1&0&1\end{pmatrix} = \begin{pmatrix}5/6&1/3&-1/6\\1/3&1/3&1/3\\-1/6&1/3&5/6\end{pmatrix}.

Multiplying by $y$ gives $(5/6,7/3,23/6)^\mathsf{T}$, agreeing with $X\hat{\beta}$. The coefficient-estimation matrix is not $H$: it has two rows, not three, and returns coefficients rather than three fitted responses.

Common Exam Trap

A frequent error is to label $(X^\mathsf{T}X)^{-1}X^\mathsf{T}$ as the hat matrix. In the preceding example, using that matrix gives $(7/3,3/2)^\mathsf{T}$. Those are the estimated intercept and slope, not the fitted responses. The correction is to multiply the coefficient estimate by $X$, or equivalently combine the matrices first:

\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}y, \qquad \hat{y}=X\hat{\beta} =X(X^\mathsf{T}X)^{-1}X^\mathsf{T}y=Hy.

A second check is dimensional: if there are $n$ observations and $p$ coefficients, $H$ must be $n \times n$, while the coefficient-estimation matrix is $p \times n$. Also, the displayed inverse formula requires full column rank. If the columns of $X$ are linearly dependent, $X^\mathsf{T}X$ is singular, so this particular inverse-based formula cannot be used as written.

Key takeaway: For a full-column-rank design matrix, $H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$ maps $y$ to $\hat{y}$. It is symmetric and idempotent, and its dimension is $n \times n$. The coefficient-estimation matrix is a different matrix with a different output.

Check Your Understanding

For each problem, write down the matrix dimensions before calculating.

An intercept-only model has four responses $y=(1,3,5,7)^\mathsf{T}$. Give the hat matrix and fitted vector. (Answer: $H=\frac14\mathbf{1}\mathbf{1}^\mathsf{T}$, a $4\times4$ matrix of $1/4$ entries; $\hat{y}=(4,4,4,4)^\mathsf{T}$.)
For $X=\begin{pmatrix}1&-1\\1&0\\1&1\end{pmatrix}$ and $y=(2,1,5)^\mathsf{T}$, calculate $\hat{\beta}$ and $\hat{y}$. (Answer: $\hat{\beta}=(8/3,3/2)^\mathsf{T}$; $\hat{y}=(7/6,8/3,25/6)^\mathsf{T}$.)
A full-column-rank design matrix has $n=6$ observations and $p=3$ columns. State the dimensions of $H$, the coefficient-estimation matrix, $y$, and $\hat{\beta}$. (Answer: $H$ is $6\times6$; the coefficient-estimation matrix is $3\times6$; $y$ is $6\times1$; $\hat{\beta}$ is $3\times1$.)
Suppose $X^\mathsf{T}X=\begin{pmatrix}2&0\\0&2\end{pmatrix}$ and $X$ has two linearly independent columns. Which properties can be concluded about $H$, and what are their algebraic statements? (Answer: $H$ is symmetric, $H^\mathsf{T}=H$, and idempotent, $H^2=H$.)
A proposed hat matrix is $A=(X^\mathsf{T}X)^{-1}X^\mathsf{T}$ for a design with $n=8$ and $p=2$. Explain the error and give the correct matrix that maps $y$ to fitted values. (Answer: $A$ is $2\times8$ and maps $y$ to $\hat{\beta}$; the fitted-value matrix is $H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}$, which is $8\times8$.)

What You'll Learn