Weighted Projection Geometry and the Hat Matrix

Why Weighted Projection Geometry Matters

Weighted least squares chooses coefficients so that the fitted response is close to the observed response, while giving observations different weights in the squared-error criterion. The resulting geometry is not mysterious: after scaling each observation by the square root of its weight, the problem becomes ordinary least squares in transformed coordinates. In those coordinates, the fitted vector is an orthogonal projection.

This viewpoint gives a precise meaning to the weighted hat matrix. It also clarifies two related ideas that are easy to confuse: the matrix that projects the transformed response, and the matrix that maps the original response to its fitted values. Their diagonal entries agree, so they give the same observation-by-observation leverage.

From Weighted Least Squares to a Projection

The objective can be rewritten as an ordinary sum of squares by defining the transformed response \(y^*=W^{1/2}y\) and transformed design matrix \(X^*=W^{1/2}X\). Indeed, \((y-X\beta)^\mathsf{T}W(y-X\beta)=\|y^*-X^*\beta\|^2\). Because the columns of \(X^*\) are linearly independent, its least-squares coefficient estimate is \(\hat{\beta}=(X^\mathsf{T}WX)^{-1}X^\mathsf{T}Wy\). Multiplying the fitted transformed response by the corresponding matrices gives the projection formula below.

Thus \(H\) acts on \(y^*\), not directly on \(y\). The fitted response on the original scale is \(\hat{y}=X\hat{\beta}\). Its mapping from the original response is given by a different matrix, often denoted \(S\):

The matrix \(S\) need not be symmetric in the ordinary Euclidean geometry. It is the matrix \(H\) that is symmetric and represents an ordinary orthogonal projection. Since \(S\) and \(H\) have the same diagonal entries, either matrix gives the same leverage for each observation.

Projection Properties and Their Proof

The geometric claims follow directly from writing the weighted hat matrix in terms of the transformed design \(X^*=W^{1/2}X\). The columns of \(X^*\) span the model’s fitted subspace in transformed coordinates.

The normal equations also show why \(\hat{y}\) is the weighted projection of \(y\): among fitted vectors \(X\beta\), the chosen one leaves a residual orthogonal to the model space under the weighted inner product \(\langle u,v\rangle_W=u^\mathsf{T}Wv\). The theorem expresses the same fact as an ordinary orthogonal projection after transformation.

Leverage: Bounds, Total, and Interpretation

The diagonal entries \(h_{ii}\) are called leverages. Since \(H\) is an orthogonal projection, each \(h_{ii}\) is the squared length of the projection of the \(i\)th coordinate unit vector onto the fitted subspace. This provides a direct proof of their bounds. It also shows that leverage depends on the design and weights, not on the observed response values.

To prove the bounds, let \(e_i\) be the \(i\)th coordinate unit vector. Symmetry and idempotence give \(h_{ii}=e_i^\mathsf{T}He_i=(He_i)^\mathsf{T}(He_i)=\|He_i\|^2\geq0\). An orthogonal projection cannot increase length, so \(\|He_i\|\leq\|e_i\|=1\), giving \(h_{ii}\leq1\). For the total, the trace of a projection equals its rank: its eigenvalues are 1 on its \(p\)-dimensional range and 0 on the orthogonal complement. Thus \(\sum_i h_{ii}=\operatorname{tr}(H)=p\). Finally, taking the \(i\)th diagonal element of the matrix formula for \(H\) gives the stated row-wise expression.

A higher leverage means that the observation’s response has more opportunity to affect the fitted value at that location, conditional on the design and weights. It does not by itself establish that the observation is an outlier or that it has a large residual. Those are separate properties of the observed response relative to the fitted model.

Common Mistake: Applying the Wrong Hat Matrix

A common error is to call \(S=X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W\) the symmetric weighted projection matrix. It maps \(y\) to the original-scale fit, but in general \(S^\mathsf{T}\neq S\). The symmetric projection matrix is \(H\), which maps \(W^{1/2}y\) to \(W^{1/2}\hat{y}\). The correction is not to discard \(S\): use \(S\) when mapping the original response to original-scale fitted values, and use \(H\) when invoking ordinary Euclidean projection properties. Their diagonal entries coincide, which is why either gives the same leverage values.

A second tempting shortcut is to interpret a large diagonal entry as proof of an unusual response. The calculation \(h_{ii}=w_i x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i\) shows why that conclusion is unsupported: no \(y_i\) appears. Leverage describes the observation’s position in the weighted design, not whether its response is surprising.