Leverage and Mahalanobis Residual Geometry

Why Residual Size Depends on the Design

A large residual can signal that a response is poorly described by a fitted regression, but the residual’s scale also depends on where the case lies in the predictor design. A case with unusual predictor values can have high leverage: the fitted model is pulled toward that case, which tends to make its own residual smaller. With multiple responses, a residual’s size must also account for the scales and correlations among outcomes. The hat matrix and a Mahalanobis distance provide a way to describe these two features together.

Let \(Y\) be the \(n \times p\) response matrix, \(X\) the \(n \times q\) design matrix of full column rank, and \(B\) the \(q \times p\) coefficient matrix. The multivariate linear model has fitted values \(\hat{Y}=HY\), where \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\). The residual matrix is \(R=(I-H)Y\). Its \(i\)th row, written \(r_i\), is the residual vector for case \(i\), regarded as a row vector of length \(p\).

Definition: The hat matrix is \(H=X(X^\mathsf{T}X)^{-1}X^\mathsf{T}\), the projection onto the column space of \(X\). The leverage of case \(i\) is the diagonal element \(h_{ii}\). It describes the case’s position in the predictor design, not whether its response is unusual. The residual-maker matrix is \(M=I-H\), so \(R=MY\).

Leverage in the Predictor Design

Because \(H\) is a symmetric idempotent projection, \(h_{ii}\) lies between zero and one, and the leverages sum to the rank of \(X\), which here is \(q\). A high value means that the case has relatively strong influence on the fitted value at its own design point. It does not, by itself, imply a large residual or an influential case under every diagnostic definition.

Worked Example: Leverage in a Simple Regression Design

Suppose a regression has an intercept and one predictor with values \(-1,0,1,4\). For a simple regression with an intercept, the leverage is \(h_{ii}=1/n+(x_i-\bar{x})^2/\sum_j(x_j-\bar{x})^2\). Calculate the four leverages.

Here \(n=4\), \(\bar{x}=(-1+0+1+4)/4=1\), and \(\sum_j(x_j-\bar{x})^2=(-2)^2+(-1)^2+0^2+3^2=14\). Thus the four values are \(1/4+4/14=15/28\), \(1/4+1/14=9/28\), \(1/4=7/28\), and \(1/4+9/14=25/28\).

The leverages sum to \((15+9+7+25)/28=2\), the number of columns of \(X\). The case with predictor value \(4\) has the largest leverage, \(25/28\), because its predictor value is farthest from the mean. This conclusion concerns the design alone; its residual cannot be determined from the predictor values.

Result: For a design matrix \(X\) of rank \(q\), \(0 \leq h_{ii} \leq 1\) for every case and \(\sum_{i=1}^n h_{ii}=q\). Consequently, the average leverage is \(q/n\).

To verify the bounds, let \(e_i\) be the \(i\)th coordinate vector. Since \(H\) is a symmetric idempotent projection, \(h_{ii}=e_i^\mathsf{T}He_i=\|He_i\|^2\). Also, \(e_i=He_i+(I-H)e_i\) is a sum of orthogonal vectors, so \(\|He_i\|^2 \leq \|e_i\|^2=1\). This proves \(0\leq h_{ii}\leq 1\). Finally, the trace of a projection equals its rank, so \(\sum_i h_{ii}=\operatorname{tr}(H)=q\). \(\square\)

Estimating Multivariate Residual Size

A residual vector cannot be summarized well by adding its squared components unless the responses are on comparable scales and uncorrelated. Instead, use the response covariance to measure residual size. Under the usual independent-case model with common response covariance, an unbiased estimate is the residual covariance matrix \(S=R^\mathsf{T}R/(n-q)\). This estimate is usable for Mahalanobis distances only when it is invertible; a sufficient condition is that \(R\) has column rank \(p\), which requires \(n-q\geq p\).

Definition: When \(S\) is positive definite, the squared Mahalanobis residual distance for case \(i\) is \(d_i^2=r_iS^{-1}r_i^\mathsf{T}\). A leverage-adjusted version is \(d_{i,\mathrm{adj}}^2=d_i^2/(1-h_{ii})\), defined only when \(h_{ii}<1\). The first distance uses the estimated response covariance to account for scale and correlation; the adjustment accounts for the reduced variance of a fitted residual at a high-leverage case.

The adjustment has a covariance-based rationale. Suppose \(Y=XB+U\), the rows \(u_j\) of \(U\) have mean zero and common covariance matrix \(\Sigma\), and distinct rows are uncorrelated. Normality is not needed for the following covariance result.

Result: Under those conditions, \(\operatorname{Cov}(r_i)=(1-h_{ii})\Sigma\). More generally, \(\operatorname{Cov}(r_i^\mathsf{T},r_j^\mathsf{T})=-h_{ij}\Sigma\) for distinct cases \(i\) and \(j\).

To prove the result, write \(M=I-H\). Since \(MX=0\), the residual matrix is \(R=MU\). In particular, \(r_i=\sum_k M_{ik}u_k\). Independence is not required; uncorrelatedness of distinct rows and their common covariance give \(\operatorname{Cov}(r_i^\mathsf{T})=\sum_k M_{ik}^2\Sigma\). Symmetry and idempotence of \(M\) imply \(\sum_k M_{ik}^2=(M^2)_{ii}=M_{ii}=1-h_{ii}\). For two cases \(i,j\), the same calculation gives \(\operatorname{Cov}(r_i^\mathsf{T},r_j^\mathsf{T})=(M^2)_{ij}\Sigma=M_{ij}\Sigma=-h_{ij}\Sigma\). \(\square\)

If \(\Sigma\) were known, the expected unadjusted squared Mahalanobis residual size would be \(E[r_i\Sigma^{-1}r_i^\mathsf{T}]=p(1-h_{ii})\). Indeed, for a zero-mean row vector \(r_i\), the expectation of this quadratic form is \(\operatorname{tr}(\Sigma^{-1}\operatorname{Cov}(r_i^\mathsf{T}))\), which here is \((1-h_{ii})\operatorname{tr}(I_p)=p(1-h_{ii})\). Dividing by \(1-h_{ii}\) removes this leverage-dependent expected scale when covariance is known. Replacing \(\Sigma\) by \(S\) produces a useful descriptive adjustment, but does not make the resulting distances independent or give them an exact chi-square distribution.

Worked Example: A Mahalanobis Distance With Two Responses

Consider an intercept-only model with four cases and two responses. Suppose the residual rows are \(r_1=(1,0)\), \(r_2=(-1,0)\), \(r_3=(0,2)\), and \(r_4=(0,-2)\). They sum to zero, as required for residuals from an intercept-only fit. Here \(q=1\), so each leverage is \(h_{ii}=1/4\), and the residual degrees of freedom are \(4-1=3\).

The residual SSCP matrix and its unbiased covariance estimate are \(R^\mathsf{T}R=\begin{pmatrix}2&0\\0&8\end{pmatrix}\) and \(S=R^\mathsf{T}R/3=\begin{pmatrix}2/3&0\\0&8/3\end{pmatrix}\). Therefore \(S^{-1}=\begin{pmatrix}3/2&0\\0&3/8\end{pmatrix}\).

For the first case, \(d_1^2=(1,0)S^{-1}(1,0)^\mathsf{T}=3/2\). For the third case, \(d_3^2=(0,2)S^{-1}(0,2)^\mathsf{T}=4(3/8)=3/2\). The raw distances are equal even though the residual vectors have different component magnitudes: the estimated covariance records that the second response varies more. Dividing either distance by \(1-h_{ii}=3/4\) gives the adjusted value \(2\).

How Leverage Changes the Interpretation

A high-leverage case has a residual with smaller variance under the model, so the same observed residual magnitude can be more notable there than at a low-leverage case. The adjustment by \(1-h_{ii}\) makes this scale difference explicit. It does not change the case’s residual vector, repair an inadequate model, or establish that the case is an outlier. It is a way to interpret residual magnitude relative to the fitted design.

Worked Example: Comparing a High-Leverage Residual

Take four predictor values \(-1,0,1,3\) in a regression with an intercept. Let the two response residuals, by case, be the rows of \(R=\begin{pmatrix}1&0\\-2&2\\1&-3\\0&1\end{pmatrix}\). Each column is orthogonal to both the intercept and the predictor: the column sums are zero, and the predictor-weighted sums are \(-1+1=0\) and \(0-3+3=0\). Thus these are valid residual columns for this design.

The design has \(\bar{x}=3/4\) and \(\sum_j(x_j-\bar{x})^2=35/4\). The leverage of the last case is \(1/4+(3-3/4)^2/(35/4)=1/4+9/35=29/35\). For the first case, the leverage is \(1/4+(-1-3/4)^2/(35/4)=1/4+7/20=3/5\).

Here \(n-q=2\), and direct multiplication gives \(R^\mathsf{T}R=\begin{pmatrix}6&-7\\-7&14\end{pmatrix}\), so \(S=\begin{pmatrix}3&-7/2\\-7/2&7\end{pmatrix}\). Its determinant is \(21-49/4=35/4>0\), and \(S^{-1}=\begin{pmatrix}4/5&2/5\\2/5&12/35\end{pmatrix}\). For the high-leverage last case, \(r_4=(0,1)\), giving \(d_4^2=12/35\) and \(d_{4,\mathrm{adj}}^2=(12/35)/(1-29/35)=2\). For the first case, \(r_1=(1,0)\), giving \(d_1^2=4/5\) and \(d_{1,\mathrm{adj}}^2=(4/5)/(1-3/5)=2\). The high-leverage case has the smaller raw distance, but the leverage-adjusted distances are equal.

Common Mistake

Wrong reasoning: “A small residual means the case fits well, regardless of leverage.” A high-leverage case can pull its fitted value toward its observed response, reducing its residual variance.
Correction: Interpret residual size in relation to both response covariance and leverage. A Mahalanobis distance accounts for response scales and correlations; dividing by \(1-h_{ii}\) accounts for the leverage-dependent residual scale. Neither quantity alone proves that a case is an outlier or influential.
Another pitfall: Treating the adjusted distance as an exact test statistic. Its covariance motivation does not establish a chi-square reference distribution, particularly when the covariance is estimated from the same residuals.

Key takeaway: The diagonal \(h_{ii}\) describes a case’s leverage in the predictor design. Under the common-covariance model, the residual covariance is scaled by \(1-h_{ii}\). A Mahalanobis residual distance uses the estimated response covariance, and its leverage adjustment helps make comparisons across cases more meaningful.

Check Your Understanding

Work through these questions using the definitions and results above.

What feature of a case does \(h_{ii}\) describe, and what does it not tell you by itself?
For a design of rank \(q\), what is the sum and average of the diagonal leverages?
Why does the covariance of a fitted residual contain the factor \(1-h_{ii}\)?
What does \(S^{-1}\) contribute to a squared Mahalanobis residual distance?
Why should a leverage-adjusted residual distance not automatically be interpreted as a chi-square test statistic?