Finding High-Leverage Patterns In Design Matrices

What Design Patterns Create High Leverage?

A GLM observation can be unusual because its response is surprising, because its predictors occupy an unusual position, or both. This tutorial focuses on the predictor side: which patterns in the design matrix make an observation carry unusually much leverage? Such patterns include extreme continuous values, rare factor levels, scarcely observed combinations of factors, and predictor combinations that reveal a direction nearly hidden by collinearity.

GLM leverage is evaluated using a weighted design matrix, so it depends on both the design row and its current working weight. It describes the observation’s geometric position in the local weighted fit, not whether its response is well predicted. A high value is a reason to inspect the design and its support; it is not, by itself, evidence of an error.

Definition: Let $x_i^\mathsf{T}$ be row $i$ of a full-column-rank design matrix $X$, and let $W=\operatorname{diag}(w_1,\ldots,w_n)$ be the diagonal matrix of positive working weights at the fitted GLM. The weighted hat matrix and the leverage of observation $i$ are $$ H=W^{1/2}X(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W^{1/2}, \qquad h_{ii}=w_i x_i^\mathsf{T}(X^\mathsf{T}WX)^{-1}x_i. $$ The $h_{ii}$ are the diagonal elements of $H$. They measure how strongly each weighted design row is represented in the fitted linear-predictor space.

The numerical leverage values below use equal positive weights to make the design geometry transparent. In an actual GLM, the working weights can vary with the fitted means, so the same design pattern may have different leverage at different fits. A pattern should therefore be interpreted in the context of the model’s predictors, coding, and current weights.

Two Basic Facts for Reading Leverage

The hat matrix is a projection in the weighted design space. Its diagonal entries are nonnegative and sum to the number of fitted coefficients. This gives a useful check on calculations and explains why observations with high leverage are balanced by others with lower leverage.

Result: If $X$ has full column rank $p$ and all $w_i>0$, then $H$ is symmetric and idempotent, $0\leq h_{ii}\leq 1$, and $$ \sum_{i=1}^n h_{ii}=p. $$

Proof. Set $A=W^{1/2}X$. Then $H=A(A^\mathsf{T}A)^{-1}A^\mathsf{T}$. The matrix $A^\mathsf{T}A$ is invertible because $X$ has full column rank and $W$ is positive definite. Symmetry follows because $A^\mathsf{T}A$ and its inverse are symmetric. Also, $$ H^2=A(A^\mathsf{T}A)^{-1}(A^\mathsf{T}A)(A^\mathsf{T}A)^{-1}A^\mathsf{T} =A(A^\mathsf{T}A)^{-1}A^\mathsf{T}=H. $$ Thus $H$ is a symmetric projection. Its eigenvalues are 0 or 1, so each diagonal entry, which equals $e_i^\mathsf{T}He_i$, is between 0 and 1: it is the squared length of the projection of the unit vector $e_i$. Finally, by the trace identity, $$ \sum_{i=1}^n h_{ii}=\operatorname{tr}(H) =\operatorname{tr}\!\left((A^\mathsf{T}A)^{-1}A^\mathsf{T}A\right) =\operatorname{tr}(I_p)=p. $$ $\square$

These facts do not establish a universal cutoff for “high” leverage. A rule based on $p/n$, or on a multiple of $p/n$, can be a screening aid, but it cannot identify whether a design pattern is scientifically valid or adequately supported. That requires examining the predictors that produce the value.

Continuous Predictors: Extremes at the Edge

With an intercept and one continuous predictor, equal working weights give a particularly clear formula. The leverage at predictor value $x_i$ is $$ h_{ii}=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{\sum_{j=1}^n(x_j-\bar{x})^2}. $$ The first term is shared across observations; the second grows with distance from the predictor mean. Values far beyond the central range can therefore have high leverage, even if their responses are ordinary.

Worked Example: One Extreme Continuous Value

Suppose the predictor values are $0,1,2,10$, with an intercept and slope and equal positive working weights. Here $n=4$ and $p=2$. First compute $$ \bar{x}=\frac{0+1+2+10}{4}=3.25 $$ and $$ \sum_{j=1}^4(x_j-\bar{x})^2 =(-3.25)^2+(-2.25)^2+(-1.25)^2+(6.75)^2=62.75. $$ The leverage for the observation at $x=10$ is $$ h_{ii}=\frac14+\frac{(10-3.25)^2}{62.75} =0.25+\frac{45.5625}{62.75} \approx 0.9761. $$ For comparison, the leverage at $x=1$ is $$ \frac14+\frac{(1-3.25)^2}{62.75} =0.25+\frac{5.0625}{62.75} \approx 0.3307. $$ The four leverage values are approximately $0.4183, 0.3307, 0.2749,$ and $0.9761$; they sum to $2=p$, as required. The value at 10 is high because it is far from the center of the observed predictor values. Whether it is a valid observation depends on the study: it may be a genuine but sparsely represented part of the predictor range, or it may signal a unit or recording error.

Categorical Predictors: Rare Levels

A factor level represented by few observations can produce high leverage, particularly when the model gives that level its own fitted mean. In that setting, the relevant comparison is not simply whether the level is unusual in the population; it is how many observations support the model parameter associated with it.

Result: In a model with one separate fitted mean for each factor level, suppose level $g$ has $m_g$ observations, all with equal positive working weights. Every observation in that level has leverage $1/m_g$.

Derivation. The design columns are indicators for the levels. For level $g$, its column has a 1 for each of its $m_g$ observations and 0 elsewhere. The corresponding diagonal entry of $X^\mathsf{T}X$ is $m_g$, and the level-indicator columns are mutually orthogonal. The projection matrix for that level therefore has entry $1/m_g$ on each of its $m_g$ diagonal positions. In particular, a level represented once has leverage 1. With unequal weights, the analogous leverage within that level is the observation’s weight divided by the sum of the weights in the level.

Worked Example: A Rare Factor Level

Consider 10 observations in a model with an intercept and a binary factor: nine are in the reference level and one is in the other level. This is a separate-mean model with $p=2$. Each reference-level observation has leverage $1/9$, while the sole observation in the other level has leverage $1/1=1$. The total is $$ 9\left(\frac19\right)+1=2=p. $$ The single observation does not have high leverage because of its response. It has leverage 1 because it is the only observation supporting that level’s fitted mean. If the level is a genuine, important group, the pattern may be scientifically meaningful but thinly supported. If its label arose from inconsistent coding, the remedy is to investigate the coding rather than to interpret the leverage as a property of the population.

Interactions: Sparse Combinations

An interaction allows the fitted relationship to vary across combinations of predictors. When two categorical factors have a full interaction, each observed combination can receive its own fitted mean. A combination with few observations then has high leverage, even if both factors separately have many observations. Marginal counts alone can hide this sparsity.

Worked Example: One Sparse Interaction Cell

Suppose a full interaction between two binary factors is fitted, with cell counts $4,4,4,$ and $1$. The model has four cell means, so $p=4$. As in a separate-mean model, leverage within a cell is the reciprocal of its count. Thus the leverage values are $1/4$ for each observation in the first three cells and $1$ for the single observation in the last cell. The sum is $$ 3\left(4\cdot\frac14\right)+1=4=p. $$ The two factors may each have substantial overall representation, but their one particular combination has only one supporting observation. Check a cross-tabulation of the factors, not just their separate frequencies. If the combination is allowed by the study design, its leverage identifies limited support for that cell. If it should not occur, check for a coding or data-collection problem. Either way, the model cannot obtain replication within that cell from observations in other combinations.

Collinear Predictors: A Thinly Supported Direction

Strong collinearity makes coefficient estimates unstable, but it does not automatically make every leverage value high. Leverage is determined by the column space of the design, not by the size of individual coefficients or by a predictor correlation considered alone. A useful warning pattern occurs when nearly redundant predictors differ noticeably only for a small number of observations. Those observations may then reveal a design direction supported by very little data.

Worked Example: A Rare Contrast Between Nearly Collinear Predictors

Take six observations, an intercept, and two predictors. Let $$ x_1=(-1,-1,-1,1,1,1),\qquad z=(0,0,0,0,0,1),\qquad x_2=x_1+0.02z. $$ The predictors $x_1$ and $x_2$ are almost identical: their sample correlation is approximately $0.99998$. Yet the design with columns $(1,x_1,x_2)$ has rank 3, because $z=(x_2-x_1)/0.02$. Replacing $x_2$ by $z$ does not change the column space, so the leverage is the same as for the columns $(1,x_1,z)$.

In that equivalent design, the rows split into three distinct patterns: $(1,-1,0)$ occurs three times, $(1,1,0)$ twice, and $(1,1,1)$ once. The fitted space is the space of three separate group means. Therefore, the leverages are $1/3$ for each of the first three observations, $1/2$ for each of the next two, and $1$ for the final observation. They sum to $$ 3\left(\frac13\right)+2\left(\frac12\right)+1=3=p. $$ The final observation exposes a contrast between predictors that is absent from the other five. Its leverage is high because that design pattern occurs once, not because a high correlation mechanically inflates every diagonal entry. Inspect whether the contrast is plausible and whether the data provide enough support to estimate its role reliably.

How to Diagnose a High-Leverage Pattern

A useful diagnosis starts by identifying the feature of the design row responsible for the value. The same numerical leverage can arise from different problems: an extreme but valid measurement, a rare factor level, an interaction cell with little replication, or a predictor contrast represented by only a few observations. Use the model’s actual coding and current weights when tracing the pattern.

Locate the pattern.
For each high-leverage observation, identify extreme continuous values, factor levels, interaction combinations, and unusual combinations of correlated predictors.

Check support and data quality.
Count observations with the same or nearby design pattern. Verify units, categories, reference levels, and interaction coding before treating the pattern as genuine.

Judge its scientific role.
Ask whether the pattern is part of the intended study population or design, and whether it has enough replication for the model’s level of detail.

Keep the interpretation limited.
High leverage flags a predictor-space pattern. It does not say that the response is poorly fitted, that the observation is erroneous, or that it changes the fitted model substantially.

A common mistake is to call every high-leverage observation an outlier and delete it. That reasoning confuses predictor geometry with response discrepancy: leverage is calculated from the design and weights, whereas residuals describe response differences from the fit. The correction is to investigate the design pattern first and assess response discrepancy separately. A valid, rare design point may be important precisely because it represents a region that would otherwise be absent; deleting it solely for high leverage can remove the only information about that region.

Key takeaway: High GLM leverage often points to a thinly represented direction or region of predictor space. Determine whether that pattern is valid and scientifically relevant, and how much support it has, before drawing conclusions from its leverage value.

Check Your Understanding

For each question, focus on the design pattern that produces leverage.

In an intercept-and-slope model with equal working weights, why can a continuous predictor value far from the sample mean have high leverage?
A factor level occurs once in a separate-mean model. What is that observation’s leverage, and what does the value say about replication?
Why can a full interaction have a high-leverage observation even when both factors have common levels overall?
When two predictors are nearly collinear, what additional design feature can make a small number of observations especially high leverage?
Why is high leverage alone not a reason to label an observation a response outlier or delete it?