Predictor Geometry and Leverage

Why Predictor Geometry Matters

A case can sit far from the others in the predictor values even when its response is ordinary. Conversely, a response can be unusual for a case whose predictor values are near the center of the data. Leverage describes the first kind of unusualness: it measures how a case is positioned in the predictor space used to fit the regression.

In a scatterplot for simple regression, predictor position is easy to see along the horizontal axis. With several predictors, position is harder to assess by eye: a case may be ordinary on each axis separately but unusual in the combination of values it has. The geometry of the predictors provides a precise way to describe both situations.

Consider the ordinary least-squares model with an intercept and \(k\) predictors. For observation \(i\), write its predictor row as \(\mathbf{x}_i^T\), and let the design matrix \(X\) contain all \(n\) predictor rows, including a column of ones for the intercept. Assume \(X\) has full column rank, so the least-squares fit is uniquely defined. The response values do not enter the leverage calculation; no normality or constant-variance assumption is needed to calculate it.

One Predictor: Distance From the Center

For simple linear regression with an intercept, leverage has a particularly direct form. Let \(x_i\) be the predictor value for case \(i\), let \(\bar{x}\) be the sample mean of the predictor values, and let \(S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2\). Then

The first term is shared by all observations. The second is nonnegative and grows with squared distance from the predictor mean, relative to the overall spread of the predictor values. Thus, in a horizontal-versus-vertical plot of response against predictor, leverage is determined by horizontal position—not by how high or low the point is vertically.

The formula also shows that leverage is relative to the arrangement of the whole predictor sample. A given distance from \(\bar{x}\) contributes less when \(S_{xx}\) is larger. For a fixed data set, points at either end of the predictor range tend to have more leverage than points near the center, though the exact values depend on the locations of all observations.

Several Predictors: Distance in the Joint Geometry

With \(k\) predictors, one-dimensional distance is no longer enough. Let \(\mathbf{z}_i\) be the column vector of the \(k\) predictor values for observation \(i\), and let \(\bar{\mathbf{z}}\) be the vector of their sample means. Define the centered predictor matrix \(Z\) to have rows \((\mathbf{z}_i-\bar{\mathbf{z}})^T\). If the predictors are not linearly dependent after centering, then \(Z^TZ\) is invertible, and leverage can be written as

The second term is a squared distance in the geometry induced by the observed predictor spread. It is not usually the ordinary squared Euclidean distance, which simply adds squared differences in the predictor coordinates. Instead, it takes account of how widely the data vary in different directions and how the predictors vary together. A displacement along a direction in which the observed data are tightly clustered can count more than an equally long displacement along a direction in which they are widely spread.

For a picture, imagine the cases as points in a plane for two predictors. Their overall pattern may form an elongated cloud, rather than a circular one. A point far from the center across the narrow width of that cloud can have greater leverage than a point with a similar ordinary distance along the cloud’s long direction. With more predictors, the same idea applies in higher-dimensional predictor space, even though it cannot be drawn on a flat page.

What Predictor Count and Arrangement Tell Us

A useful check on any leverage calculation is that the leverages sum to the number of columns in the full-rank design matrix. With an intercept and \(k\) predictors, there are \(k+1\) columns. Consequently,

This means that the average leverage increases as the number of predictors increases, for a fixed sample size. It does not mean that every case must have the same leverage, or that every individual leverage necessarily rises when a predictor is added. In a nested model, adding a new independent predictor expands the fitted column space; the corresponding projection gives each case at least as much leverage as before, while the total increases by one.

The arrangement of predictor values determines how that total is distributed among the cases. In simple regression, distance from \(\bar{x}\) relative to \(S_{xx}\) controls the distribution. In multiple regression, the distribution also reflects predictor correlations and the shape of the joint cloud. Leverage remains unchanged if predictor units are converted or the predictor coordinates are otherwise changed by an invertible linear transformation, provided the model includes the same intercept and spans the same predictor space. A change from meters to centimeters, for example, does not make a case more or less leveraged.

A Short Justification of the Multivariable Formula

With an intercept, write the design matrix as \(X=[\mathbf{1}\; Z]\), where \(\mathbf{1}\) is the column of ones and \(Z\) is centered, so \(\mathbf{1}^TZ=\mathbf{0}^T\). The columns of \(Z\) are therefore orthogonal to the intercept column. The projection onto the intercept column is \(\mathbf{1}\mathbf{1}^T/n\), and the projection onto the centered predictor columns is \(Z(Z^TZ)^{-1}Z^T\). Since these spaces are orthogonal, their projection matrices add:

Taking the \(i\)th diagonal entry gives \(1/n\) from the intercept projection and \((\mathbf{z}_i-\bar{\mathbf{z}})^T(Z^TZ)^{-1}(\mathbf{z}_i-\bar{\mathbf{z}})\) from the centered predictor projection. This yields the stated leverage formula. Also, the trace of a projection matrix is its rank; here the rank is \(k+1\). Because the trace is the sum of the diagonal entries, the leverages sum to \(k+1\).

Common Mistakes and Why Leverage Matters

• Using response position instead of predictor position. A large vertical residual is not, by itself, high leverage. Leverage is determined by the predictor values; residual size and influence are distinct diagnostic ideas.
• Checking each predictor in isolation. A case can be ordinary on each separate axis but unusual in the combination of predictor values. Inspect joint predictor patterns where possible.
• Equating raw distance with leverage. In multiple regression, distance must be considered relative to the spread and association of the predictors. A long direction of the predictor cloud and a narrow direction do not count equally.
• Assuming more predictors make every case equally high-leverage. The average rises with the number of independent predictors, but individual values depend on the arrangement of the observations.

Predictor geometry matters because it identifies where the regression fit has observations that are separated from the rest in predictor space. Leverage alone does not establish that a case is influential: a case’s effect on the fitted results also depends on how its response relates to the model. It is therefore a description of design position, not a verdict about whether a case should be removed.