Random Leverage and Its Role in OLS Limits

Why Leverage Matters Under Random Sampling

In random-design regression, the design matrix is itself random. A particular sample can place some observations in directions that strongly affect the fitted values, while other observations contribute little in those directions. Leverage gives a geometric measure of each observation’s position in the realized design. Its diagonal entries are random across samples, so their algebraic properties and their probability properties must be kept distinct.

Leverage is useful for understanding how design concentration can affect OLS calculations. It is not, by itself, a measure of an observation’s residual, its realized effect on the estimated coefficient, or whether a normal approximation holds. Those also depend on the error and on the full design.

Definition and Basic Geometry

Let $x_i\in\mathbb{R}^p$ be the column vector of regressors for observation $i$, and let $X$ be the $n\times p$ matrix whose $i$th row is $x_i'$. Suppose $X$ has full column rank, so $X'X$ is invertible. The hat matrix maps the response vector to its OLS fitted values.

Definition: The hat matrix and the diagonal leverage of observation $i$ are $$ H=X(X'X)^{-1}X', \qquad h_{ii}=x_i'(X'X)^{-1}x_i. $$ When the design is random, $h_{ii}$ is a random variable. It depends on the entire realized design $X$, not just on $x_i$.

The matrix $H$ is the orthogonal projection onto the column space of $X$. Consequently it is symmetric and idempotent: $H'=H$ and $H^2=H$. These facts give useful restrictions on its diagonal.

Theorem: If $X$ has full column rank $p$, then $0\leq h_{ii}\leq 1$ for every $i$, and $$ \sum_{i=1}^n h_{ii}=p. $$

Proof. Let $e_i$ be the $i$th coordinate vector. Since $H$ is an orthogonal projection, $h_{ii}=e_i'He_i=\|He_i\|^2$, so $h_{ii}\geq0$. Also $e_i=He_i+(I-H)e_i$, and the two terms are orthogonal. Therefore $\|He_i\|^2\leq\|e_i\|^2=1$, proving $h_{ii}\leq1$. Finally, the trace of a projection equals its rank. Here $\operatorname{rank}(H)=p$, so $\sum_i h_{ii}=\operatorname{tr}(H)=p$. $\square$

The sum restriction is exact for each full-rank sample. It implies that at least one leverage is at least $p/n$, and at least one is at most $p/n$. It does not imply that every observation has leverage $p/n$, nor does it imply that the largest leverage is small as $n$ grows.

Worked Example: Leverage in a Simple Regression Design

Consider a regression with an intercept and one predictor, with observed predictor values $z=( -1,-1,1,5)'$. The sample mean is $\bar z=1$, and the centered sum of squares is $$ S_{zz}=\sum_{i=1}^4(z_i-\bar z)^2 =4+4+0+16=24. $$

In a simple regression with an intercept, the diagonal leverage is $h_{ii}=1/n+(z_i-\bar z)^2/S_{zz}$. Thus the four leverages are $$ h_{11}=\frac14+\frac4{24}=\frac5{12},\qquad h_{22}=\frac14+\frac4{24}=\frac5{12}, $$ $$ h_{33}=\frac14+\frac0{24}=\frac14,\qquad h_{44}=\frac14+\frac{16}{24}=\frac{11}{12}. $$

All four values lie between zero and one, and their sum is $5/12+5/12+1/4+11/12=2$, the number of regressors including the intercept. The fourth observation has high leverage because its predictor value is far from the sample mean; this is a statement about the design, not about its response or residual.

Random Leverage and Exchangeability

Under random sampling, each $h_{ii}$ varies with the sampled design. A useful expectation follows when the rows are jointly exchangeable: their joint distribution is unchanged by any permutation of the observation indices. IID rows are one sufficient example. Identical marginal distributions alone are not enough.

Theorem: Suppose $X$ has full column rank $p$ with probability one and its rows are jointly exchangeable. Then the leverage vector $(h_{11},\ldots,h_{nn})$ is exchangeable, and $$ E[h_{ii}]=\frac pn,\qquad i=1,\ldots,n. $$

Proof. Let $P$ be any $n\times n$ permutation matrix. Permuting the rows replaces $X$ by $PX$. The corresponding hat matrix is $$ H(PX)=PX\big((PX)'PX\big)^{-1}(PX)' =PX(X'X)^{-1}X'P'=PHP'. $$ Thus permuting the rows of $X$ permutes the diagonal entries of $H$ in the same way. Joint exchangeability of the rows therefore implies exchangeability of the leverage vector. In particular, all $E[h_{ii}]$ are equal; they exist because $0\leq h_{ii}\leq1$. Taking expectations in the samplewise identity $\sum_i h_{ii}=p$ gives $\sum_i E[h_{ii}]=p$. The $n$ equal expectations must each be $p/n$. $\square$

This is an unconditional expectation over the random design. It does not say that the realized leverages are equal, and it does not automatically describe their expectations conditional on additional information about the sample.

Worked Example: Expected Leverage Under IID Sampling

Let $n=2$, $p=1$, and suppose the scalar rows $x_1,x_2$ are IID $N(0,1)$. The design has full column rank with probability one, since $P(x_1=x_2=0)=0$. Its leverages are $$ h_{11}=\frac{x_1^2}{x_1^2+x_2^2}, \qquad h_{22}=\frac{x_2^2}{x_1^2+x_2^2}. $$

The pair $(x_1,x_2)$ is exchangeable, so the theorem gives equal expected leverages. For every full-rank sample, $h_{11}+h_{22}=1$. Hence $$ E[h_{11}]=E[h_{22}], \qquad E[h_{11}]+E[h_{22}]=1, \qquad E[h_{11}]=E[h_{22}]=\frac12=\frac pn. $$

For an individual sample the two values need not be $1/2$: if $x_1^2$ is much larger than $x_2^2$, then $h_{11}$ is close to one. The equality $p/n$ describes the expectation, not each random realization.

Leverage and Individual Contributions to OLS

Suppose the linear regression model is $y_i=x_i'\beta+u_i$, and $X$ has full column rank. The OLS estimation error can be written as a sum of observation-level contributions: $$ \hat\beta-\beta=(X'X)^{-1}\sum_{i=1}^n x_i u_i. $$ To see the geometric meaning of leverage in this expression, use standardized coefficient coordinates. Let $q_i=(X'X)^{-1/2}x_i$, where the inverse square root is the symmetric positive-definite one. Then $$ (X'X)^{1/2}(\hat\beta-\beta)=\sum_{i=1}^n q_i u_i, \qquad q_i'q_i=h_{ii}. $$

Thus $\sqrt{h_{ii}}$ is the length of observation $i$’s regressor vector after standardizing by the sample Gram matrix. The standardized contribution is the vector $q_i u_i$: its length is $\sqrt{h_{ii}}\,|u_i|$. Leverage describes the design factor in this contribution, while the error supplies another factor. A high-leverage observation with a small error can have a small realized contribution; a lower-leverage observation with a large error need not.

Result: If $E[u\mid X]=0$ and the errors are conditionally uncorrelated with $\operatorname{Var}(u_i\mid X)=\sigma_i^2$, then $$ \operatorname{Var}\!\left((X'X)^{1/2}(\hat\beta-\beta)\mid X\right) =\sum_{i=1}^n \sigma_i^2q_iq_i'. $$ The trace of observation $i$’s summand is $\sigma_i^2h_{ii}$.

Derivation. Conditional on $X$, the vectors $q_i$ are fixed. The standardized estimation error is $\sum_i q_i u_i$. Conditional uncorrelatedness makes all cross-covariance terms zero, leaving the displayed sum of conditional covariance matrices. The trace of the $i$th term is $\sigma_i^2\operatorname{tr}(q_iq_i')=\sigma_i^2q_i'q_i=\sigma_i^2h_{ii}$. $\square$

This trace statement uses standardized coefficient coordinates. It does not say that leverage alone determines the variance of a particular coefficient, which depends on the direction of $q_i$, or that the same variance decomposition holds when errors are conditionally correlated.

Worked Example: A Realized OLS Contribution and Its Variance

Take a one-regressor model with no intercept and the fixed design $x=(1,2,2)'$. Then $X'X=1+4+4=9$, so the leverages are $$ (h_{11},h_{22},h_{33})=\left(\frac19,\frac49,\frac49\right). $$ Their sum is $1=p$. If the realized errors are $u=(0,0,3)'$, the individual contributions to $\hat\beta-\beta$ are $x_i u_i/(X'X)$: $$ \frac{1\cdot0}{9}=0,\qquad \frac{2\cdot0}{9}=0,\qquad \frac{2\cdot3}{9}=\frac23. $$ The third observation’s leverage is $4/9$, but its realized coefficient contribution is $2/3$ because its error is also part of that contribution.

Now suppose instead that, conditional on this design, the errors are uncorrelated and have variances $(1,1,4)$. The conditional variance of the scalar coefficient error is $$ \operatorname{Var}(\hat\beta-\beta\mid X) =\sum_{i=1}^3\left(\frac{x_i}{9}\right)^2\sigma_i^2 =\frac1{81}+\frac4{81}+\frac{16}{81} =\frac{21}{81}=\frac7{27}. $$ The corresponding standardized error is $3(\hat\beta-\beta)$, whose variance is $9(7/27)=7/3$. The three contributions to that standardized variance are $\sigma_i^2h_{ii}$, namely $1/9$, $4/9$, and $16/9$, which sum to $7/3$. This verifies both the scale distinction and the role of the conditional error variances.

What Leverage Can and Cannot Say About OLS Limits

When $p$ is fixed and $n$ increases, the samplewise average leverage is exactly $p/n$, so it tends to zero. This average fact does not rule out a small number of observations having much larger leverage than the rest. Such observations indicate concentration of the sample design in the standardized geometry of the regressors. This matters when assessing whether individual terms in an OLS score can be neglected in a limiting argument.

Leverage is therefore a useful diagnostic for the design component of OLS asymptotics, not a complete asymptotic condition. It does not encode the magnitude or tail behavior of $u_i$, and the conditional variance calculation also depends on $\sigma_i^2$. A large leverage signals that an observation has substantial geometric capacity to matter; it does not alone prove that a limit theorem fails or succeeds.

Common Mistakes

Assuming equal marginal row distributions imply equal expected leverages. They do not. For example, let $n=2,p=1$, and let $(x_1,x_2)$ be equally likely to be $(1,2),(2,3),(3,1)$. Both rows have the same marginal distribution, and the design has full rank in every case, but the joint distribution is not invariant to swapping rows. The first leverage is respectively $1/5,4/13,9/10$, giving $E[h_{11}]=(1/5+4/13+9/10)/3=61/130$. The second is $4/5,9/13,1/10$, giving $E[h_{22}]=69/130$. Joint exchangeability, not merely identical marginals, is what justifies the $p/n$ expectation result.
Calling leverage an outlier measure. Leverage depends on $X$, not on $y$ or the residual. A response outlier can have low leverage, and a high-leverage point can lie close to the fitted regression surface.
Reading $h_{ii}$ as the observation’s coefficient effect. The realized contribution involves $x_i u_i$ and the inverse Gram matrix. Leverage describes its standardized design length, not its realized value by itself.
Turning an average into a maximum bound. The identity $\sum_i h_{ii}=p$ implies an average of $p/n$, but does not establish that every leverage is near that average.

Key takeaway: Random leverage is a sample-dependent geometric measure. Under full rank its values lie in $[0,1]$ and sum to $p$; under joint exchangeability its expectation is $p/n$. Its connection to OLS variability becomes precise after accounting for both standardized regressors and error variation.

Check Your Understanding

Use the definitions and derivations above to answer the following questions.

Why does the sum of the diagonal leverages equal $p$ for every full-rank design, rather than only on average?
What extra distributional assumption, beyond identical row marginals, supports the conclusion $E[h_{ii}]=p/n$?
In standardized coefficient coordinates, what is the length of observation $i$’s design vector, and what additional factor enters its realized contribution?
Under conditionally uncorrelated errors, why does the trace of observation $i$’s standardized covariance contribution equal $\sigma_i^2h_{ii}$?
Give one reason why a high-leverage observation does not necessarily have a large realized effect on $\hat\beta$.