A Worked Proof of a Random-Design OLS Limit

What a Random-Design OLS Limit Requires

In random-design regression, each observation consists of a random covariate vector and a response. The design matrix is therefore random as well as the error term. An asymptotic-normality proof must control both: the empirical Gram matrix must approach a nonsingular population matrix, and the empirical score must obey a central limit theorem. The argument below makes those requirements explicit and connects them in a complete proof.

Let $(X_i,Y_i)$, $i=1,\ldots,n$, be iid, where $X_i$ is a $p$-dimensional column vector and $Y_i$ is scalar. The dimension $p$ is fixed as $n$ grows. Suppose $Y_i=X_i^\top\beta_0+u_i$, where $\beta_0$ is a fixed vector and $u_i=Y_i-X_i^\top\beta_0$. We do not require a conditional mean model $E[u_i\mid X_i]=0$; the condition needed for the coefficient expansion is the unconditional population normal equation $E[X_i u_i]=0$.

Definition: The population Gram matrix is $Q=E[X_iX_i^\top]$. The score vector for observation $i$ is $X_i u_i$, and its covariance matrix, when $E[X_i u_i]=0$, is $\Omega=E[X_iX_i^\top u_i^2]$. The OLS estimator minimizes $\sum_{i=1}^n(Y_i-X_i^\top b)^2$ over $b\in\mathbb R^p$.

For the proof, assume $E[X_i u_i]=0$, $Q$ is positive definite, $E\|X_i\|^4<\infty$, and $E[\|X_i\|^2u_i^2]<\infty$. The fourth-moment condition is convenient for establishing consistency of the residual-based covariance estimate as well as the OLS limit. The score moment condition gives a finite covariance matrix for the central limit theorem. These are sufficient conditions, not the weakest possible ones.

Theorem: Under the assumptions above, the OLS estimator exists with probability tending to one and $$ \sqrt n(\hat\beta-\beta_0)\xrightarrow{d}N\left(0,Q^{-1}\Omega Q^{-1}\right). $$ The limiting covariance may be singular if $\Omega$ is singular. If $\Omega$ is positive definite, the limit is nondegenerate.

Worked Examples: Checking the Population Quantities

Worked Example: A Scalar Homoskedastic Design

Consider regression through the origin, $Y_i=2X_i+u_i$, where $P(X_i=1)=P(X_i=-1)=1/2$ and $u_i\sim N(0,1)$ independently of $X_i$. Here $p=1$ and $\beta_0=2$.

The population Gram quantity is $Q=E[X_i^2]=1$, so it is positive. Independence and $E[u_i]=0$ give $E[X_i u_i]=E[X_i]E[u_i]=0$. The score variance is $ \Omega=E[X_i^2u_i^2]=E[X_i^2]E[u_i^2]=1$. All required moments are finite because $X_i$ is bounded and $u_i$ is normal. Thus the theorem gives $\sqrt n(\hat\beta-2)\xrightarrow{d}N(0,1)$.

Worked Example: Heteroskedasticity Changes the Limit Variance

Suppose $Y_i=3X_i+u_i$, with $P(X_i=1)=P(X_i=2)=1/2$. Conditional on $X_i=1$, let $u_i\sim N(0,1)$; conditional on $X_i=2$, let $u_i\sim N(0,4)$. In both groups the conditional error mean is zero, so $E[X_i u_i]=0$.

The Gram quantity is $Q=E[X_i^2]=(1+4)/2=5/2$. The score variance is $\Omega=E[X_i^2u_i^2]=\{1\cdot1+4\cdot4\}/2=17/2$. Therefore the limiting variance is \[ Q^{-2}\Omega=\frac{17/2}{(5/2)^2}=\frac{17/2}{25/4}=\frac{34}{25}=1.36. \] The result is $\sqrt n(\hat\beta-3)\xrightarrow{d}N(0,1.36)$. Using only the average error variance would miss the different contributions of the two covariate values to the score.

The Proof: Gram Matrix, Score, and OLS Expansion

Write $Q_n=n^{-1}\sum_{i=1}^n X_iX_i^\top$ and $S_n=n^{-1}\sum_{i=1}^n X_i u_i$. Since the observations are iid and $E\|X_i\|^2<\infty$, the law of large numbers applied to each matrix entry gives $Q_n\xrightarrow{p}Q$. Positive definiteness of $Q$ implies that $Q$ is invertible. Matrix inversion is continuous at an invertible matrix, so $Q_n^{-1}\xrightarrow{p}Q^{-1}$ on the event that $Q_n$ is invertible. In particular, that event has probability tending to one.

The vectors $X_i u_i$ are iid, have mean zero, and have finite second moments because $E\|X_i u_i\|^2=E[\|X_i\|^2u_i^2]<\infty$. Their covariance is $\Omega$. The multivariate central limit theorem therefore gives \[ \sqrt n S_n=\frac{1}{\sqrt n}\sum_{i=1}^n X_i u_i\xrightarrow{d}N(0,\Omega). \] This is the score limit; the Gram matrix limit alone would not provide it.

On the event that $Q_n$ is invertible, the OLS normal equations yield an exact identity. Substituting $Y_i=X_i^\top\beta_0+u_i$ into the estimator gives

\hat\beta-\beta_0 =\left(\sum_{i=1}^n X_iX_i^\top\right)^{-1}\sum_{i=1}^n X_i u_i =Q_n^{-1}S_n.

Multiplying by $\sqrt n$, and applying Slutsky's theorem to $Q_n^{-1}\xrightarrow{p}Q^{-1}$ and the score central limit theorem, proves

\sqrt n(\hat\beta-\beta_0) =Q_n^{-1}\sqrt n S_n \xrightarrow{d}N(0,Q^{-1}\Omega Q^{-1}).

The covariance follows by transforming the score limit by $Q^{-1}$: for a mean-zero vector with covariance $\Omega$, multiplication by $Q^{-1}$ on the left gives covariance $Q^{-1}\Omega(Q^{-1})^\top$. Since $Q$ is symmetric, $(Q^{-1})^\top=Q^{-1}$. This completes the limit proof. $\square$

Establish the design limit.
The iid law of large numbers gives $Q_n\xrightarrow{p}Q$, and positive definiteness makes the limiting inverse well-defined.

Establish the score limit.
The mean-zero iid score vectors have finite covariance $\Omega$, so the multivariate central limit theorem applies.

Combine the limits.
The exact OLS expansion and Slutsky's theorem give the normal limit and its covariance.

Consistency of the Residual-Based Covariance Estimate

The theorem's covariance can be estimated without assuming homoskedasticity. Let $\hat u_i=Y_i-X_i^\top\hat\beta$, and define the empirical score covariance and the corresponding sandwich estimate by

\hat\Omega_n=\frac{1}{n}\sum_{i=1}^n X_iX_i^\top\hat u_i^2, \qquad \widehat{\operatorname{Var}}(\hat\beta) =\frac{1}{n}Q_n^{-1}\hat\Omega_nQ_n^{-1}.

First, the law of large numbers applies to $X_iX_i^\top u_i^2$, since its entries are integrable under $E[\|X_i\|^2u_i^2]<\infty$. Hence $n^{-1}\sum_iX_iX_i^\top u_i^2\xrightarrow{p}\Omega$. It remains to check that replacing $u_i$ by $\hat u_i$ does not change this limit.

Put $d_n=\hat\beta-\beta_0$. The OLS limit just proved implies $d_n\xrightarrow{p}0$, and $\hat u_i=u_i-X_i^\top d_n$. For any vector $x$, the matrix norm of $xx^\top$ is $\|x\|^2$, so

\left\|\frac1n\sum_{i=1}^nX_iX_i^\top(\hat u_i^2-u_i^2)\right\| \leq 2\|d_n\|\frac1n\sum_{i=1}^n\|X_i\|^3|u_i| +\|d_n\|^2\frac1n\sum_{i=1}^n\|X_i\|^4.

The bound follows from $|\hat u_i^2-u_i^2|\leq2|u_i||X_i^\top d_n|+|X_i^\top d_n|^2$ and $|X_i^\top d_n|\leq\|X_i\|\|d_n\|$. By Cauchy–Schwarz, $E[\|X_i\|^3|u_i|]\leq\{E\|X_i\|^4\,E[\|X_i\|^2u_i^2]\}^{1/2}<\infty$. The law of large numbers makes both empirical averages in the bound $O_p(1)$. Since $d_n=o_p(1)$, the bound is $o_p(1)$. Thus $\hat\Omega_n\xrightarrow{p}\Omega$. Combining this with $Q_n^{-1}\xrightarrow{p}Q^{-1}$ proves

n\,\widehat{\operatorname{Var}}(\hat\beta) =Q_n^{-1}\hat\Omega_nQ_n^{-1} \xrightarrow{p}Q^{-1}\Omega Q^{-1}.

This establishes consistency of the sandwich covariance estimate under the stated moments. The factor $1/n$ in the estimated covariance of $\hat\beta$ is essential: it is $n$ times that estimate which converges to the limiting covariance of $\sqrt n(\hat\beta-\beta_0)$. $\square$

Worked Example: A Two-Group Regression and Its Sandwich Covariance

Let $X_i=(1,Z_i)^\top$, where $P(Z_i=0)=P(Z_i=1)=1/2$, and let $Y_i=X_i^\top(1,2)^\top+u_i$. Suppose $u_i\mid Z_i=0\sim N(0,1)$ and $u_i\mid Z_i=1\sim N(0,4)$. The conditional error means are zero, so $E[X_i u_i]=0$. The covariate is bounded and the conditional errors have finite moments of every order.

The population Gram matrix is $Q=E[X_iX_i^\top]=\begin{pmatrix}1&1/2\\1/2&1/2\end{pmatrix}$. Its determinant is $1/2-1/4=1/4>0$, and $Q^{-1}=\begin{pmatrix}2&-2\\-2&4\end{pmatrix}$. The score covariance is $\Omega=E[X_iX_i^\top u_i^2]=\begin{pmatrix}5/2&2\\2&2\end{pmatrix}$: the $Z=0$ group contributes $\frac12\begin{pmatrix}1&0\\0&0\end{pmatrix}$, while the $Z=1$ group contributes $2\begin{pmatrix}1&1\\1&1\end{pmatrix}$.

Multiplication gives $Q^{-1}\Omega=\begin{pmatrix}1&0\\3&4\end{pmatrix}$, and therefore $Q^{-1}\Omega Q^{-1}=\begin{pmatrix}2&-2\\-2&10\end{pmatrix}$. The robust limiting variances of the intercept and slope are $2$ and $10$, with covariance $-2$. For comparison, the unconditional error variance is $E[u_i^2]=(1+4)/2=5/2$. A homoskedastic covariance formula would use $(5/2)Q^{-1}$, which is $\begin{pmatrix}5&-5\\-5&10\end{pmatrix}$, and gives a different intercept variance and covariance. Here the robust estimate targets the actual $\Omega$.

Common Mistake: Checking Only the Gram Matrix

A tempting but incomplete argument is: “The empirical Gram matrix converges to an invertible matrix, so OLS is asymptotically normal.” This establishes that the inverse design matrix behaves well, but says nothing by itself about the distribution of the score sum. The score must be centered and must meet a central limit condition; in this proof, $E[X_i u_i]=0$ and $E[\|X_i\|^2u_i^2]<\infty$ provide the needed conditions. If $E[X_i u_i]\ne0$, the displayed expansion contains a score with a nonzero population mean, and the estimator generally does not converge to $\beta_0$. A nonsingular Gram matrix cannot repair that failure.

A second important distinction is between unconditional and conditional statements. The proof uses the unconditional equation $E[X_i u_i]=0$, not the stronger condition $E[u_i\mid X_i]=0$. Conditional mean zero implies the unconditional equation when the expectations exist, but is not necessary for this OLS limit. By contrast, the sandwich covariance accounts for heteroskedasticity through $E[X_iX_i^\top u_i^2]$; replacing it by a scalar error variance times $Q$ requires additional homoskedasticity assumptions.

Key takeaway: A random-design OLS limit is built from three checks: a nonsingular population Gram matrix, a centered score with a valid central limit theorem, and the exact OLS expansion linking them. For residual-based sandwich inference, also verify the moments that justify replacing errors by residuals.

Check Your Understanding

Use the assumptions and proof structure above to answer these questions.

Which population quantity must be positive definite for the inverse empirical Gram matrix to converge to a stable limit?
Why does $E[X_i u_i]=0$ matter in applying the central limit theorem to the score?
In the exact OLS expansion, which empirical quantity converges to $Q^{-1}$, and which has limiting distribution $N(0,\Omega)$ after multiplication by $\sqrt n$?
Why can a sandwich covariance estimate remain appropriate when the error variance depends on the covariates?
What role does $E\|X_i\|^4<\infty$ play in the residual-replacement argument?