Asymptotic Normality of Random-Design OLS

From the OLS Representation to a Normal Limit

For random-design least squares, the regressors vary across sampled observations as well as the errors. The key asymptotic question is how that sampling variation enters the distribution of the coefficient estimator. The answer comes from combining two limits: the empirical Gram matrix settles near its population value, while the normalized sum of regression scores approaches a multivariate normal distribution.

The result below concerns a fixed number of regressors and an independent, identically distributed sample. It gives the first-order limiting distribution of OLS, not a finite-sample distribution. In particular, the covariance in that limit is built from the variability of the score $X_i u_i$ and the population Gram matrix.

Setup and assumptions: For $i=1,\ldots,n$, observe $(Y_i,X_i)$, where $X_i$ is a $p$-dimensional column vector and $Y_i=X_i'\beta_0+u_i$, with fixed $p$ and coefficient vector $\beta_0$. The pairs $(Y_i,X_i)$ are iid. Assume $E[X_i u_i]=0$, $E[\|X_i\|^2]<\infty$, and $E[u_i^2\|X_i\|^2]<\infty$. Define $Q=E[X_iX_i']$ and $\Omega=E[u_i^2X_iX_i']$, and assume $Q$ is positive definite. These assumptions make $Q$ and $\Omega$ finite, and ensure the score $X_i u_i$ has mean zero and finite covariance $\Omega$.

The condition $E[X_i u_i]=0$ is the population moment condition that centers the OLS score. It does not require that the error be independent of the regressors. The positive definiteness of $Q$ is the population identification condition: no nonzero coefficient direction has zero second moment in the regressors.

The OLS Linear Representation

Write the empirical Gram matrix as $Q_n=n^{-1}\sum_{i=1}^n X_iX_i'$. On the event that $Q_n$ is invertible, the OLS estimator is $\hat{\beta}=(\sum_iX_iX_i')^{-1}\sum_iX_iY_i$. Substituting the model into this expression gives an exact identity:

\sqrt{n}(\hat{\beta}-\beta_0) = Q_n^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n X_i u_i.

This identity separates the two components of the limit. The inverse empirical Gram matrix determines how score variation is translated into coefficient variation. The normalized score sum supplies the random fluctuation. By the law of large numbers, $Q_n\xrightarrow{p}Q$; since $Q$ is positive definite, $Q_n^{-1}\xrightarrow{p}Q^{-1}$ on events whose probability tends to one.

Theorem (Asymptotic normality of random-design OLS): Under the setup and assumptions above, $$ \sqrt{n}(\hat{\beta}-\beta_0)\xrightarrow{d}N\!\left(0,Q^{-1}\Omega Q^{-1}\right). $$ The limiting covariance matrix $Q^{-1}\Omega Q^{-1}$ is allowed to be singular; in that case the multivariate normal limit is degenerate in one or more directions.

Deriving the Limiting Distribution

The derivation uses the iid multivariate central limit theorem for the scores, followed by convergence of the Gram matrix and Slutsky's theorem. The finite score second moment in the assumptions is what allows the central limit theorem to be applied.

Apply the score central limit theorem.
Let $S_i=X_i u_i$. Then $E[S_i]=0$, and $E[S_iS_i']=\Omega$. The iid multivariate central limit theorem gives $n^{-1/2}\sum_iS_i\xrightarrow{d}N(0,\Omega)$.

Take the Gram-matrix limit.
Each entry of $Q_n$ is an average of iid variables with finite expectation. The law of large numbers gives $Q_n\xrightarrow{p}Q$. Continuity of matrix inversion at the nonsingular matrix $Q$ gives $Q_n^{-1}\xrightarrow{p}Q^{-1}$.

Combine the limits.
The exact OLS identity and Slutsky's theorem yield $Q_n^{-1}n^{-1/2}\sum_iS_i\xrightarrow{d}Q^{-1}Z$, where $Z\sim N(0,\Omega)$. A linear transformation of a multivariate normal vector has covariance $Q^{-1}\Omega(Q^{-1})'$. Since $Q$ is symmetric, this is $Q^{-1}\Omega Q^{-1}$.

For completeness, the finite-sample invertibility issue does not change the conclusion. Since $Q_n\xrightarrow{p}Q$ and $Q$ is positive definite, the probability that $Q_n$ is positive definite tends to one. To see this, let $\lambda_{\min}(Q)>0$ be the smallest eigenvalue of $Q$. With probability tending to one, the operator norm $\|Q_n-Q\|$ is less than $\lambda_{\min}(Q)/2$. For any unit vector $v$, on that event, $v'Q_nv\geq v'Qv-\|Q_n-Q\|>\lambda_{\min}(Q)/2>0$. Thus $Q_n$ is positive definite there, so OLS is uniquely defined with probability tending to one. Defining the estimator arbitrarily on the remaining event does not alter its limiting distribution, because that event has probability tending to zero. $\square$

Worked Examples

Worked Example: The Intercept-Only Model

Suppose $X_i=1$, so $Y_i=\beta_0+u_i$, with iid errors satisfying $E[u_i]=0$ and $\operatorname{Var}(u_i)=4$. Here $Q=E[1]=1$, and $\Omega=E[u_i^2]=4$. The limiting variance is therefore $Q^{-1}\Omega Q^{-1}=1\cdot4\cdot1=4$.

In this case $\hat{\beta}=\bar{Y}$, and the theorem gives $\sqrt{n}(\bar{Y}-\beta_0)\xrightarrow{d}N(0,4)$. Equivalently, for large $n$, the estimator $\bar{Y}$ has an approximate variance of $4/n$. This agrees with the familiar variance of a sample mean with error variance 4.

Worked Example: A Two-Regressor Heteroskedastic Design

Let $X_i=(1,Z_i)'$, where $P(Z_i=0)=P(Z_i=2)=1/2$. Suppose $E[u_i\mid Z_i]=0$, with $E[u_i^2\mid Z_i=0]=1$ and $E[u_i^2\mid Z_i=2]=9$. These moments imply $E[X_i u_i]=0$. First calculate the Gram matrix:

Q=\frac12 \begin{pmatrix}1&0\\0&0\end{pmatrix} +\frac12 \begin{pmatrix}1&2\\2&4\end{pmatrix} = \begin{pmatrix}1&1\\1&2\end{pmatrix}, \qquad Q^{-1}= \begin{pmatrix}2&-1\\-1&1\end{pmatrix}.

The determinant of $Q$ is $1\cdot2-1\cdot1=1>0$, so $Q$ is positive definite. Next, weight each regressor outer product by the conditional error second moment at that value of $Z_i$:

\Omega =\frac12(1) \begin{pmatrix}1&0\\0&0\end{pmatrix} +\frac12(9) \begin{pmatrix}1&2\\2&4\end{pmatrix} = \begin{pmatrix}5&9\\9&18\end{pmatrix}.

For example, the off-diagonal entry is $\frac12(1)(0)+\frac12(9)(2)=9$, and the lower-right entry is $\frac12(1)(0^2)+\frac12(9)(2^2)=18$. Multiplying out the sandwich gives

Q^{-1}\Omega = \begin{pmatrix}2&-1\\-1&1\end{pmatrix} \begin{pmatrix}5&9\\9&18\end{pmatrix} = \begin{pmatrix}1&0\\4&9\end{pmatrix}, \qquad Q^{-1}\Omega Q^{-1} = \begin{pmatrix}2&-1\\-1&5\end{pmatrix}.

Thus the intercept and slope estimates, jointly and after multiplication by $\sqrt{n}$, have limiting covariance matrix $\begin{pmatrix}2&-1\\-1&5\end{pmatrix}$. The off-diagonal entry is negative; covariance matrices need not have positive off-diagonal entries. The matrix is positive definite: its leading entry is 2 and its determinant is $2(5)-(-1)^2=9>0$.

Worked Example: A Linear Contrast

Suppose a fitted model has $Q=I_2$ and score covariance $\Omega=\begin{pmatrix}4&1\\1&3\end{pmatrix}$. Then $Q^{-1}=I_2$, so the limiting covariance of $\sqrt n(\hat{\beta}-\beta_0)$ is $\Omega$. Consider the contrast $c'\hat{\beta}$, where $c=(1,-1)'$. Its centered, scaled version is $c'\sqrt n(\hat{\beta}-\beta_0)$. The limiting variance is

c'\Omega c = \begin{pmatrix}1&-1\end{pmatrix} \begin{pmatrix}4&1\\1&3\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} = \begin{pmatrix}1&-1\end{pmatrix} \begin{pmatrix}3\\-2\end{pmatrix} =5.

Consequently, $ \sqrt n\{c'\hat{\beta}-c'\beta_0\}\xrightarrow{d}N(0,5)$. The variance of the unscaled contrast estimator is approximately $5/n$. This calculation illustrates how the coefficient limit immediately gives a limit for any fixed linear combination of coefficients.

What the Covariance Means

The two appearances of $Q^{-1}$ have a useful interpretation. The score $X_i u_i$ fluctuates across observations, with covariance $\Omega$. The inverse Gram matrix converts those score fluctuations into changes in the coefficient vector. Hence the covariance is a sandwich: the score covariance is in the middle, and the inverse design information appears on both sides.

This is an asymptotic covariance for the scaled estimator. In particular, it says that the unscaled estimator has fluctuations of order $n^{-1/2}$, and its approximate covariance is $n^{-1}Q^{-1}\Omega Q^{-1}$. The result does not, by itself, provide a finite-sample exact normal distribution. Nor does it prescribe an estimator of the unknown covariance; that is a separate question.

Common Mistake

A common mistake is to report $Q^{-1}$ as the limiting covariance merely because the estimator is OLS. That omits the score's variability. The correct covariance is $Q^{-1}\Omega Q^{-1}$; it simplifies to $Q^{-1}$ only under additional conditions that make $\Omega=Q$. A different mistake is to replace $\Omega$ by a conditional error variance without averaging over the random regressors. By definition, $\Omega=E[u_i^2X_iX_i']$, so the regressor outer product and squared error must be considered together before taking the expectation.

Key takeaway: Under iid sampling, a finite score second moment, the population moment condition, and a positive definite population Gram matrix, OLS is asymptotically normal. Its limiting covariance is $Q^{-1}\Omega Q^{-1}$, obtained by combining the score central limit theorem with Gram-matrix convergence.

Check Your Understanding

Check that you can connect the assumptions, the OLS representation, and the covariance formula.

Which condition centers the regression score $X_i u_i$, and which condition ensures the population Gram matrix can be inverted?
In the limiting covariance $Q^{-1}\Omega Q^{-1}$, what does $\Omega$ measure?
If $Q=I_2$ and $\Omega=\begin{pmatrix}2&0\\0&8\end{pmatrix}$, what is the limiting variance of the contrast with $c=(1,1)'$?
Why does convergence of $Q_n$ to a positive definite $Q$ imply that the OLS estimator is uniquely defined with probability tending to one?
Does the theorem require the errors to be independent of the regressors? State the relevant moment condition instead.