Asymptotic Normality of OLS Under a Linear Mean

Why OLS Has an Asymptotic Distribution

Consistency says that an OLS estimate approaches the population coefficient as the sample grows. Asymptotic normality adds a description of the estimate’s remaining sampling variation: after multiplying its error by \(\sqrt{n}\), the error approaches a normal distribution. This result explains why the regressor distribution and the disturbances both matter. Regressors determine how much information the sample provides about each coefficient; disturbances determine how much random variation enters the estimate.

The result here concerns independent sampling and a correctly specified linear conditional mean. It does not require normally distributed disturbances. Its limiting covariance also allows the disturbance variance to depend on the regressors.

Model and assumptions: For observation \(i\), let \(Y_i\) be the outcome and let \(X_i\) be a \(k\)-dimensional column vector of regressors, including an intercept if the model has one. The observations \((Y_i,X_i)\) are independent and identically distributed, and \(Y_i=X_i'\beta_0+u_i\), where \(E[u_i\mid X_i]=0\). Assume \(E[\|X_i\|^2]<\infty\), \(E[\|X_i u_i\|^2]<\infty\), and \(Q=E[X_iX_i']\) is positive definite. These conditions give a finite second moment for the OLS score \(X_i u_i\) and a nonsingular population regressor moment matrix.

The conditional-mean assumption means \(E[Y_i\mid X_i]=X_i'\beta_0\). The disturbance \(u_i=Y_i-X_i'\beta_0\) therefore has conditional mean zero, but it need not be independent of \(X_i\), normally distributed, or have constant conditional variance. The moment conditions are conditions on the population from which the observations are drawn; they are not claims that every finite sample has full rank.

The Exact Estimation-Error Representation

Write the OLS estimate as \(\hat{\beta}=(\sum_{i=1}^n X_iX_i')^{-1}\sum_{i=1}^nX_iY_i\) whenever the sample matrix is invertible. Substituting the model equation separates the population coefficient from the sample disturbance contribution. The resulting identity is the starting point for the limiting distribution.

\begin{aligned} \hat{\beta} &=\left(\sum_{i=1}^n X_iX_i'\right)^{-1} \sum_{i=1}^n X_i(X_i'\beta_0+u_i)\\ &=\beta_0+\left(\sum_{i=1}^n X_iX_i'\right)^{-1} \sum_{i=1}^n X_i u_i,\\ \sqrt{n}(\hat{\beta}-\beta_0) &=\left(\frac{1}{n}\sum_{i=1}^n X_iX_i'\right)^{-1} \left(\frac{1}{\sqrt{n}}\sum_{i=1}^n X_i u_i\right). \end{aligned}

This is an algebraic identity, not an approximation. The first factor in the last line describes the sample’s regressor geometry. The second is a normalized sum of score contributions \(X_i u_i\). The conditional-mean restriction centers each contribution: \(E[X_i u_i]=E[X_iE[u_i\mid X_i]]=0\).

Worked Example: Checking the Exact Identity

Consider a hypothetical sample with three observations, a regressor \(x\), and an intercept. Suppose the true coefficient is \(\beta_0=(10,2)'\), and the disturbances are \(1,-1,2\) at \(x=0,1,2\), respectively. The corresponding outcomes are \(11,11,16\). These values illustrate the identity; three observations are not being used to claim a normal approximation.

With \(X_i=(1,x_i)'\), the sample cross-product matrix and score sum are \(\sum_iX_iX_i'=\begin{pmatrix}3&3\\3&5\end{pmatrix}\) and \(\sum_iX_i u_i=(1+(-1)+2,\;0(1)+1(-1)+2(2))'=(2,3)'\).

The inverse cross-product matrix is \(\frac{1}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\). Therefore, \(\hat{\beta}-\beta_0 =\frac{1}{6}\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\begin{pmatrix}2\\3\end{pmatrix} =(1/6,1/2)'\). Thus the fitted intercept is \(10+1/6=10.167\), and the fitted slope is \(2+1/2=2.500\). Directly, these values fit the three outcomes by ordinary least squares. The example shows how the sample score and regressor matrix combine; the limiting normal result requires the sampling assumptions stated above.

Deriving the Limiting Normal Distribution

Define the population score covariance matrix \(\Omega=E[X_iX_i'u_i^2]\). The conditional-mean assumption makes the score mean zero, and the finite second-moment condition makes \(\Omega\) finite. The central limit theorem applied to the independent, identically distributed score vectors gives a normal limit for their normalized sum. The law of large numbers gives convergence of the sample regressor matrix to \(Q\).

Theorem: Under the model and assumptions above, \(\sqrt{n}(\hat{\beta}-\beta_0)\) converges in distribution to a mean-zero multivariate normal vector with covariance matrix \(Q^{-1}\Omega Q^{-1}\). Equivalently, for large \(n\), the sampling distribution of \(\hat{\beta}\) is approximated by a normal distribution centered at \(\beta_0\), with covariance approximately \(Q^{-1}\Omega Q^{-1}/n\).

Apply the central limit theorem.
The vectors \(X_i u_i\) are iid, have mean zero, and have finite second moments. Hence \(n^{-1/2}\sum_iX_i u_i\) converges in distribution to a mean-zero normal vector with covariance \(\Omega\).

Apply the law of large numbers.
Finite \(E[\|X_i\|^2]\) implies that \(n^{-1}\sum_iX_iX_i'\) converges in probability to \(Q\). Since \(Q\) is positive definite, its inverse exists, and the sample matrix is invertible with probability approaching one.

Combine the limits.
By the continuous mapping theorem, the inverse sample matrix converges in probability to \(Q^{-1}\). Slutsky’s theorem applied to the exact representation gives the limiting normal vector \(Q^{-1}Z\), where \(Z\) has covariance \(\Omega\).

Compute its covariance.
A linear transformation \(Q^{-1}Z\) has covariance \(Q^{-1}\Omega(Q^{-1})'\). Because \(Q\) is symmetric, \(Q^{-1}\) is symmetric too, so this covariance is \(Q^{-1}\Omega Q^{-1}\).

The probability limit of the sample matrix is nonsingular, so the representation applies with probability approaching one. The theorem describes a limit as \(n\) grows; it does not assert that the finite-sample OLS estimate is exactly normal. Also, if \(\Omega\) is singular, the limiting normal distribution can be degenerate in some directions.

What Determines the Limiting Covariance?

The matrix \(Q=E[X_iX_i']\) summarizes the population distribution of the regressors. If the regressors provide little variation for distinguishing coefficients, the inverse \(Q^{-1}\) reflects that limited information. The matrix \(\Omega\) summarizes how disturbances vary in the directions weighted by the regressors. In particular, by conditioning on \(X_i\), \(\Omega=E[X_iX_i'\operatorname{Var}(u_i\mid X_i)]\), since \(E[u_i\mid X_i]=0\).

If the conditional disturbance variance is constant, \(\operatorname{Var}(u_i\mid X_i)=\sigma^2\), then \(\Omega=\sigma^2Q\) and the limiting covariance simplifies to \(\sigma^2Q^{-1}\). The linear conditional mean alone does not imply this simplification. With varying conditional variance, the general expression \(Q^{-1}\Omega Q^{-1}\) applies.

Worked Example: Regressors and Heterogeneous Disturbance Variance

Suppose a hypothetical population has a regressor \(Z\) equal to \(0\) or \(2\), each with probability \(1/2\), and \(X=(1,Z)'\). Let \(Y=\beta_{0,1}+\beta_{0,2}Z+u\). Conditional on \(Z=0\), let \(u\) be equally likely to be \(-1\) or \(1\); conditional on \(Z=2\), let it be equally likely to be \(-3\) or \(3\). Thus \(E[u\mid Z]=0\), while the conditional variances are \(1\) and \(9\). The regressors are bounded, the disturbances have finite second moments, and both regressor values occur with positive probability, so \(Q\) is positive definite.

The population regressor matrix is \(Q=\frac12\begin{pmatrix}1&0\\0&0\end{pmatrix} +\frac12\begin{pmatrix}1&2\\2&4\end{pmatrix} =\begin{pmatrix}1&1\\1&2\end{pmatrix}\), with \(Q^{-1}=\begin{pmatrix}2&-1\\-1&1\end{pmatrix}\).

Weighting each regressor outer product by its conditional disturbance variance gives \(\Omega=\frac12(1)\begin{pmatrix}1&0\\0&0\end{pmatrix} +\frac12(9)\begin{pmatrix}1&2\\2&4\end{pmatrix} =\begin{pmatrix}5&9\\9&18\end{pmatrix}\). Multiplying, \(Q^{-1}\Omega=\begin{pmatrix}1&0\\4&9\end{pmatrix}\), and hence \(Q^{-1}\Omega Q^{-1}=\begin{pmatrix}2&-1\\-1&5\end{pmatrix}\).

Thus the large-sample covariance of \(\hat{\beta}\) is approximately \(\frac{1}{n}\begin{pmatrix}2&-1\\-1&5\end{pmatrix}\). At \(n=100\), the corresponding asymptotic standard deviations are \(\sqrt{2/100}=0.1414\) for the intercept and \(\sqrt{5/100}=0.2236\) for the slope. The slope is measured in outcome units per unit of \(Z\). The off-diagonal entry gives an asymptotic covariance of \(-1/100\) between these coefficient estimates. These are large-sample approximations, not exact finite-sample quantities.

Worked Example: The Homoskedastic Special Case

Suppose instead that a hypothetical population has \(X=(1,Z)'\), where \(Z\) equals \(-1\) or \(1\) with equal probability, and \(u\) equals \(-2\) or \(2\) with equal probability independently of \(Z\). Set \(Y=X'\beta_0+u\). The disturbance has mean zero and variance \(4\); independence gives \(E[u\mid X]=0\) and \(\operatorname{Var}(u\mid X)=4\). All variables are bounded, and the two regressor values make \(Q\) positive definite.

Here \(Q=E[XX']=\begin{pmatrix}1&E[Z]\\E[Z]&E[Z^2]\end{pmatrix}=I_2\), because \(E[Z]=0\) and \(E[Z^2]=1\). Since the conditional variance is \(4\), \(\Omega=4Q=4I_2\), and the limiting covariance is \(Q^{-1}\Omega Q^{-1}=4I_2\).

For \(n=400\), the asymptotic covariance of \(\hat{\beta}\) is approximately \(4I_2/400=0.01I_2\). Each coefficient therefore has asymptotic standard deviation \(0.1\) in its respective coefficient units. The zero off-diagonal covariance follows here from the symmetric regressor distribution and the stated disturbance assumptions.

Interpretation and a Common Pitfall

The theorem is useful because it gives a principled large-sample description of OLS uncertainty without requiring normally distributed outcomes or disturbances. Its interpretation must retain the assumptions: the conditional mean is linear in the included regressors, the observations are independent and identically distributed, and the specified moments exist. The covariance depends on the regressor population and on how disturbance variation relates to those regressors.

Common mistake: “The conditional mean is linear, so the limiting covariance must be \(\sigma^2Q^{-1}\).” This conclusion does not follow. A linear conditional mean ensures \(E[u\mid X]=0\), not constant conditional variance. If \(\operatorname{Var}(u\mid X)\) varies, the covariance is generally \(Q^{-1}\Omega Q^{-1}\), as the preceding example demonstrates. The simpler formula requires the additional condition \(\operatorname{Var}(u\mid X)=\sigma^2\).

A related mistake is to treat asymptotic normality as an exact statement for any sample size. The theorem concerns the behavior as \(n\) increases. How accurate a normal approximation is at a particular sample size depends on the data-generating distribution and the quality of the approximation; the theorem itself does not guarantee accuracy for a small sample.

Key takeaway: Under independent sampling, a correct linear conditional mean, finite second moments, and a nonsingular population regressor matrix, the OLS error is asymptotically normal. Regressors enter through \(Q\); disturbances enter through \(\Omega\). The general limiting covariance is \(Q^{-1}\Omega Q^{-1}\), and the familiar homoskedastic simplification requires an additional variance assumption.

Check Your Understanding

Use the assumptions and results in this tutorial to answer the following questions.

Why does \(E[u_i\mid X_i]=0\) imply that the score \(X_i u_i\) has mean zero?
In the estimation-error representation, which term contains the population regressor information, and which term is the normalized disturbance contribution?
Which condition permits \(Q^{-1}\Omega Q^{-1}\) to simplify to \(\sigma^2Q^{-1}\)?
Does asymptotic normality require normally distributed disturbances? Explain what condition instead allows the central limit theorem to be used here.
Why does a positive definite \(Q\) matter for the limiting result?