The Asymptotic Linear Representation of OLS

From the OLS Formula to a Limit-Theory Representation

The ordinary least-squares estimator is defined by a finite-sample optimization problem. For asymptotic analysis, a more revealing form separates two ingredients: the empirical Gram matrix of the regressors and the average of regressor-error products. The first describes the geometry of the observed design; the second is the sample score whose fluctuations drive the estimator.

The separation begins with an exact algebraic identity. Under appropriate moment and identification conditions, it then yields an asymptotic linear representation: the centered, scaled estimator equals a fixed matrix times a normalized sum, up to a term that converges to zero in probability. This tutorial establishes that representation but does not derive the limiting distribution of the sum.

Definition: Let \(X_i\in\mathbb{R}^p\) be a column vector of regressors, \(Y_i\) a scalar response, and \(\beta_0\in\mathbb{R}^p\) a coefficient vector. Write \(Y_i=X_i'\beta_0+u_i\), and define \(Q_n=n^{-1}\sum_{i=1}^n X_iX_i'\) and \(S_n=n^{-1}\sum_{i=1}^n X_i u_i\). When \(Q_n\) is nonsingular, the OLS estimator is \(\hat{\beta}=(\sum_{i=1}^n X_iX_i')^{-1}\sum_{i=1}^n X_iY_i\). The vector \(S_n\) is the empirical regressor-error score.

The moment condition \(E[X_i u_i]=0\) centers the score at zero. It is the relevant population orthogonality condition here; it need not be interpreted as a claim that every regressor is independent of the error. The population Gram matrix is \(Q=E[X_iX_i']\). A positive definite \(Q\) ensures that no nonzero coefficient direction has zero population regressor variation.

The Exact Finite-Sample Identity

Substitute the regression equation into the OLS formula. The resulting identity is purely algebraic: it does not use independence, a law of large numbers, or a central limit theorem.

Theorem: If \(Q_n\) is nonsingular and \(Y_i=X_i'\beta_0+u_i\) for every observation, then \(\hat{\beta}-\beta_0=Q_n^{-1}S_n\). Equivalently, \(\sqrt{n}(\hat{\beta}-\beta_0)=Q_n^{-1}n^{-1/2}\sum_{i=1}^n X_i u_i\).

Proof. By the definition of \(\hat{\beta}\) and the model,

\begin{aligned} \hat{\beta} &=\left(\sum_{i=1}^n X_iX_i'\right)^{-1} \sum_{i=1}^n X_i(X_i'\beta_0+u_i)\\ &=\left(\sum_{i=1}^n X_iX_i'\right)^{-1} \left\{\left(\sum_{i=1}^n X_iX_i'\right)\beta_0 +\sum_{i=1}^n X_i u_i\right\}\\ &=\beta_0+ \left(\sum_{i=1}^n X_iX_i'\right)^{-1} \sum_{i=1}^n X_i u_i. \end{aligned}

Since \(\sum_{i=1}^n X_iX_i'=nQ_n\), the last term is \(Q_n^{-1}S_n\). Multiplying by \(\sqrt{n}\) gives the second identity. \(\square\)

Worked Example: Checking the Identity in a Small Sample

Consider a regression through the origin with scalar regressor values \(X_1=1\), \(X_2=2\), and \(X_3=1\). Suppose the responses follow \(Y_i=\beta_0X_i+u_i\), with errors \(u_1=1\), \(u_2=-1\), and \(u_3=2\). Calculate the OLS difference from \(\beta_0\) in two ways.

First, the empirical Gram matrix (a scalar in this example) is \(Q_n=(1^2+2^2+1^2)/3=6/3=2\). The average score is \(S_n=(1\cdot1+2\cdot(-1)+1\cdot2)/3=(1-2+2)/3=1/3\). Thus the identity gives \(\hat{\beta}-\beta_0=Q_n^{-1}S_n=(1/2)(1/3)=1/6\).

Directly from the through-origin OLS formula, \(\hat{\beta}=\sum_i X_iY_i/\sum_i X_i^2\). Substituting \(Y_i=\beta_0X_i+u_i\) gives \[ \hat{\beta}-\beta_0 =\frac{\sum_i X_i u_i}{\sum_i X_i^2} =\frac{1-2+2}{6} =\frac{1}{6}. \] The two calculations agree. This example illustrates the exact algebra, not an asymptotic approximation.

The Asymptotic Linear Representation

Now suppose the observations \((X_i,u_i)\) are identically distributed and independent. Assume \(E[X_i u_i]=0\), \(E[\|X_i\|^2]<\infty\), and \(E[\|X_i u_i\|^2]<\infty\). Let \(Q=E[X_iX_i']\) be positive definite. These conditions ensure that the population Gram matrix is finite and nonsingular and that the score has a finite second moment.

By the law of large numbers, \(Q_n\xrightarrow{p}Q\). Consequently, \(Q_n\) is nonsingular with probability tending to one, and \(Q_n^{-1}\xrightarrow{p}Q^{-1}\). If the inverse does not exist for a particular finite sample, the statements below are understood on the event that it exists; that event has probability tending to one.

Theorem: Under the assumptions above, \[ \sqrt{n}(\hat{\beta}-\beta_0) =Q^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n X_i u_i+o_p(1). \] This is the asymptotic linear representation of OLS. The displayed relation does not by itself assert a limiting distribution for the normalized score.

Proof. First show that the normalized score is bounded in probability. Let \(Z_i=X_i u_i\). The vectors \(Z_i\) are iid, \(E[Z_i]=0\), and \(E[\|Z_i\|^2]<\infty\). Independence and the zero mean make the cross terms vanish:

\begin{aligned} E\left[\left\|\frac{1}{\sqrt{n}}\sum_{i=1}^n Z_i\right\|^2\right] &=\frac{1}{n}\sum_{i=1}^n E[\|Z_i\|^2] +\frac{1}{n}\sum_{i\ne j}E[Z_i'Z_j]\\ &=E[\|Z_1\|^2]. \end{aligned}

Indeed, for \(i\ne j\), independence gives \(E[Z_i'Z_j]=E[Z_i]'E[Z_j]=0\). Markov's inequality therefore implies that \(n^{-1/2}\sum_i Z_i=O_p(1)\).

On the event that \(Q_n\) is nonsingular, the exact identity gives

\begin{aligned} \sqrt{n}(\hat{\beta}-\beta_0) &=Q_n^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n Z_i\\ &=Q^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n Z_i +(Q_n^{-1}-Q^{-1})\frac{1}{\sqrt{n}}\sum_{i=1}^n Z_i. \end{aligned}

Continuity of matrix inversion at the positive definite \(Q\) gives \(Q_n^{-1}-Q^{-1}=o_p(1)\). The second factor in the final term is \(O_p(1)\), so their product is \(o_p(1)\). The probability of the event on which the identity applies tends to one, completing the representation. \(\square\)

The word linear refers to the leading term being a fixed linear transformation of a sum of individual score vectors. It does not mean that OLS is a linear function of the observed responses for every realized random design with a fixed coefficient matrix. In the representation, the random empirical inverse has been replaced by its population limit, and the error from doing so is asymptotically negligible at the \(\sqrt{n}\) scale.

Worked Example: A Random Regressor with an Exogenous Error

Let the scalar regressor \(X_i\) take values \(-1\) and \(1\), each with probability \(1/2\). Let \(u_i\) be independent of \(X_i\), with \(E[u_i]=0\) and \(E[u_i^2]=\sigma^2<\infty\), and set \(Y_i=\beta_0X_i+u_i\). Then \(Q=E[X_i^2]=1\), while \(E[X_i u_i]=E[X_i]E[u_i]=0\). Also \(E[(X_i u_i)^2]=E[X_i^2]E[u_i^2]=\sigma^2<\infty\).

The representation therefore becomes \[ \sqrt{n}(\hat{\beta}-\beta_0) =\frac{1}{\sqrt{n}}\sum_{i=1}^n X_i u_i+o_p(1). \] Here the population Gram matrix is the scalar 1, so its inverse does not change the leading score. The formula identifies the quantity whose distribution must be studied to obtain a more specific limit result; it does not supply that distribution on its own.

Why the Centering Condition Matters

The zero-mean score condition is not just a technical device for the proof. If \(E[X_i u_i]\ne0\), the exact identity still holds whenever OLS exists, but the average score generally converges to a nonzero vector. In that case, centering the estimator at \(\beta_0\) need not leave a difference that shrinks to zero.

Worked Example: Correlation Between the Regressor and Error

Let \(X_i\) take values \(-1\) and \(1\) with equal probability, let \(\varepsilon_i\) be iid with mean zero and finite variance, independent of \(X_i\), and set \(u_i=\rho X_i+\varepsilon_i\), where \(\rho\ne0\). Consider the regression through the origin \(Y_i=\beta_0X_i+u_i\).

Since \(X_i^2=1\), \(Q=E[X_i^2]=1\). But \[ E[X_i u_i] =E[\rho X_i^2+X_i\varepsilon_i] =\rho+E[X_i]E[\varepsilon_i] =\rho. \] The exact identity is \[ \hat{\beta}-\beta_0 =\frac{n^{-1}\sum_i X_i u_i}{n^{-1}\sum_i X_i^2} =\frac{n^{-1}\sum_i(\rho+X_i\varepsilon_i)}{1} =\rho+\frac{1}{n}\sum_i X_i\varepsilon_i. \] The law of large numbers gives \(n^{-1}\sum_iX_i\varepsilon_i\xrightarrow{p}0\), so \(\hat{\beta}-\beta_0\xrightarrow{p}\rho\), not zero. The exact algebra has not failed; rather, the score is not centered at zero. The representation centered at \(\beta_0\) with a zero-mean leading score is unavailable under these assumptions.

OLS as a Population Least-Squares Projection

The coefficient in the representation need not describe a correctly specified conditional mean. Suppose instead that \(Q=E[XX']\) is positive definite and define the population least-squares coefficient by \(\beta^*=Q^{-1}E[XY]\), assuming these expectations exist. Set \(u=Y-X'\beta^*\). Then \(E[Xu]=E[XY]-E[XX']\beta^*=0\). Thus the same algebra and asymptotic representation can apply to deviations from the population projection, provided the required second moments hold. This distinction separates the algebraic target of least squares from a stronger claim that the conditional mean is exactly linear.

Worked Example: Projection of a Curved Conditional Mean

Let \(Z\) be uniform on \([-1,1]\), take \(X=(1,Z)'\), and let \(Y=Z^2\). The response is a curved function of \(Z\), but we can calculate its population linear least-squares projection.

Symmetry gives \(E[Z]=0\), \(E[Z^2]=1/3\), and \(E[Z^3]=0\). Hence \[ Q=E[XX'] =\begin{pmatrix}1&0\\0&1/3\end{pmatrix}, \qquad E[XY] =\begin{pmatrix}E[Z^2]\\E[Z^3]\end{pmatrix} =\begin{pmatrix}1/3\\0\end{pmatrix}. \] Therefore \(Q^{-1}=\begin{pmatrix}1&0\\0&3\end{pmatrix}\), and \(\beta^*=Q^{-1}E[XY]=(1/3,0)'\).

The projection residual is \(u=Z^2-1/3\). Its score has zero expectation: \[ E[Xu] =\begin{pmatrix}E[Z^2-1/3]\\E[Z^3-Z/3]\end{pmatrix} =\begin{pmatrix}1/3-1/3\\0-0\end{pmatrix} =\begin{pmatrix}0\\0\end{pmatrix}. \] All variables here are bounded, so the required second moments are finite. The representation is consequently \[ \sqrt{n}(\hat{\beta}-\beta^*) =Q^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n \begin{pmatrix}1\\Z_i\end{pmatrix}(Z_i^2-1/3)+o_p(1). \] This describes OLS fluctuations around its population linear projection even though \(Y\) is not a linear function of \(Z\).

Common Pitfalls and the Role of the Representation

The representation matters because it isolates the part of OLS that is specific to the sample from the part determined by the population design. Once a suitable limit theorem is available for the normalized score, the representation transfers that limit through \(Q^{-1}\). The conditions for such a score limit require their own analysis; finite second moments and the algebra above should not be mistaken for a proof of a particular limiting law.

Dropping the empirical Gram matrix too early. The exact identity contains \(Q_n^{-1}\), not \(Q^{-1}\). Replacing it is justified asymptotically only after establishing \(Q_n^{-1}\xrightarrow{p}Q^{-1}\).
Claiming a limit distribution from the representation alone. The representation reduces the task to the score, but a distributional limit for that score requires a separate result and its assumptions.
Ignoring the score's mean. If \(E[X_i u_i]\ne0\), the estimator may not converge to the coefficient used for centering. The endogeneity example shows the resulting nonzero drift.
Confusing the population projection with a correctly specified conditional mean. The projection is defined by a population least-squares criterion and its orthogonality condition; it can exist even when the conditional mean is nonlinear.

Key takeaway: The exact identity is \(\hat{\beta}-\beta_0=Q_n^{-1}S_n\). With a nonsingular population Gram matrix, a zero-mean score, and a finite score second moment, it yields \(\sqrt{n}(\hat{\beta}-\beta_0)=Q^{-1}n^{-1/2}\sum_iX_i u_i+o_p(1)\).

Check Your Understanding

Use the identity and assumptions in this tutorial to answer the following questions.

Which finite-sample term in the exact OLS identity is replaced by its population limit in the asymptotic linear representation?
Why does \(E[X_i u_i]=0\) help establish that the normalized score is bounded in probability?
In the correlated-error example, what is the probability limit of \(\hat{\beta}-\beta_0\), and which condition fails?
Can a population linear projection have a zero-mean score when the conditional mean is nonlinear? Explain using the definition of \(\beta^*\).
Does the asymptotic linear representation alone establish a normal limiting distribution? Why or why not?