Consistency of OLS From Sample Moments

Why Consistency Is a Moment Question

An OLS estimate is calculated from a sample, so it changes when the sample changes. Consistency asks whether this estimate approaches a fixed population quantity as the sample size grows. For OLS, the key is to express the estimate in terms of sample moments: if those moments approach their population counterparts and the population regressor second-moment matrix is nonsingular, the OLS coefficients approach the corresponding population projection coefficients.

This conclusion has an important limit. OLS can be consistent for the best linear projection even when the conditional mean is not linear. Calling the limit a structural parameter requires a further link between the regression equation, its regressors, and the structural parameter of interest. The calculation of OLS alone does not supply that link.

Population and Sample Moment Ratios

Let $Y_i$ be a scalar outcome and $X_i$ a $k$-dimensional column vector of regressors for observation $i$. An intercept, if included, is one component of $X_i$. Assume initially that $(Y_i,X_i)$, for $i=1,\ldots,n$, are independent and identically distributed copies of $(Y,X)$. Define the population regressor second-moment matrix and the population regressor-outcome cross-moment by

Q=E[X_iX_i'], \qquad q=E[X_iY_i].

The corresponding population linear projection coefficient is the coefficient vector that minimizes the expected squared prediction error over linear functions of $X_i$. To ensure this objective and the moments in the formula are well defined, assume $E[\|X_i\|^2]<\infty$ and $E[Y_i^2]<\infty$. Also assume $Q$ is nonsingular. Since $Q$ is a second-moment matrix, nonsingularity means it is positive definite.

Definition: Under $E[\|X_i\|^2]<\infty$, $E[Y_i^2]<\infty$, and nonsingular $Q=E[X_iX_i']$, the population linear projection coefficient is $$ \beta^*=Q^{-1}q=(E[X_iX_i'])^{-1}E[X_iY_i]. $$ It is the unique minimizer over $b\in\mathbb R^k$ of $E[(Y_i-X_i'b)^2]$. It is a projection coefficient; it need not be a structural parameter.

The finite-second-moment conditions matter. By the Cauchy–Schwarz inequality, each component of $E[X_iY_i]$ is finite, because $E[|X_{ij}Y_i|]\leq \sqrt{E[X_{ij}^2]E[Y_i^2]}$. The squared-error objective is finite for any fixed $b$, since $(Y_i-X_i'b)^2\leq 2Y_i^2+2(X_i'b)^2$, and $E[(X_i'b)^2]$ is finite under the regressor moment condition. Nonsingularity then ensures a unique minimizer, as shown below.

For a sample, define the sample versions of these moments. When the sample second-moment matrix is invertible, the OLS estimator has the same ratio form:

Q_n=\frac{1}{n}\sum_{i=1}^n X_iX_i', \qquad q_n=\frac{1}{n}\sum_{i=1}^n X_iY_i, \qquad \hat{\beta}_{\mathrm{OLS}}=Q_n^{-1}q_n.

The factor $1/n$ appears in both moments and cancels in their ratio. Equivalently, the usual normal equations are $\sum_{i=1}^n X_i(Y_i-X_i'\hat{\beta}_{\mathrm{OLS}})=0$. This representation makes the consistency argument transparent: the sample estimator combines two sample moments, then inverts the first.

Worked Example: A Scalar Regression Through the Origin

Suppose a hypothetical population gives equal probability to the three pairs $(X,Y)=(1,2),(2,1),(3,5)$. Fit a regression through the origin, so the regressor vector has the single component $X$ and there is no intercept. Calculate the population projection coefficient.

The population second moment and cross-moment are $$ E[X^2]=\frac{1^2+2^2+3^2}{3}=\frac{14}{3}, \qquad E[XY]=\frac{(1)(2)+(2)(1)+(3)(5)}{3}=\frac{19}{3}. $$

Thus the population coefficient is the ratio $$ \beta^*=\frac{E[XY]}{E[X^2]}=\frac{19/3}{14/3}=\frac{19}{14}\approx 1.357. $$

For example, a sample containing each of the three pairs once has $\sum X_iY_i=19$ and $\sum X_i^2=14$, so its OLS estimate is $19/14$ as well. A single such sample does not establish consistency. Consistency concerns what happens across repeated random samples as $n$ grows; the sample moments then approach the population moments used in this calculation.

Why the Ratio Converges

The argument has two ingredients. First, the law of large numbers makes each sample moment converge to its population counterpart. Second, matrix inversion and multiplication preserve convergence when the limiting matrix is nonsingular. Here the finite-second-moment conditions ensure the products in the sample moments have finite expectations: $E[X_{ij}X_{i\ell}]$ is finite by Cauchy–Schwarz, as is $E[X_{ij}Y_i]$. Applying the law of large numbers to each matrix and vector entry gives

Q_n \xrightarrow{p} Q, \qquad q_n \xrightarrow{p} q.

Because $Q$ is nonsingular, matrix inversion is continuous at $Q$. In particular, $Q_n$ is nonsingular with probability approaching one. One way to see the latter is that the determinant is a continuous function of the matrix entries, so $\det(Q_n)\xrightarrow{p}\det(Q)$, and $\det(Q)\ne 0$. Therefore, with probability approaching one, the OLS ratio is defined and converges to $Q^{-1}q$.

Theorem: If $(Y_i,X_i)$ are independent and identically distributed, $E[\|X_i\|^2]<\infty$, $E[Y_i^2]<\infty$, and $Q=E[X_iX_i']$ is nonsingular, then OLS is defined with probability approaching one and $$ \hat{\beta}_{\mathrm{OLS}}\xrightarrow{p}\beta^*=Q^{-1}q. $$ This is consistency for the population linear projection. No linear conditional-mean assumption is needed for this conclusion.

The probability limit statement means that for every $\varepsilon>0$, the probability that the coefficient estimate differs from $\beta^*$ by more than $\varepsilon$ approaches zero as $n$ increases. It does not say that every finite-sample estimate equals the projection coefficient, nor does it provide a rate of convergence or a limiting distribution. Those are separate questions.

Establish finite moments.
The assumed second moments make the entries of $X_iX_i'$ and $X_iY_i$ integrable, so the law of large numbers applies.

Take limits of the sample moments.
Entry by entry, $Q_n$ converges in probability to $Q$, and $q_n$ converges in probability to $q$.

Pass the limits through the ratio.
Nonsingularity of $Q$ makes inversion continuous at the limit; hence $Q_n^{-1}q_n$ converges in probability to $Q^{-1}q$.

Justification of the Population Projection Formula

The formula for $\beta^*$ follows directly from the objective. For any coefficient vector $b$, expand the expected squared error:

$$ E[(Y_i-X_i'b)^2] =E[Y_i^2]-2b'q+b'Qb. $$

Since $Q$ is positive definite, it is invertible. Set $\beta^*=Q^{-1}q$, so $q=Q\beta^*$. Completing the square gives

E[(Y_i-X_i'b)^2] =E[Y_i^2]-q'Q^{-1}q +(b-\beta^*)'Q(b-\beta^*).

The final term is nonnegative and equals zero only when $b=\beta^*$, because $Q$ is positive definite. Thus $\beta^*$ is the unique minimizer. This also explains why nonsingularity is essential: if $Q$ is singular, some regressor direction has no variation in the population second moments, and the coefficient vector need not be uniquely identified by the projection objective.

Worked Example: A Projection With an Intercept

Consider a hypothetical population with three equally likely observations in its support: $(Z,Y)=(-1,0),(0,1),(1,4)$. Regress $Y$ on an intercept and $Z$, so $X=(1,Z)'$. Calculate the population projection coefficients.

The second-moment matrix is $$ Q=E[XX']=\begin{pmatrix}1&E[Z]\\E[Z]&E[Z^2]\end{pmatrix} =\begin{pmatrix}1&0\\0&2/3\end{pmatrix}, $$ because $E[Z]=(-1+0+1)/3=0$ and $E[Z^2]=(1+0+1)/3=2/3$. The cross-moment vector is $$ q=E[XY]=\begin{pmatrix}E[Y]\\E[ZY]\end{pmatrix} =\begin{pmatrix}5/3\\4/3\end{pmatrix}. $$ Here $E[Y]=(0+1+4)/3=5/3$, and $E[ZY]=((-1)(0)+(0)(1)+(1)(4))/3=4/3$. Since $Q$ is nonsingular, $$ \beta^*=Q^{-1}q =\begin{pmatrix}1&0\\0&3/2\end{pmatrix} \begin{pmatrix}5/3\\4/3\end{pmatrix} =\begin{pmatrix}5/3\\2\end{pmatrix}. $$

The projection intercept is $5/3$ and its slope on $Z$ is $2$. These are coefficients defined by the population least-squares objective. The example specifies a distribution, not a structural equation claiming that changing $Z$ causes $Y$ to change by two units.

When the Projection Limit Is Structural

A structural interpretation requires assumptions that connect the observed outcome and regressors to a structural coefficient. For example, suppose the model is

Y_i=X_i'\beta_0+u_i, \qquad E[X_i u_i]=0,

with the moment conditions above and nonsingular $Q$. Then

q=E[X_iY_i] =E[X_iX_i']\beta_0+E[X_i u_i] =Q\beta_0.

It follows that $\beta^*=Q^{-1}q=\beta_0$, so the OLS consistency result gives $\hat{\beta}_{\mathrm{OLS}}\xrightarrow{p}\beta_0$. The orthogonality condition is doing the linking: it ensures the structural coefficient also solves the population projection moment equation. Other identification arguments may be relevant in a particular application, but they cannot be inferred merely from the fact that OLS has been calculated.

Worked Example: Orthogonality Links the Two Coefficients

Suppose a hypothetical population has $X=-1,0,1$, each with probability $1/3$, and disturbances $u=1,-2,1$, respectively. Let the structural equation be $Y=2+3X+u$. Include an intercept, so the regressor vector is $X_i=(1,X_i)'$. Verify the moment condition and find the population projection.

The outcome values are $Y=0,0,6$. The mean disturbance is $E[u]=(1-2+1)/3=0$, and the regressor-disturbance moments are $$ E[u]=0,\qquad E[Xu]=\frac{(-1)(1)+(0)(-2)+(1)(1)}{3}=0. $$ Thus $E[X_i u_i]=(E[u_i],E[X_i u_i])'=(0,0)'$. Also, $$ Q=\begin{pmatrix}1&E[X]\\E[X]&E[X^2]\end{pmatrix} =\begin{pmatrix}1&0\\0&2/3\end{pmatrix}, \qquad q=\begin{pmatrix}E[Y]\\E[XY]\end{pmatrix} =\begin{pmatrix}2\\2\end{pmatrix}. $$ Therefore $$ \beta^*=Q^{-1}q =\begin{pmatrix}1&0\\0&3/2\end{pmatrix} \begin{pmatrix}2\\2\end{pmatrix} =\begin{pmatrix}2\\3\end{pmatrix} =\beta_0. $$ The projection and structural coefficients coincide because the stated orthogonality condition holds, not simply because the equation was written with a disturbance term.

Common Mistakes and Practical Importance

Assuming a finite regressor matrix is enough. Nonsingular $E[X_iX_i']$ does not by itself guarantee that $E[X_iY_i]$ exists or that the squared-error objective is finite. For example, if $X_i=1$ and $Y_i$ has a Cauchy distribution, the regressor second moment is finite and nonsingular, but $E[X_iY_i]$ is undefined and $E[(Y_i-b)^2]$ is infinite for every $b$. The outcome second-moment condition rules out this failure.
Calling the projection coefficient a structural coefficient automatically. The OLS moment-ratio argument establishes convergence to $Q^{-1}q$. Equality with a structural coefficient requires an appropriate model and identification condition, such as $E[X_i u_i]=0$ in the displayed linear equation.
Treating nonsingularity as a small-sample guarantee. A nonsingular population matrix implies that the sample matrix is invertible with probability approaching one, not necessarily in every finite sample. Perfect collinearity in a particular sample can still prevent the usual OLS formula from being computed.
Claiming causation from consistency. Consistency says an estimator approaches its stated probability limit under specified sampling and moment conditions. It does not establish that the limit is a causal effect.

The moment representation is useful because it separates the large-sample argument into verifiable components: sampling behavior of the moments, existence of those moments, and identification through nonsingularity. It also makes clear exactly what OLS learns when the conditional mean or structural interpretation is uncertain: under the stated conditions, it learns the population linear projection.

Check Your Understanding

Answer each question using the moment-ratio argument and its assumptions.

Write the sample-moment expression for OLS in terms of $Q_n$ and $q_n$. Why does the factor $1/n$ not change the estimate?
Why are both $E[\|X_i\|^2]<\infty$ and $E[Y_i^2]<\infty$ useful for defining the population projection?
State the probability limit of OLS under the theorem's conditions. What role does nonsingularity play?
In the structural equation $Y_i=X_i'\beta_0+u_i$, show which moment condition ensures that $\beta^*=\beta_0$.
Does consistency for the population projection establish a causal interpretation? Explain what it does establish.