Consistency of Random-Design Least Squares

From Sample Equations to a Population Target

In random-design regression, the regressors vary from one observation to the next. OLS consistency is not a consequence of the normal equations alone: it depends on whether the sample equations approach a population equation with a unique solution. Two ingredients do the work. The empirical Gram matrix must converge to a nonsingular population matrix, and the sample regressor-error cross-product must converge to zero.

Let \(X_i\) be a \(p\)-dimensional column vector of regressors and let \(Y_i\) be a scalar response. The observations \((X_i,Y_i)\), for \(i=1,\ldots,n\), are independent and identically distributed. Suppose the data satisfy

Y_i=X_i'\beta_0+u_i,

where \(\beta_0\) is a fixed \(p\)-dimensional coefficient vector and \(u_i\) is the error. The vector \(X_i\) may include a constant component when the regression has an intercept.

Definition: The population Gram matrix is \(Q=E[X_iX_i']\). The population regressor-error cross-product is \(E[X_i u_i]\). The empirical versions are \(Q_n=n^{-1}\sum_{i=1}^n X_iX_i'\) and \(r_n=n^{-1}\sum_{i=1}^n X_i u_i\). The OLS estimator minimizes \(\sum_{i=1}^n(Y_i-X_i'b)^2\) over coefficient vectors \(b\), when a unique minimizer exists.

The conditions used below are: the observations are iid; \(E[\lVert X_i\rVert^2]<\infty\); \(E[\lVert X_i u_i\rVert]<\infty\); \(E[X_i u_i]=0\); and \(Q\) is positive definite. The moment conditions make the relevant expectations finite, and the zero cross-product condition makes \(\beta_0\) solve the population least-squares equation. Positive definiteness means that no nonzero coefficient direction has zero population variation in its fitted value.

The Sample Normal Equations

On any sample for which the columns of the design matrix have full rank, differentiating the sum of squared residuals gives the sample normal equations. Substituting the model into those equations isolates the estimation error:

\sum_{i=1}^n X_i(Y_i-X_i'\hat{\beta}) =0 \quad\Longrightarrow\quad Q_n(\hat{\beta}-\beta_0)=r_n.

If \(Q_n\) is invertible, this is the exact finite-sample identity

\hat{\beta}-\beta_0=Q_n^{-1}r_n.

Thus consistency follows if \(Q_n^{-1}\) remains bounded in probability and \(r_n\) converges in probability to zero. The first requirement comes from identification in the population; the second comes from the population orthogonality condition and a law of large numbers.

Worked Example: A Random-Design Regression with an Intercept

Let \(W_i\) take values \(-1\) and \(1\) with probability \(1/2\) each. Set \(X_i=(1,W_i)'\), and suppose \(Y_i=2+3W_i+u_i\), where \(u_i\) is independent of \(W_i\), has mean zero, and has finite variance. Verify the population conditions for OLS consistency.

Since \(E[W_i]=0\) and \(W_i^2=1\),

Q=E[X_iX_i'] = \begin{pmatrix} 1&E[W_i]\\ E[W_i]&E[W_i^2] \end{pmatrix} = \begin{pmatrix} 1&0\\ 0&1 \end{pmatrix}.

This matrix is positive definite: for every nonzero \(a=(a_1,a_2)'\), \(a'Qa=a_1^2+a_2^2>0\). Independence and \(E[u_i]=0\) give

E[X_i u_i] = \begin{pmatrix} E[u_i]\\ E[W_i u_i] \end{pmatrix} = \begin{pmatrix} 0\\ E[W_i]E[u_i] \end{pmatrix} = \begin{pmatrix} 0\\ 0 \end{pmatrix}.

The regressors are bounded, and \(u_i\) has a finite second moment, so the required moments are finite. Therefore the sample OLS coefficients converge in probability to the intercept \(2\) and slope \(3\).

Why the Sample Matrix Becomes Invertible

The weak law of large numbers applied to each entry of \(X_iX_i'\) gives \(Q_n\xrightarrow{p}Q\). Entrywise convergence implies convergence in any matrix norm because \(p\) is fixed. Positive definiteness then ensures that the sample matrix is invertible with probability approaching one.

Result: If \(Q_n\xrightarrow{p}Q\), where \(Q\) is a fixed positive-definite matrix, then \(P(Q_n\text{ is invertible})\to1\), and \(Q_n^{-1}\xrightarrow{p}Q^{-1}\) on those events.

To see this, let \(\lambda\) be the smallest eigenvalue of \(Q\). Positive definiteness gives \(\lambda>0\). For symmetric matrices, the smallest eigenvalue changes by at most the operator norm of the matrix perturbation. Hence, whenever \(\lVert Q_n-Q\rVert_{\mathrm{op}}<\lambda/2\), the smallest eigenvalue of \(Q_n\) is greater than \(\lambda/2\). On that event \(Q_n\) is invertible and \(\lVert Q_n^{-1}\rVert_{\mathrm{op}}\leq 2/\lambda\). The probability of the event tends to one. Moreover, on these events,

Q_n^{-1}-Q^{-1}=Q_n^{-1}(Q-Q_n)Q^{-1},

so the boundedness of \(Q_n^{-1}\), together with \(Q_n-Q\xrightarrow{p}0\), implies \(Q_n^{-1}-Q^{-1}\xrightarrow{p}0\). This proves the result. \(\square\)

For the cross-product, the vectors \(X_i u_i\) are iid and integrable by assumption. Applying the weak law to each coordinate yields \(r_n\xrightarrow{p}E[X_i u_i]=0\). This step needs integrability, not a central limit theorem or a finite variance for every coordinate of \(X_i u_i\).

Consistency of Random-Design OLS

Theorem: Suppose \((X_i,Y_i)\) are iid and \(Y_i=X_i'\beta_0+u_i\). If \(E[\lVert X_i\rVert^2]<\infty\), \(E[\lVert X_i u_i\rVert]<\infty\), \(E[X_i u_i]=0\), and \(Q=E[X_iX_i']\) is positive definite, then OLS is consistent: \(\hat{\beta}\xrightarrow{p}\beta_0\). If OLS is not uniquely defined on samples where \(Q_n\) is singular, it may be assigned any value on those samples without changing this conclusion.

Proof. The entrywise weak law gives \(Q_n\xrightarrow{p}Q\). The positive-definiteness argument above implies that \(Q_n\) is invertible with probability tending to one and that its inverse is bounded in probability on those events. Integrability of \(X_i u_i\) and the weak law give \(r_n\xrightarrow{p}0\). On samples where \(Q_n\) is invertible, the exact normal-equation identity applies, so

\hat{\beta}-\beta_0=Q_n^{-1}r_n.

To verify convergence explicitly, fix any \(\epsilon>0\). Choose a finite \(M\) large enough that the probability of \(\lVert Q_n^{-1}\rVert_{\mathrm{op}}>M\), on the invertible event, is as small as desired for all sufficiently large \(n\). On the event \(\lVert Q_n^{-1}\rVert_{\mathrm{op}}\leq M\), the inequality

\lVert\hat{\beta}-\beta_0\rVert \leq M\lVert r_n\rVert

shows that \(\lVert\hat{\beta}-\beta_0\rVert\leq\epsilon\) whenever \(\lVert r_n\rVert\leq\epsilon/M\). The probability of this last event tends to one, while the probability of a singular \(Q_n\) also tends to zero. Therefore \(P(\lVert\hat{\beta}-\beta_0\rVert>\epsilon)\to0\), which is consistency in probability. \(\square\)

Worked Example: Orthogonality Without a Zero Conditional Mean

Consider a regression through the origin with scalar regressor \(X_i\), where \(P(X_i=1)=P(X_i=2)=1/2\). Let \(u_i=2\) when \(X_i=1\), and \(u_i=-1\) when \(X_i=2\). Define \(Y_i=\beta_0 X_i+u_i\). Check whether OLS is consistent for \(\beta_0\).

The population Gram matrix is the scalar

Q=E[X_i^2]=\tfrac12(1^2)+\tfrac12(2^2)=\tfrac52>0.

The regressor-error cross-product is

E[X_i u_i]=\tfrac12(1\cdot2)+\tfrac12(2\cdot(-1))=1-1=0.

All variables have finite support, so the required moments are finite. The theorem therefore gives \(\hat{\beta}\xrightarrow{p}\beta_0\). Yet \(E[u_i\mid X_i=1]=2\) and \(E[u_i\mid X_i=2]=-1\), neither of which is zero. This example shows that the theorem needs unconditional orthogonality \(E[X_i u_i]=0\), not necessarily the stronger condition \(E[u_i\mid X_i]=0\).

Worked Example: Endogeneity Shifts the OLS Limit

Let \(X_i\sim N(0,1)\), let \(\varepsilon_i\) be independent of \(X_i\) with mean zero and finite variance, and set \(u_i=X_i+\varepsilon_i\). Suppose \(Y_i=\beta_0X_i+u_i\) and fit a regression through the origin. Determine the probability limit of OLS.

Here \(Q=E[X_i^2]=1\), but the cross-product is not zero:

E[X_i u_i] =E[X_i(X_i+\varepsilon_i)] =E[X_i^2]+E[X_i]E[\varepsilon_i] =1.

The scalar normal equations give \(\hat{\beta}-\beta_0=(n^{-1}\sum_i X_i u_i)/(n^{-1}\sum_i X_i^2)\). By the weak law, the numerator converges in probability to \(1\), and the denominator converges in probability to \(1\). The denominator limit is nonzero, so the ratio converges to \(1\). Consequently, \(\hat{\beta}\xrightarrow{p}\beta_0+1\), not \(\beta_0\). A well-behaved Gram matrix alone does not ensure consistency; the regressor-error cross-product condition is essential.

Common Mistakes and Why the Conditions Matter

Using \(E[u_i]=0\) in place of \(E[X_i u_i]=0\). A zero mean error does not ensure that errors average to zero in the directions weighted by the regressors. The relevant quantity in the normal equations is the vector \(X_i u_i\), not \(u_i\) alone.
Assuming a nonsingular sample matrix is guaranteed for every sample size. Positive definiteness of \(Q\) guarantees invertibility with probability tending to one, not necessarily for every finite sample. Consistency tolerates singular samples whose probability vanishes.
Ignoring population identification. If \(Q\) is singular, some coefficient directions cannot be distinguished using the regressors. The sample equations cannot establish consistency of a uniquely identified full coefficient vector in that case.
Requiring more than the argument needs. A zero conditional mean \(E[u_i\mid X_i]=0\) implies \(E[X_i u_i]=0\) under suitable integrability, but the proof uses the latter condition directly. It does not establish that conditional mean zero is necessary.

The consistency argument ends with convergence in probability. It does not, by itself, describe the scale or distribution of the estimation error around \(\beta_0\); those questions require additional asymptotic results.

Check Your Understanding

Use the sample normal equations and the population moment conditions to answer the following.

Which two sample quantities determine the OLS estimation error through the identity \(\hat{\beta}-\beta_0=Q_n^{-1}r_n\)?
Why does positive definiteness of \(Q\), together with \(Q_n\xrightarrow{p}Q\), imply that sample OLS is uniquely defined with probability tending to one?
Which integrability condition permits the weak law to be applied to \(n^{-1}\sum_i X_i u_i\)?
Does consistency require \(E[u_i\mid X_i]=0\), or does the proof use a weaker moment condition? State it.
If \(Q_n\xrightarrow{p}Q\) with \(Q\) positive definite but \(E[X_i u_i]\neq0\), what does the normal-equation argument suggest about the probability limit of OLS?