The IID Random-Design Regression Framework

Why Specify the Regression Design?

Regression asymptotics begin by specifying how the observations are generated. In a random-design model, each observation contains a random covariate vector and a random response. The pairs are sampled from one common distribution, so the covariates themselves vary across repeated samples. This setup differs from fixed-design regression, where the covariate values are treated as given.

The setup also distinguishes a model's population target from the coefficient computed in a particular sample. The population target is defined by minimizing expected squared prediction error. Under a linear conditional-mean model and a suitable nondegeneracy condition, that target is the model's coefficient. Without those assumptions, least squares still has a population target, but it may describe a best linear approximation rather than a true conditional-mean relationship.

The IID Random-Design Setup

The iid condition applies to the pairs. It means that different observations are independent and have the same joint distribution of covariates and response. It does not say that \(X_i\) and \(Y_i\) are independent within an observation; regression is generally used precisely because they are related. Nor does it require the error to be independent of the covariates.

A common representation of the relationship is

where \(\beta_0\in\mathbb{R}^p\) is a fixed coefficient vector and \(u_i\) is the error. The corresponding population conditional-mean assumption is

Conditional mean zero is stronger than merely requiring \(E[u]=0\): it rules out systematic error patterns across covariate values. It does not require constant conditional variance, normal errors, or independence between \(u\) and \(X\). It does require that the conditional expectation be well defined; below, finite second moments ensure the needed integrability.

The Population Least-Squares Target

For a candidate coefficient \(b\in\mathbb{R}^p\), define its population squared-error risk by \(R(b)=E[(Y-X'b)^2]\). A population least-squares coefficient is any vector minimizing this risk. To state a simple sufficient condition for existence and uniqueness, assume \(E[Y^2]<\infty\), \(E[\|X\|^2]<\infty\), and that \(Q=E[XX']\) is positive definite. Positive definiteness means \(a'Qa>0\) for every nonzero vector \(a\in\mathbb{R}^p\).

The expectations in this definition are with respect to the population distribution, not the observed sample. The finite-moment conditions make the risk finite for every fixed \(b\); for example, \(E[|XY|]\) is finite by the Cauchy–Schwarz inequality. The following derivation establishes the target and its uniqueness.

Proof. Expanding the square and taking expectations gives

Positive definiteness makes \(Q\) invertible, so define \(\beta^*=Q^{-1}c\). Since \(Q\beta^*=c\), expanding the quadratic around \(\beta^*\) yields

The right-hand side is nonnegative and is strictly positive whenever \(b\ne\beta^*\), by positive definiteness. Therefore \(\beta^*\) is the unique minimizer, and it equals \(Q^{-1}c\). \(\square\)

The next result connects this population target to the conditional mean assumption. It also makes explicit why the assumption concerns the error's mean at each covariate value, rather than only its overall mean.

Proof. The finite second moments imply that each component of \(Xu\) is integrable, since \(E[|X_j u|]\leq \{E[X_j^2]E[u^2]\}^{1/2}<\infty\). Because \(X\) is determined by itself, the defining property of conditional expectation gives

Consequently,

The first theorem gives \(\beta^*=Q^{-1}c=Q^{-1}Q\beta_0=\beta_0\). Positive definiteness supplies uniqueness. \(\square\)

Worked Examples

What the Framework Does—and Does Not—Assume

The setup gives a clear interpretation to the target of least squares without imposing a full probability model for the response. In particular, errors may be heteroskedastic: \(E[u^2\mid X]\) can vary with \(X\). They may also have nonnormal distributions. Neither feature changes the definition of \(\beta^*\). The conditional mean-zero assumption, when imposed, identifies that target with the coefficient in the linear conditional mean.

The fixed-\(p\) condition means that the number of regressors stays constant as the sample size grows. It keeps the parameter dimension stable in the asymptotic framework. This tutorial defines the data-generating setup and population target; it does not by itself establish that a sample estimator converges to the target or has a particular limiting distribution. Such conclusions require additional results and assumptions.

If \(Q\) is not positive definite, the quadratic risk need not have a unique coefficient vector: different vectors may produce the same predictions on the population covariate support. Positive definiteness is therefore a convenient sufficient condition for a unique coefficient target, not a property guaranteed merely by calling the data random-design.

Common Mistake

Wrong reasoning: “The observations are iid and the errors average to zero, so the regression error is exogenous and the least-squares target equals the model coefficient.” This confuses three different claims. Iid sampling concerns independence and identical distribution across observations; \(E[u]=0\) is an unconditional statement; and \(E[u\mid X]=0\) rules out a nonzero conditional error mean across covariate values.

Correction: Check the conditional mean explicitly. In the third example, \(E[u]=E[X]=0\), but \(E[u\mid X]=X\ne0\), and the target slope is 2 rather than the proposed slope 1. When conditional mean zero is not justified, describe \(\beta^*\) as the population best linear least-squares coefficient, not automatically as a structural or conditional-mean parameter.