Population Linear Projection Coefficients

Why Define a Population Linear Predictor?

A linear regression equation is often used to summarize how an outcome varies with several predictors. That summary can be useful even when the relationship in the population is not exactly linear. To define the summary precisely, we need to specify which line or plane is being chosen and what criterion makes it the best one.

The population linear projection answers this question by choosing coefficients that minimize the population’s expected squared prediction error. Its coefficients are features of the joint distribution of the outcome and regressors. They are not, by definition, causal effects, and their definition does not require the conditional mean of the outcome to be linear.

Definition: Let \(Y\) be an outcome and let \(X=(X_1,\ldots,X_k)'\) be a vector of regressors. The population linear projection of \(Y\) on \(X\) is the linear predictor \(a+X'b\) whose coefficients \((a,b)\) minimize \(E[(Y-a-X'b)^2]\). The minimizing intercept is the population projection intercept; the entries of the minimizing vector \(b\) are the population projection coefficients.

Here \(E[\cdot]\) denotes expectation under the population distribution of \((Y,X)\). The expectation weights each possible outcome–regressor combination according to how it occurs in that distribution. Thus a population projection is not defined by selecting a particular finite sample or by requiring the predictor to match every outcome exactly.

Conditions for Unique Coefficients

Assume that \(Y\) and every component of \(X\) have finite second moments, so that the expected squared prediction error is finite for finite coefficients. Define the regressor mean vector, the covariance matrix of the regressors, and their covariance vector with the outcome as

\mu_X=E[X],\qquad \Sigma_X=E[(X-\mu_X)(X-\mu_X)'],\qquad c=E[(X-\mu_X)(Y-E[Y])].

For a unique slope vector, assume \(\Sigma_X\) is positive definite: for every nonzero vector \(v\), \(v'\Sigma_Xv>0\). This condition means that no nonzero linear combination of the centered regressors has zero variance. If it fails, the coefficients may not be uniquely determined.

Under these conditions, the population projection coefficients are

b=\Sigma_X^{-1}c,\qquad a=E[Y]-\mu_X'b.

The slope formula says that the coefficients depend on the joint distribution through the regressors’ covariance matrix and their covariances with the outcome. The intercept then makes the predictor’s population mean equal to the outcome’s population mean.

Deriving the Coefficients from Prediction Error

The derivation can be separated into two steps. First, fix any slope vector \(b\) and choose the intercept that gives the smallest expected squared error for those slopes. Write \(\mu_Y=E[Y]\). Expanding the error around the means gives

Y-a-X'b=(Y-\mu_Y)-(X-\mu_X)'b+\mu_Y-a-\mu_X'b.

The first two terms on the right have expectation zero. Consequently, the expected squared error is the variance of \((Y-\mu_Y)-(X-\mu_X)'b\), plus the square of the constant \(\mu_Y-a-\mu_X'b\). For fixed \(b\), that square is minimized at zero, giving \(a=\mu_Y-\mu_X'b\). With this intercept, the remaining objective is

E\left[\{(Y-\mu_Y)-(X-\mu_X)'b\}^2\right] =\operatorname{Var}(Y)-2b'c+b'\Sigma_Xb.

Now define \(\beta=\Sigma_X^{-1}c\), so \(c=\Sigma_X\beta\). Completing the square in the objective shows exactly why this slope vector minimizes it:

\operatorname{Var}(Y)-2b'c+b'\Sigma_Xb = \operatorname{Var}(Y)-c'\Sigma_X^{-1}c +(b-\beta)'\Sigma_X(b-\beta).

The first two terms do not depend on \(b\). Positive definiteness makes the final term nonnegative and equal to zero only when \(b=\beta\). Thus \(b=\Sigma_X^{-1}c\) is the unique minimizing slope vector, and the intercept found in the first step is \(a=E[Y]-\mu_X'b\). This establishes the formulas from the expected squared prediction error, rather than assuming a particular conditional-mean model.

Worked Examples

Worked Example: A Projection for a Discrete Nonlinear Relationship

Consider a hypothetical population with three equally likely cases. The regressor \(X\) takes values \(0,1,2\), and the corresponding outcome \(Y\) takes values \(1,2,5\). Find the population projection of \(Y\) on an intercept and \(X\).

The means are \(E[X]=(0+1+2)/3=1\) and \(E[Y]=(1+2+5)/3=8/3\). The regressor variance is

\operatorname{Var}(X)=\frac{(0-1)^2+(1-1)^2+(2-1)^2}{3}=\frac{2}{3}.

The covariance is

\operatorname{Cov}(X,Y) =\frac{(0-1)(1-8/3)+(1-1)(2-8/3)+(2-1)(5-8/3)}{3} =\frac{4}{3}.

With one regressor, the slope is covariance divided by variance: \(b=(4/3)/(2/3)=2\). The intercept is \(a=8/3-2(1)=2/3\). Therefore the population linear projection is \(2/3+2X\).

The relationship in these three cases is not exactly linear: at \(X=0,1,2\), the projection predicts \(2/3,8/3,14/3\), while the outcomes are \(1,2,5\). The coefficients summarize the best linear prediction under this population distribution; they do not require every outcome to lie on the line.

Worked Example: Two Regressors and a Nonlinear Remainder

Consider a hypothetical population with four equally likely cases. The regressor pairs \((X_1,X_2)\) are \((-1,-1),(-1,1),(1,-1),(1,1)\), and the respective outcomes are \(10,6,12,12\). Find the projection coefficients for a predictor with an intercept and both regressors.

Both regressor means are zero. Each has variance 1, and their covariance is zero, so \(\Sigma_X=\begin{pmatrix}1&0\\0&1\end{pmatrix}\). The outcome mean is \(E[Y]=(10+6+12+12)/4=10\). The covariance vector is

c= \begin{pmatrix} E[X_1(Y-10)]\\ E[X_2(Y-10)] \end{pmatrix} = \begin{pmatrix} (0+4+2+2)/4\\ (0-4-2+2)/4 \end{pmatrix} = \begin{pmatrix}2\\-1\end{pmatrix}.

Since \(\Sigma_X\) is the identity matrix, \(b=\Sigma_X^{-1}c=(2,-1)'\). The intercept is \(a=E[Y]-E[X]'b=10\), so the projection is \(10+2X_1-X_2\).

Within this linear predictor, increasing \(X_1\) by one unit while holding \(X_2\) fixed changes the predicted value by 2 outcome units; increasing \(X_2\) by one unit while holding \(X_1\) fixed changes it by \(-1\) outcome unit. These are descriptions of the fitted linear summary, not claims that changing either regressor causes an outcome change.

Worked Example: Projecting a Quadratic Conditional Mean

Suppose, hypothetically, that \(X\) is uniformly distributed on the interval from \(-1\) to \(2\), and \(Y=X^2\). Find the population projection of \(Y\) on an intercept and \(X\).

For a uniform distribution on this interval, \(E[X]=1/2\). Direct integration gives \(E[X^2]=1\) and \(E[X^3]=5/4\). Therefore \(\operatorname{Var}(X)=E[X^2]-E[X]^2=1-1/4=3/4\), and \(\operatorname{Cov}(X,Y)=E[X^3]-E[X]E[X^2]=5/4-1/2=3/4\).

The slope is \(b=(3/4)/(3/4)=1\), and the intercept is \(a=E[Y]-bE[X]=1-(1)(1/2)=1/2\). The projection is \(1/2+X\).

Here the conditional mean is exactly \(E[Y\mid X=x]=x^2\), which is curved, whereas the projection is a line. The line is the best predictor among linear functions under the stated distribution of \(X\); it is not a replacement claim that the conditional mean is linear.

How to Interpret Projection Coefficients

A projection coefficient has a precise interpretation within the chosen linear predictor. In a model with several regressors, the coefficient on \(X_j\) gives the change in the predictor \(a+X'b\) associated with a one-unit increase in \(X_j\), holding the other regressor values fixed. Its units are outcome units per unit of \(X_j\). The intercept is the predictor’s value when all regressors equal zero, though that value may not describe a typical or even possible case.

These statements concern the predictor, not necessarily the conditional mean \(E[Y\mid X]\). If the conditional mean is nonlinear, the projection coefficient is a single distribution-dependent summary of the relationship. Changing the population distribution of the regressors can therefore change the projection coefficients, even if the conditional relationship itself stays the same.

Nor does minimizing prediction error establish a causal interpretation. A causal claim requires additional design or identification assumptions; it does not follow from the projection definition alone. When interpreting a coefficient, keep separate the mathematical meaning of the linear predictor and any substantive claim about interventions.

Common Mistake

Wrong reasoning: “The coefficient on \(X\) is the increase in the conditional mean for every one-unit increase in \(X\), so the conditional mean must be linear.”

Correction: The coefficient describes the change in the population’s best linear predictor. It equals a constant conditional-mean slope only under additional structure, such as a correctly specified linear conditional mean. The quadratic example shows that a well-defined projection exists even when the conditional mean is curved.

Key takeaway: With finite second moments and a positive-definite regressor covariance matrix, the population linear projection has slope vector \(\Sigma_X^{-1}c\) and intercept \(E[Y]-E[X]'b\). These coefficients minimize expected squared prediction error; they do not by themselves imply a correct linear conditional mean or a causal effect.

Check Your Understanding

Try these questions using the definitions and results above.

In your own words, what population criterion defines the linear projection coefficients?
What condition on the regressor covariance matrix ensures a unique slope vector, and what can go wrong if it fails?
If \(E[Y]=12\), \(E[X]=3\), and the projection slope is 2 for a single regressor, what is the projection intercept?
Does a linear projection require \(E[Y\mid X]\) to be linear? Explain briefly.
Why does a projection coefficient not, by itself, establish that changing a regressor causes a change in the outcome?