Conditional Expectation as an Orthogonal Projection

From Gaussian Conditioning to Projection

A conditional mean answers a prediction question: given the observed value of a predictor vector \(X\), what is the mean of a response \(Y\)? A different-looking question asks which affine function of \(X\) minimizes mean squared prediction error. For jointly Gaussian variables, these questions have the same answer. The conditional mean is itself an affine function, and its coefficients satisfy the least-squares projection equations.

This connection is useful because it gives two complementary ways to understand Gaussian conditioning. The distributional view identifies the conditional law; the projection view explains why its mean is the best predictor and how to calculate it from covariance information. The argument also works when the components of \(X\) are redundant, provided we distinguish the fitted projection from a possibly nonunique coefficient vector.

Affine Predictors and Orthogonality

Assume \(Y\) is a scalar random variable and \(X\) is a \(p\)-dimensional random vector, with \(E[Y^2]<\infty\) and \(E[\|X\|^2]<\infty\). An affine predictor has the form \(a+b'X\), where \(a\) is a scalar and \(b\in\mathbb{R}^p\). These predictors form a linear subspace of the space of square-integrable random variables: constants are included as well as the components of \(X\).

Definition: The affine least-squares predictor of \(Y\) based on \(X\) is an affine function \(a+b'X\) that minimizes \(E[(Y-a-b'X)^2]\) over all \(a\in\mathbb{R}\) and \(b\in\mathbb{R}^p\). A residual is orthogonal to a random variable \(W\) when its expected product with \(W\) is zero.

The projection equations say that the residual is orthogonal to every constant and every component of \(X\). Writing \(R=Y-a-b'X\), these conditions are \(E[R]=0\) and \(E[(X-E[X])R]=0\). The second equation is a vector equation with \(p\) entries. Together they are the least-squares normal equations.

To see why they characterize a minimum, suppose a candidate predictor \(m=a+b'X\) has residual \(R=Y-m\) satisfying those equations. Any other affine predictor differs from it by an affine random variable \(D=d+q'X\). Then \(Y-(m+D)=R-D\), and \(E[(R-D)^2]=E[R^2]-2E[RD]+E[D^2]\). The orthogonality equations give \(E[RD]=0\), so this equals \(E[R^2]+E[D^2]\), which cannot be smaller than \(E[R^2]\). The fitted random variable \(m\) is therefore the unique least-squares projection, even if multiple coefficient vectors represent it when the predictors are redundant.

The Gaussian Projection Formula

Now suppose \((Y,X')'\) is jointly Gaussian, allowing a degenerate joint Gaussian law. Define \(\mu_Y=E[Y]\), \(\mu_X=E[X]\), \(\Sigma=\operatorname{Cov}(X,X)\), \(c=\operatorname{Cov}(X,Y)\), and \(\sigma_Y^2=\operatorname{Var}(Y)\). The matrix \(\Sigma\) is symmetric and positive semidefinite. When it is singular, \(\Sigma^\dagger\) denotes its Moore–Penrose pseudoinverse.

Theorem: For jointly Gaussian \(Y\) and \(X\), a version of the conditional mean is \(E[Y\mid X]=\mu_Y+c'\Sigma^\dagger(X-\mu_X)\) almost surely. This is an affine least-squares predictor. Its residual is orthogonal to constants and to every component of \(X\), and its residual variance is \(\sigma_Y^2-c'\Sigma^\dagger c\). If \(\Sigma\) is nonsingular, \(\Sigma^\dagger=\Sigma^{-1}\), and the affine coefficient vector is unique.

The pseudoinverse is needed only to state a convenient coefficient vector in the singular case. The fitted random variable in the formula is unique almost surely. Here is a proof that also explains the role of Gaussianity.

Check that the covariance vector is estimable.
If \(u\) is in the null space of \(\Sigma\), then \(\operatorname{Var}(u'(X-\mu_X))=u'\Sigma u=0\). Thus \(u'(X-\mu_X)=0\) almost surely, so \(u'c=\operatorname{Cov}(u'X,Y)=0\). Therefore \(c\) is orthogonal to the null space of \(\Sigma\). Since \(\Sigma\) is symmetric, its range is the orthogonal complement of its null space; hence \(c\) belongs to the range of \(\Sigma\), and \(\Sigma\Sigma^\dagger c=c\).

Construct the residual and verify orthogonality.
Let \(b=\Sigma^\dagger c\), \(m=\mu_Y+b'(X-\mu_X)\), and \(R=Y-m\). Then \(E[R]=0\), and \(\operatorname{Cov}(X,R)=c-\Sigma b=c-\Sigma\Sigma^\dagger c=0\). Thus the residual satisfies the projection equations.

Use joint Gaussianity to identify the conditional mean.
\((R,X')'\) is jointly Gaussian because it is an affine transformation of \((Y,X')'\). Its zero cross-covariance implies \(R\) and \(X\) are independent, including in the degenerate case. Consequently \(E[R\mid X]=E[R]=0\), and \(E[Y\mid X]=m\) almost surely.

Calculate the residual variance and establish optimality.
Because \(R=Y-\mu_Y-b'(X-\mu_X)\), \(\operatorname{Var}(R)=\sigma_Y^2-2b'c+b'\Sigma b\). Using \(\Sigma b=c\), this becomes \(\sigma_Y^2-b'c=\sigma_Y^2-c'\Sigma^\dagger c\). Orthogonality proves that \(m\) minimizes squared error among affine predictors. Moreover, as a conditional mean it minimizes squared error among all square-integrable predictors that are measurable functions of \(X\): for such a function \(g(X)\), the conditional-mean property gives \(E[(Y-g(X))^2]=E[(Y-m)^2]+E[(m-g(X))^2]\). The cross term vanishes since \(E[Y-m\mid X]=0\). \(\square\)

The last identity makes an important distinction. Least squares over affine functions gives a projection onto the span of \(1\) and the components of \(X\). Conditional expectation is a projection onto the larger space of square-integrable functions of \(X\). In the jointly Gaussian setting, the conditional mean happens to lie in the affine space, so both projections coincide.

Worked Gaussian Projections

Worked Example: One Predictor

Suppose \(Y\) and \(X\) are jointly Gaussian with \(E[Y]=10\), \(E[X]=4\), \(\operatorname{Var}(Y)=9\), \(\operatorname{Var}(X)=4\), and \(\operatorname{Cov}(X,Y)=3\). Find the conditional mean, the affine predictor, and the residual variance.

Here \(\Sigma=4\) and \(c=3\), so the slope is \(b=\Sigma^{-1}c=3/4\). The intercept is \(a=\mu_Y-b\mu_X=10-(3/4)(4)=7\). Thus the affine predictor and conditional mean are \(m=7+(3/4)X=10+(3/4)(X-4)\).

Its residual is \(R=Y-10-(3/4)(X-4)\). It has mean zero, and \(\operatorname{Cov}(X,R)=3-(3/4)(4)=0\), as required by the projection equation. Since the pair is jointly Gaussian, \(R\) is independent of \(X\), so this predictor is the conditional mean. The residual variance is \(9-(3^2/4)=9-9/4=27/4\).

Worked Example: Two Correlated Predictors

Let \((Y,X_1,X_2)\) be jointly Gaussian with \(E[Y]=5\), \(E[X]=(1,-1)'\), \(\operatorname{Var}(Y)=6\), and \(\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}\), \(c=\operatorname{Cov}(X,Y)=(3,2)'\). Calculate the least-squares coefficients and check the normal equations.

The determinant of \(\Sigma\) is \(4(2)-1(1)=7\), so \(\Sigma^{-1}=(1/7)\begin{pmatrix}2&-1\\-1&4\end{pmatrix}\). Therefore \(b=\Sigma^{-1}c=(1/7)(4,5)'=(4/7,5/7)'\).

The normal equations check directly: \(\Sigma b=(1/7)(16+5,\,4+10)'=(3,2)'=c\). The fitted conditional mean is \(m=5+(4/7)(X_1-1)+(5/7)(X_2+1)\). Equivalently, its intercept in the form \(a+b_1X_1+b_2X_2\) is \(a=5-(4/7)(1)-(5/7)(-1)=36/7\).

The explained variance is \(c'b=3(4/7)+2(5/7)=22/7\). Hence the residual variance is \(6-22/7=20/7\), which is positive. The residual is orthogonal to both centered predictors by the checked normal equations; joint Gaussianity then makes it independent of the predictor vector.

Worked Example: Redundant Predictors

Let \(Z\sim N(0,1)\) and \(\varepsilon\sim N(0,1)\) be independent. Set \(X_1=Z\), \(X_2=2Z\), and \(Y=2Z+\varepsilon\). Find the conditional mean and explain why the least-squares coefficients are not unique.

The predictor covariance and covariance vector are \(\Sigma=\begin{pmatrix}1&2\\2&4\end{pmatrix}\) and \(c=(2,4)'\). The matrix is singular because \(X_2=2X_1\) exactly. A coefficient vector \(b=(b_1,b_2)'\) satisfies the normal equations precisely when \(\Sigma b=(b_1+2b_2)(1,2)'=(2,4)'\), or \(b_1+2b_2=2\).

There are many solutions, such as \((2,0)'\) and \((0,1)'\). Each gives the same fitted value: \(b_1X_1+b_2X_2=(b_1+2b_2)Z=2Z\). The pseudoinverse selects \(b=\Sigma^\dagger c=(2/5,4/5)'\), which also satisfies the equation and gives \(m=2Z\). Since \(Y=2Z+\varepsilon\) and \(\varepsilon\) is independent of \(X\), \(E[Y\mid X]=2Z\). The residual variance is \(\operatorname{Var}(\varepsilon)=1\); the formula agrees, since \(\operatorname{Var}(Y)=5\) and \(c'\Sigma^\dagger c=4\).

What Orthogonality Does—and Does Not—Say

The least-squares equations always express orthogonality to the predictors. For any square-integrable \(Y\) and \(X\), a minimizing affine predictor has residual orthogonal to \(1,X_1,\ldots,X_p\); this follows from the projection argument and does not require Gaussianity. What is special about the jointly Gaussian case is that the affine residual, being jointly Gaussian with \(X\), is independent of \(X\) when their cross-covariance is zero. That independence makes the affine predictor equal to the full conditional mean.

A common mistake is to argue that an uncorrelated residual must be independent of the predictors, and therefore the conditional mean must be affine. Uncorrelatedness alone does not imply independence for general distributions. Without joint Gaussianity, the affine least-squares predictor still solves the normal equations, but \(E[Y\mid X]\) may be nonlinear and may predict better. Gaussianity is the condition that closes this gap here.

A second practical caution concerns singular predictors. If \(\Sigma\) is singular, using \(\Sigma^{-1}\) is invalid and reporting one coefficient vector as uniquely determined is misleading. The fitted projection remains unique almost surely; redundant predictors can yield several coefficient vectors for that same fitted random variable. The pseudoinverse gives a consistent choice, while the projection itself is the substantive object.

Key takeaway: In a jointly Gaussian model, the conditional mean is the affine projection of the response onto the constants and predictors. Its residual is orthogonal to every predictor, and Gaussianity upgrades that orthogonality to independence. With redundant predictors, coefficients may not be unique even though the conditional mean is.

Check Your Understanding

Use the projection equations and the Gaussian conditioning argument to answer these questions.

For jointly Gaussian \(Y\) and \(X\) with nonsingular \(\operatorname{Var}(X)\), what equation determines the slope vector of the affine conditional mean?
Why must \(\operatorname{Cov}(X,Y)\) lie in the range of \(\operatorname{Var}(X)\), even when that covariance matrix is singular?
What additional property of a jointly Gaussian residual and predictor vector turns zero cross-covariance into the conditional-mean result?
If two different coefficient vectors give the same fitted random variable because predictors are redundant, which object is the unique least-squares projection?
Does the affine least-squares predictor have to equal \(E[Y\mid X]\) when \(Y\) and \(X\) are not jointly Gaussian? Explain the role of orthogonality in your answer.