Gaussian Regression as Conditional Expectation

Why Gaussian Regression Is a Conditional Mean

In regression, a response is predicted from observed predictors. For a general distribution, the conditional mean can be a nonlinear function of those predictors. Joint Gaussianity gives a particularly structured case: the conditional mean is affine, and its slope is determined entirely by covariance blocks. This makes Gaussian regression a direct consequence of the geometry of the joint distribution, rather than an additional modelling assumption.

Let \(X\) be a \(p\)-dimensional predictor vector and \(Y\) a \(q\)-dimensional response vector. The response may be scalar, but the vector form also allows several outcomes to be predicted at once. Assume that the joint vector \((X^T,Y^T)^T\) is Gaussian, has mean \((\mu_X^T,\mu_Y^T)^T\), and has positive-definite covariance matrix

Here \(\Sigma_{XX}\) is \(p\times p\), \(\Sigma_{YY}\) is \(q\times q\), and \(\Sigma_{YX}\) is \(q\times p\). Positive definiteness ensures that \(\Sigma_{XX}\) is invertible and that the Schur complement used below is positive definite.

Deriving the Conditional Mean from Covariance Blocks

The formula can be obtained from the conditional Gaussian density. Write \(A=\Sigma_{XX}\), \(C=\Sigma_{XY}\), and \(D=\Sigma_{YY}\), and define \(S=D-C^TA^{-1}C\). This is the Schur complement of \(A\) in \(\Sigma\), hence the conditional covariance of \(Y\) given \(X\). The conditional Gaussian law follows by completing the square in the joint density; the key quadratic-form identity is

where \(u=x-\mu_X\) and \(v=y-\mu_Y\). The determinant identity \(\det(\Sigma)=\det(A)\det(S)\) means that, when \(x\) is fixed, the joint density as a function of \(y\) is proportional to a Gaussian density with mean \(\mu_Y+C^TA^{-1}(x-\mu_X)\) and covariance \(S\). Since \(C^T=\Sigma_{YX}\), this gives the regression coefficient in the definition.

For completeness, the density argument also verifies that the stated location is the conditional expectation, not just the centre of a quadratic expression. The conditional density is symmetric about that location, and its centred first moment is zero: after translating by the location, the integrand for each coordinate of the first moment is odd, while the Gaussian density is even and has a finite first moment. Thus integration gives the conditional mean. The covariance is \(S\), which is independent of the observed value \(x\). The affine conditional mean and constant conditional covariance are both consequences of joint Gaussianity.

Multiple Predictors: Matrix Calculation

With several predictors, the scalar slope is replaced by a matrix. Each row of \(B\) describes how the conditional mean of one response component changes with all predictor components. The matrix product must be ordered carefully: \(\Sigma_{YX}\) is \(q\times p\), so it multiplies the \(p\times p\) inverse \(\Sigma_{XX}^{-1}\) on the right.

Why This Is a Regression Coefficient

The covariance formula has a second interpretation: it gives the unique best affine predictor under squared-error loss. This connection explains why the same coefficient appears in regression, even when the goal is described as prediction rather than conditioning.

To prove the result, set \(B_*=\Sigma_{YX}\Sigma_{XX}^{-1}\), \(a_*=\mu_Y-B_*\mu_X\), and \(\varepsilon=Y-\mu_Y-B_*(X-\mu_X)\). Then \(E[\varepsilon]=0\), and \(\operatorname{Cov}(\varepsilon,X)=\Sigma_{YX}-B_*\Sigma_{XX}=0\). Also \(\operatorname{Cov}(\varepsilon)=\Sigma_{YY}-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY}\). For arbitrary \(a,B\), write \(\Delta=B-B_*\) and \(d=a+B\mu_X-\mu_Y\). The prediction error is \(Y-a-BX=\varepsilon-\Delta(X-\mu_X)-d\). Taking its squared Euclidean norm and expectation, the cross terms vanish: the terms involving \(d\) vanish because both centred random vectors have mean zero, and the term involving \(\varepsilon\) and \(X-\mu_X\) vanishes because their cross-covariance is zero. Hence

Both additional terms are nonnegative. Since \(\Sigma_{XX}\) is positive definite, \(\operatorname{tr}(\Delta\Sigma_{XX}\Delta^T)=0\) only when \(\Delta=0\); also \(\|d\|^2=0\) only when \(d=0\). These conditions give \(B=B_*\) and \(a=a_*\), proving existence and uniqueness of the minimizer. \(\square\)

For a jointly Gaussian vector, the conditional mean is itself affine and has exactly these coefficients. Thus the best affine predictor is also the conditional mean. The proof uses zero cross-covariance to evaluate the squared-error criterion; it does not require an independence claim.

Common Mistakes and Interpretation

• Reversing a covariance block. For vector \(Y\) and predictor \(X\), the coefficient is \(\Sigma_{YX}\Sigma_{XX}^{-1}\), with dimensions \(q\times p\). The transposed block \(\Sigma_{XY}\) has the wrong dimensions for a coefficient matrix in this order.
• Using covariance alone as the slope. The cross-covariance must be adjusted by the predictor covariance matrix. When predictors are correlated or have different scales, \(\Sigma_{YX}\) alone does not give the regression coefficient.
• Confusing the conditional mean with the observed response. The formula gives the mean of the conditional distribution, not a claim that every observed \(Y\) lies on the affine function. Conditional variation is described by the Schur complement.
• Assuming the affine formula holds for every distribution. The best affine predictor has the covariance-block coefficient under finite second moments, but it need not equal the conditional mean unless additional structure, such as joint Gaussianity, makes that conditional mean affine.
• Making an independence claim from a zero covariance. The calculations above establish a covariance relationship and a conditional mean formula. Independence is a separate conclusion that requires its own argument.