Gaussian Regression Residuals and Independence

Introduction

In Gaussian regression, a response is decomposed into a part explained linearly by a predictor block and a remaining residual. For jointly Gaussian variables, this decomposition has a strong property: the residual is not merely uncorrelated with the predictors; it is independent of them. That property makes the linear predictor especially useful for understanding conditional distributions.

The construction depends only on the means and covariance blocks of the joint Gaussian vector. It also works when the predictor covariance matrix is singular, as can happen when predictors are exact linear combinations of one another. We will derive the predictor and residual covariance, and prove independence directly for this construction.

Definition of the Gaussian Regression Residual

Let \(Y\in\mathbb{R}^q\) be a response vector and \(X\in\mathbb{R}^p\) a predictor vector, with \((Y,X)\) jointly Gaussian. Write their means and covariance blocks as

The matrices \(\Sigma_{YY}\) and \(\Sigma_{XX}\) are symmetric and positive semidefinite. If \(\Sigma_{XX}\) is invertible, the regression coefficient matrix is \(\Sigma_{YX}\Sigma_{XX}^{-1}\). For a possibly singular \(\Sigma_{XX}\), use its Moore–Penrose inverse, denoted \(\Sigma_{XX}^{+}\).

When \(q=1\), the response is scalar and \(\Sigma_{YX}\) is a row vector. The predictor is then an intercept plus a weighted sum of the centered predictors. The formula uses centered variables so that the predictor has mean \(\mu_Y\); the intercept in an uncentered expression is \(\mu_Y-\Sigma_{YX}\Sigma_{XX}^{+}\mu_X\).

Why the Formula Works, Including in the Singular Case

The singular case requires one compatibility fact: the cross-covariance with \(Y\) cannot point in a direction where \(X\) has zero variance. To see this, suppose \(v\in\mathbb{R}^p\) satisfies \(\Sigma_{XX}v=0\). Then

A random variable with zero variance equals its mean almost surely, so \(v^{\mathsf T}(X-\mu_X)=0\) almost surely. It follows that \(\operatorname{Cov}(Y,v^{\mathsf T}X)=\Sigma_{YX}v=0\). Thus every vector in the null space of \(\Sigma_{XX}\) is also in the null space of \(\Sigma_{YX}\). Since \(\Sigma_{XX}\) is symmetric, its row space is the orthogonal complement of its null space. Consequently, the rows of \(\Sigma_{YX}\) lie in the row space of \(\Sigma_{XX}\), and

Deriving the Residual Covariance and Independence

Set \(A=\Sigma_{YX}\Sigma_{XX}^{+}\) and \(X_c=X-\mu_X\). Then \(R=(Y-\mu_Y)-AX_c\), so \(E[R]=0\). Using the compatibility identity above,

Expanding the covariance of \(R\) gives

For the second equality, \(\Sigma_{XX}^{+}\) is symmetric, and \(A\Sigma_{XX}=\Sigma_{YX}\); hence the last two terms involving \(A\) combine to subtract one copy of \(\Sigma_{YX}\Sigma_{XX}^{+}\Sigma_{XY}\). In particular, \(\Omega\) is positive semidefinite: it is the covariance matrix of \(R\).

To establish independence, note that \(R\) and \(X_c\) are affine transformations of the jointly Gaussian vector \((Y,X)\). They are therefore jointly Gaussian, including when their joint covariance is singular. Their cross-covariance is zero, as just calculated. For any \(t\in\mathbb{R}^q\) and \(s\in\mathbb{R}^p\), their joint characteristic function is

The final expression is the product of the marginal characteristic functions of \(R\) and \(X_c\). The factorization holds for all \(t,s\), so uniqueness of characteristic functions implies that \(R\) and \(X_c\), and therefore \(R\) and \(X\), are independent.

Finally, \(Y=\widehat Y_{\mathrm{lin}}+R\), where \(R\) is independent of \(X\) and has mean zero. Taking conditional expectations gives the stated formula for \(E[Y\mid X]\). The conditional covariance is the covariance of \(R\), namely \(\Omega\), because the predictor is determined by \(X\). The characteristic-function argument also shows that the residual has the same Gaussian distribution conditionally on \(X\) as it has marginally. This proves the theorem. \(\square\)

Worked Examples

Interpretation and a Common Pitfall

The residual covariance \(\Omega\) measures the response variation left after the predictor block has been used. Its value is no larger than the original response covariance in the positive-semidefinite sense: the removed term \(\Sigma_{YX}\Sigma_{XX}^{+}\Sigma_{XY}\) is positive semidefinite. When the predictor covariance is nonsingular and the response is scalar, this reduces to the familiar variance subtraction \(\operatorname{Var}(Y)-\Sigma_{YX}\Sigma_{XX}^{-1}\Sigma_{XY}\).

A frequent mistake is to argue that zero covariance alone proves independence. In general, that is false: uncorrelated random variables need not be independent. The correction here is specific and important. The residual and predictors are jointly Gaussian, and the characteristic-function factorization proves their independence. Without joint Gaussianity, the same linear predictor can have an uncorrelated residual without an independent one.

Another potential confusion is treating the coefficients as uniquely determined when the predictors are linearly dependent. With singular \(\Sigma_{XX}\), coefficient representations can differ, but the fitted predictor is unique almost surely: any difference between valid coefficient matrices has zero variance when applied to \(X-\mu_X\). It is the prediction and its residual covariance, rather than a particular coefficient representation, that carry the statistical content.