Independent Residuals in Gaussian Regression

Why the Regression Residual Matters

In Gaussian regression, the conditional mean gives the best predictor of a response from the observed predictors. Subtracting that mean from the response leaves a residual. The important Gaussian fact is stronger than saying this residual has mean zero: it is independent of the predictors. That fact lets us represent the response as a predictable linear part plus fresh Gaussian variation, and it gives a useful factorization of the joint law.

We consider a response vector \(Y\in\mathbb{R}^q\) and a predictor vector \(X\in\mathbb{R}^p\). A scalar response is the special case \(q=1\). Assume \((Y,X)\) is jointly Gaussian, with means \(\mu_Y\) and \(\mu_X\), and covariance blocks

We assume \(\Sigma_{XX}\) is positive definite, so it is invertible. The full joint covariance need only be positive semidefinite; the residual covariance below can be singular.

For a jointly Gaussian pair with \(\Sigma_{XX}\) positive definite, the conditional distribution is

Here \(N_q(a,V)\) denotes a \(q\)-dimensional Gaussian distribution with mean vector \(a\) and covariance matrix \(V\), including the possibility that \(V\) is singular. The formula says that, once \(X=x\) is fixed, the conditional mean is exactly \(m(x)\), while the conditional covariance is the same matrix \(S\) for every \(x\).

Independence of the Residual and Predictors

The proof uses the conditional distribution directly. It is important to do this rather than infer independence merely from a covariance calculation; independence does not follow from zero covariance for arbitrary distributions.

Proof. Because \(m(x)\) is a deterministic function of \(x\), the conditional distribution of \(\varepsilon\) given \(X=x\) is the distribution of \(Y-m(x)\) given \(X=x\). By the conditional Gaussian formula,

This holds as a conditional distribution for \(X\)-almost every \(x\); it also describes the usual Gaussian conditional kernel for every \(x\). In particular, for every Borel set \(A\subseteq\mathbb{R}^q\), \(P(\varepsilon\in A\mid X)=\nu(A)\) almost surely, where \(\nu\) is the fixed probability law \(N_q(0,S)\). For any Borel set \(C\subseteq\mathbb{R}^p\), the defining property of conditional expectation gives

The last equality follows because averaging the constant conditional probability gives \(P(\varepsilon\in A)=\nu(A)\). The product identity for all such Borel sets proves independence, and the conditional law also gives \(\varepsilon\sim N_q(0,S)\). \(\square\)

Factoring the Joint Law

The theorem gives a product representation of the joint law without requiring a density. Let \(P_X\) be the law of \(X\), and let \(\nu=N_q(0,S)\). Since \(X\) and \(\varepsilon\) are independent, their joint law is the product measure \(P_X\otimes\nu\). The pair \((X,Y)\) is obtained by applying the map \((x,e)\mapsto (x,m(x)+e)\) to this product law. Equivalently, for Borel sets \(C\subseteq\mathbb{R}^p\) and \(D\subseteq\mathbb{R}^q\),

where \(D-m(x)=\{y-m(x):y\in D\}\). This is the joint-law factorization into the predictor distribution and an independent residual distribution, followed by the regression map.

If \(S\) is positive definite, both \(X\) and the residual have densities. The change of variables from \((x,e)\) to \((x,y)=(x,m(x)+e)\) has Jacobian determinant \(1\): its derivative is block triangular with identity matrices on the diagonal. Thus the joint density factors as

This is also the familiar conditional-density factorization \(f_{X,Y}(x,y)=f_X(x)f_{Y\mid X}(y\mid x)\), with the conditional density written as a fixed residual density shifted by \(m(x)\). If \(S\) is singular, a Lebesgue density for the residual may not exist, but the product-measure statement and the probability factorization remain valid.

What the Result Does—and Does Not—Say

The result matters because it gives an exact probability model, not just a convenient prediction formula. Conditional on the predictors, all remaining randomness in \(Y\) is represented by a residual whose distribution does not change with the predictor value. Conversely, the response is recovered by adding that independent residual to its conditional mean. This representation is useful for deriving joint probabilities, simulating Gaussian regression models, and separating systematic variation from residual variation.

The scope of the conclusion is precise. It applies to a jointly Gaussian population vector with an invertible predictor covariance matrix. The independence proved here is between the population residual \(Y-m(X)\) and the random predictor vector \(X\). It is not, by itself, a claim that residuals computed after fitting a regression to a finite sample are independent of the observed design, nor that the result holds for non-Gaussian variables.

Common Pitfall

• Wrong reasoning: “The residual has zero covariance with \(X\), so it must be independent of \(X\).” Zero covariance alone does not establish independence in a general distribution.
• Correction: Use the joint Gaussian assumption and the conditional distribution. Subtracting the conditional mean makes the conditional law of the residual \(N_q(0,S)\), the same for every predictor value. The resulting constant conditional law proves independence by the product-probability identity.
• Wrong interpretation: “The residual is independent because it is defined as an error.” The definition alone does not confer independence; the Gaussian model and conditional-mean construction are essential.