Orthogonal Projections and Independent Gaussian Residuals

Why Projection Does Not Always Mean Independence

An orthogonal projection divides a vector into a part lying in a chosen subspace and a remainder perpendicular to that subspace. For ordinary Euclidean geometry, these pieces are orthogonal by construction. For random vectors, however, geometric orthogonality alone does not generally make the two random pieces independent. Their dependence is governed by the covariance matrix as well as by the projection.

For a Gaussian vector, this relationship has a particularly exact form: the projected component and its residual are independent precisely when their cross-covariance is zero. This tutorial derives that criterion, interprets it in terms of covariance-invariant subspaces, and shows how it behaves in isotropic and anisotropic examples.

Projection Matrices and Gaussian Decomposition

Definition: A matrix \(P\in\mathbb{R}^{d\times d}\) is an orthogonal projection matrix if \(P'=P\) and \(P^2=P\). It projects vectors onto its range, \(\operatorname{Range}(P)\). The complementary matrix \(I-P\) is also an orthogonal projection; its range is perpendicular to \(\operatorname{Range}(P)\).

Let \(X\) be a \(d\)-dimensional Gaussian vector with mean \(\mu\) and covariance matrix \(\Sigma\), where \(\Sigma\) is allowed to be positive semidefinite and singular. Centering at the mean is often useful, so define the projected component \(U=P(X-\mu)\) and the residual \(V=(I-P)(X-\mu)\). They satisfy \(X-\mu=U+V\). One can instead project \(X\) directly; that only adds the fixed means \(P\mu\) and \((I-P)\mu\), and does not change whether the parts are independent.

Linear images of a Gaussian vector are jointly Gaussian, including when their covariance is singular. Their covariance matrices follow by matrix multiplication:

\operatorname{Cov}(U)=P\Sigma P,\qquad \operatorname{Cov}(V)=(I-P)\Sigma(I-P),\qquad \operatorname{Cov}(U,V)=P\Sigma(I-P).

The first two expressions simplify using \(P'=P\) and \(P^2=P\), but their unsimplified forms make clear how each linear map acts on the covariance. The last expression is the key quantity: it measures covariance between the projected vector and the residual vector.

Worked Example: Mean Projection of an Isotropic Gaussian Vector

Let \(X\sim N(\mu,4I)\) in three dimensions, with \(\mu=(1,2,0)'\). Project onto the line spanned by \((1,1,1)'\). Find the projected and residual means and covariances, and determine whether the components are independent.

The unit vector along the line is \(e=(1,1,1)'/\sqrt{3}\), so the orthogonal projection matrix is

P=ee'=\frac{1}{3} \begin{pmatrix} 1&1&1\\ 1&1&1\\ 1&1&1 \end{pmatrix}, \qquad I-P=\frac{1}{3} \begin{pmatrix} 2&-1&-1\\ -1&2&-1\\ -1&-1&2 \end{pmatrix}.

Since the coordinates of \(\mu\) sum to \(3\), \(P\mu=(1,1,1)'\), and therefore \((I-P)\mu=(0,1,-1)'\). The centered covariance calculations are

\operatorname{Cov}(PX)=4P,\qquad \operatorname{Cov}((I-P)X)=4(I-P),\qquad \operatorname{Cov}(PX,(I-P)X)=4P(I-P)=0.

The zero follows from \(P(I-P)=P-P^2=0\). The two components are jointly Gaussian, and the zero cross-covariance therefore implies independence. The projected vector varies only along the line, while the residual varies only in its perpendicular plane; in this isotropic model, those random variations are independent.

The Independence Criterion

Theorem: Let \(X\) be a possibly degenerate Gaussian vector with covariance matrix \(\Sigma\), and let \(P\) be an orthogonal projection matrix. Then \(P(X-\mu)\) and \((I-P)(X-\mu)\) are independent if and only if \(P\Sigma(I-P)=0\).

The theorem is a special application of the Gaussian independence criterion: jointly Gaussian vectors are independent if and only if their cross-covariance is zero. The criterion applies to singular Gaussian vectors as well as nonsingular ones. Here is a direct justification for the projected pair.

The pair \((U,V)\) is jointly Gaussian because every linear combination of its coordinates is a linear combination of \(X-\mu\). Its joint characteristic function at vectors \(s,t\in\mathbb{R}^d\) is

E\!\left[\exp\{i(s'U+t'V)\}\right] =\exp\!\left\{-\frac12 \left(s'P\Sigma P s+2s'P\Sigma(I-P)t +t'(I-P)\Sigma(I-P)t\right)\right\}.

If \(P\Sigma(I-P)=0\), the middle term vanishes. The displayed characteristic function then factors into the characteristic function of \(U\) at \(s\) times that of \(V\) at \(t\). Factorization of the joint characteristic function is equivalent to independence, so \(U\) and \(V\) are independent. Conversely, if \(U\) and \(V\) are independent, their cross-covariance is zero: for every pair of coordinates, independence factors the expectation of the product of their centered values. Since \(\operatorname{Cov}(U,V)=P\Sigma(I-P)\), this gives the stated matrix condition. Thus the condition is both necessary and sufficient. \(\square\)

This proof uses only the Gaussian characteristic function, which is valid for positive-semidefinite covariance matrices, including singular ones. No density or matrix inverse is needed.

Worked Example: Orthogonal Subspaces That Are Not Independent

Let \(X\sim N(0,\Sigma)\) with \(\Sigma=\operatorname{diag}(1,4,9)\). Let \(P\) project onto the line spanned by \(e=(1,1,0)'/\sqrt{2}\). The projected and residual subspaces are perpendicular. Check whether their random components are independent.

Here \(P=ee'\), and the scalar coordinate of \(X\) along the projected line is \(e'X\). Its covariance with the residual vector is

\operatorname{Cov}(e'X,(I-P)X) =e'\Sigma(I-P) =\frac{1}{\sqrt{2}}(1,4,0) \begin{pmatrix} 1/2&-1/2&0\\ -1/2&1/2&0\\ 0&0&1 \end{pmatrix} =\frac{1}{\sqrt{2}}(-3/2,3/2,0).

This row vector is not zero. Equivalently,

P\Sigma(I-P)= \begin{pmatrix} -3/4&3/4&0\\ -3/4&3/4&0\\ 0&0&0 \end{pmatrix}\ne0.

Thus the projected component and residual are not independent. The Euclidean subspaces are perpendicular, but the unequal variances in the first two coordinate directions make variation along the projection statistically associated with variation in the residual direction.

Covariance-Invariant Subspaces

The matrix condition has a useful geometric interpretation. Since \(\Sigma\) is symmetric, \(P\Sigma(I-P)=0\) holds if and only if its transpose \((I-P)\Sigma P\) is zero. The latter condition says that applying \(\Sigma\) to a vector in \(\operatorname{Range}(P)\) leaves the result in \(\operatorname{Range}(P)\). In other words, that projected subspace must be invariant under the covariance matrix.

Equivalent condition: The projected and residual components of a Gaussian vector are independent if and only if \(\operatorname{Range}(P)\) is invariant under \(\Sigma\). Because \(\Sigma\) is symmetric, the perpendicular complement is then invariant as well.

This viewpoint also explains the isotropic case. If \(\Sigma=\sigma^2 I\), every subspace is invariant, so every orthogonal projection splits the centered Gaussian vector into independent projected and residual components. For a general covariance matrix, the subspace must align appropriately with the covariance structure; arbitrary Euclidean orthogonality is not enough.

Worked Example: Projection Onto a Covariance-Invariant Subspace

Let \(X\sim N(\mu,\Sigma)\), where \(\mu=(1,-1,2)'\) and \[ \Sigma= \begin{pmatrix} 2&1&0\\ 1&2&0\\ 0&0&3 \end{pmatrix}. \] Take \(P=\operatorname{diag}(1,1,0)\), the projection onto the first two coordinate directions. Compute the component means and covariances, then assess independence.

The projected and residual means are \(P\mu=(1,-1,0)'\) and \((I-P)\mu=(0,0,2)'\). Multiplying the covariance matrix by the two projections gives

P\Sigma P= \begin{pmatrix} 2&1&0\\ 1&2&0\\ 0&0&0 \end{pmatrix},\qquad (I-P)\Sigma(I-P)= \begin{pmatrix} 0&0&0\\ 0&0&0\\ 0&0&3 \end{pmatrix},\qquad P\Sigma(I-P)=0.

The first two coordinates may be correlated with each other, but they have zero covariance with the third coordinate. The Gaussian criterion therefore gives independence of the projected vector and residual. Notice that independence concerns the two blocks; it does not require coordinates within either block to be mutually independent.

Common Pitfall and Practical Use

A common mistake is to argue, “The projected subspace is perpendicular to the residual subspace, so the random components are independent.” Perpendicularity establishes \(P(I-P)=0\), an identity about the projection matrices. It does not establish \(P\Sigma(I-P)=0\), which is the relevant covariance condition. The anisotropic example shows that the latter can fail even when the subspaces are exactly perpendicular.

A reliable calculation is to identify \(P\), write down \(\Sigma\), and compute \(P\Sigma(I-P)\). If it vanishes, the Gaussian components are independent; if it does not, they are dependent. This distinction matters when decomposing Gaussian variation into fitted and residual directions, separating signal subspaces from their complements, or constructing independent Gaussian components. It is specifically the Gaussian assumption that turns zero cross-covariance into independence; for arbitrary random vectors, zero cross-covariance alone is not sufficient.

Key takeaway: Orthogonal projection guarantees a geometric decomposition, not statistical independence. For a Gaussian vector, projected and residual components are independent exactly when the projection separates the covariance matrix, \(P\Sigma(I-P)=0\).

Check Your Understanding

Work through these questions using the projection and covariance criteria.

What two matrix identities define an orthogonal projection, and what is the complementary projection?
For a Gaussian vector with covariance \(\Sigma\), write the cross-covariance between \(P(X-\mu)\) and \((I-P)(X-\mu)\).
Why does the criterion \(P\Sigma(I-P)=0\) imply independence for a Gaussian vector, even if its covariance matrix is singular?
If \(\Sigma=\sigma^2 I\), why are the projected and residual components independent for every orthogonal projection?
Can perpendicular projected and residual subspaces still yield dependent Gaussian components? Explain what matrix calculation decides.