Orthogonal Gaussian Projections and Independent Components

Gaussian Variation Along Orthogonal Directions

A standard Gaussian vector has no preferred direction: rotating its coordinate system leaves its distribution unchanged. This symmetry makes orthogonal projection especially useful. When a standard Gaussian vector is split into its part inside a subspace and its part perpendicular to that subspace, the two pieces are not merely uncorrelated; they are independent. This gives a geometric route to understanding independent Gaussian components and the fitted-value/residual decomposition in linear regression.

The result concerns projections onto orthogonal subspaces. The components need not have densities in the full ambient space: a projection onto a lower-dimensional subspace is concentrated entirely on that subspace. The appropriate distributional description uses coordinates within each subspace.

Definition: A matrix $P\in\mathbb{R}^{p\times p}$ is an orthogonal projection matrix if $P=P^\mathsf{T}$ and $P^2=P$. It projects vectors onto its range $\mathcal{S}=\{Pv:v\in\mathbb{R}^p\}$. The complementary projection is $I_p-P$; its range is the orthogonal complement $\mathcal{S}^\perp$. The two ranges are perpendicular, and every $v\in\mathbb{R}^p$ has the decomposition $v=Pv+(I_p-P)v$.

A standard Gaussian vector $Z\sim N_p(0,I_p)$ has density $(2\pi)^{-p/2}\exp(-z^\mathsf{T}z/2)$ on $\mathbb{R}^p$. If $P$ has rank $r$, then $PZ$ lies in an $r$-dimensional subspace, while $(I_p-P)Z$ lies in its $(p-r)$-dimensional orthogonal complement. Their covariance matrices are $P$ and $I_p-P$, respectively; these matrices may be singular when the corresponding subspace has dimension less than $p$.

The Independent-Projection Theorem

Theorem: Let $Z\sim N_p(0,I_p)$, and let $P$ be a fixed orthogonal projection matrix. Then $PZ$ and $(I_p-P)Z$ are independent. In orthonormal coordinates adapted to the range of $P$, the first component has $r=\operatorname{rank}(P)$ independent $N(0,1)$ coordinates and zero remaining coordinates; the second has zero coordinates in those $r$ directions and $p-r$ independent $N(0,1)$ coordinates. Also, $$ E[PZ]=0,\qquad \operatorname{Cov}(PZ)=P,\qquad E[(I_p-P)Z]=0,\qquad \operatorname{Cov}((I_p-P)Z)=I_p-P. $$

Proof. Since $P$ is symmetric and idempotent, its eigenvalues are either 0 or 1: if $Pv=\lambda v$ for a nonzero eigenvector, then $P^2v=Pv$ implies $\lambda^2=\lambda$. The spectral theorem therefore gives an orthogonal matrix $Q$ such that $$ Q^\mathsf{T}PQ=\begin{pmatrix}I_r&0\\0&0\end{pmatrix}. $$ Let $W=Q^\mathsf{T}Z$. The density of $W$ is the density of $Z$: the change of coordinates preserves $z^\mathsf{T}z$, and the absolute determinant of $Q$ is 1. Hence $W\sim N_p(0,I_p)$, whose coordinates are independent standard normal variables. In these coordinates, $$ Q^\mathsf{T}PZ=\begin{pmatrix}W_1\\ \vdots\\ W_r\\0\\ \vdots\\0\end{pmatrix}, \qquad Q^\mathsf{T}(I_p-P)Z=\begin{pmatrix}0\\ \vdots\\0\\W_{r+1}\\ \vdots\\W_p\end{pmatrix}. $$ The first vector is a function only of $W_1,\ldots,W_r$, and the second only of $W_{r+1},\ldots,W_p$. Those coordinate groups are independent, so the vectors are independent. Multiplication by the fixed matrix $Q$ preserves independence, giving the claim in the original coordinates. For the moments, $E[Z]=0$ gives $E[PZ]=0$. Using $\operatorname{Cov}(AZ)=A\operatorname{Cov}(Z)A^\mathsf{T}$ and $P^\mathsf{T}=P$, $$ \operatorname{Cov}(PZ)=P I_p P^\mathsf{T}=P^2=P. $$ Applying the same calculation to $I_p-P$, which is also symmetric and idempotent, gives the other covariance formula. $\square$

Worked Example: Projection onto a Line in Two Dimensions

Let $Z=(Z_1,Z_2)^\mathsf{T}\sim N_2(0,I_2)$, and project onto the line spanned by $u=(1,1)^\mathsf{T}/\sqrt{2}$. The projection matrix is $$ P=uu^\mathsf{T}=\frac12\begin{pmatrix}1&1\\1&1\end{pmatrix}, \qquad I_2-P=\frac12\begin{pmatrix}1&-1\\-1&1\end{pmatrix}. $$

The unit vector $v=(1,-1)^\mathsf{T}/\sqrt{2}$ spans the perpendicular line. Thus $$ PZ=Uu,\quad U=u^\mathsf{T}Z=\frac{Z_1+Z_2}{\sqrt{2}}, \qquad (I_2-P)Z=Vv,\quad V=v^\mathsf{T}Z=\frac{Z_1-Z_2}{\sqrt{2}}. $$

The change from $(Z_1,Z_2)$ to $(U,V)$ is orthogonal. Consequently $(U,V)^\mathsf{T}\sim N_2(0,I_2)$, so $U$ and $V$ are independent $N(0,1)$ variables. The projected vectors are therefore independent as well. Their covariance matrices are $P$ and $I_2-P$; neither projected vector has a nonsingular density over all of $\mathbb{R}^2$, because each is confined to a line.

Coordinates Within a Subspace

If the columns of $A\in\mathbb{R}^{p\times r}$ are orthonormal, then $P=AA^\mathsf{T}$ projects onto their span. For $Z\sim N_p(0,I_p)$, the vector of coordinates along those directions is $A^\mathsf{T}Z\sim N_r(0,I_r)$, and $PZ=A(A^\mathsf{T}Z)$. The remaining component is $(I_p-AA^\mathsf{T})Z$. The theorem says these two vector components are independent. This coordinate description makes it possible to work with a lower-dimensional Gaussian vector even when the projected vector itself is singular in the ambient space.

Worked Example: Projection onto a Plane in Three Dimensions

Let $Z\sim N_3(0,I_3)$. Consider the vectors $$ a=\frac{1}{\sqrt{2}}(1,1,0)^\mathsf{T}, \qquad b=\frac{1}{\sqrt{6}}(1,-1,2)^\mathsf{T}. $$ Their squared lengths are $1$, and $a^\mathsf{T}b=(1-1+0)/\sqrt{12}=0$. They are therefore an orthonormal basis for a plane. The unit vector $c=(1,-1,-1)^\mathsf{T}/\sqrt{3}$ is perpendicular to both: $a^\mathsf{T}c=0$ and $b^\mathsf{T}c=(1+1-2)/\sqrt{18}=0$.

The projection onto the plane is $P=aa^\mathsf{T}+bb^\mathsf{T}=I_3-cc^\mathsf{T}$. The two coordinates within the plane are $$ U=a^\mathsf{T}Z=\frac{Z_1+Z_2}{\sqrt{2}}, \qquad V=b^\mathsf{T}Z=\frac{Z_1-Z_2+2Z_3}{\sqrt{6}}, $$ and the coordinate perpendicular to it is $W=c^\mathsf{T}Z=(Z_1-Z_2-Z_3)/\sqrt{3}$. The matrix with rows $a^\mathsf{T},b^\mathsf{T},c^\mathsf{T}$ is orthogonal, so $(U,V,W)^\mathsf{T}\sim N_3(0,I_3)$. In particular, $U,V,W$ are mutually independent $N(0,1)$ variables.

The two projected vectors are $PZ=Ua+Vb$ and $(I_3-P)Z=Wc$. The plane component uses $U,V$, while the perpendicular component uses $W$; thus the components are independent. Notice that the plane projection has two independent Gaussian coordinates, not three: its ambient three-dimensional covariance $P$ has rank 2.

Regression as an Orthogonal Decomposition

Least squares uses the same geometry. Let $D\in\mathbb{R}^{n\times q}$ be a fixed design matrix with full column rank, and let $\mathcal{C}(D)$ denote its column space. Its projection matrix is $$ P_D=D(D^\mathsf{T}D)^{-1}D^\mathsf{T}, \qquad M_D=I_n-P_D. $$ Here $P_Dy$ is the least-squares fitted vector for an observed response $y$, and $M_Dy$ is its residual vector. The fitted vector belongs to $\mathcal{C}(D)$; the residual is perpendicular to every column of $D$, since $D^\mathsf{T}M_D=0$.

Theorem: Suppose $Y=D\beta+\varepsilon$, where $D$ is fixed with full column rank, $\beta\in\mathbb{R}^q$ is fixed, and $\varepsilon\sim N_n(0,\sigma^2I_n)$ for $\sigma^2>0$. Then $P_DY$ and $M_DY$ are independent. The least-squares coefficient estimate $\hat{\beta}=(D^\mathsf{T}D)^{-1}D^\mathsf{T}Y$ is also independent of the residual vector $M_DY$.

Proof. The matrices $P_D$ and $M_D$ are complementary orthogonal projections. Write $\varepsilon=\sigma Z$, where $Z\sim N_n(0,I_n)$. Then $$ P_DY=D\beta+\sigma P_DZ,\qquad M_DY=\sigma M_DZ, $$ because $P_DD=D$ and $M_DD=0$. The independent-projection theorem shows that $P_DZ$ and $M_DZ$ are independent. Adding a fixed vector to one component and multiplying both by the fixed scalar $\sigma$ preserves independence, so $P_DY$ and $M_DY$ are independent. Furthermore, $\hat{\beta}=(D^\mathsf{T}D)^{-1}D^\mathsf{T}P_DY$. Thus $\hat{\beta}$ is a function of $P_DY$, and is independent of $M_DY$. $\square$

Worked Example: Fitted Trend and Residual Direction

Take three observations at predictor values $-1,0,1$, with design matrix $D$ having columns $\mathbf{1}=(1,1,1)^\mathsf{T}$ and $x=(-1,0,1)^\mathsf{T}$. These columns are perpendicular because $\mathbf{1}^\mathsf{T}x=0$, and their squared lengths are 3 and 2. A unit vector perpendicular to both is $c=(1,-2,1)^\mathsf{T}/\sqrt{6}$. Hence the residual projection is $M_D=cc^\mathsf{T}$, and $P_D=I_3-cc^\mathsf{T}$.

For the observed vector $y=(2,1,5)^\mathsf{T}$, the fitted intercept and slope are $$ \hat{\beta}_0=\frac{\mathbf{1}^\mathsf{T}y}{3}=\frac{8}{3}, \qquad \hat{\beta}_1=\frac{x^\mathsf{T}y}{2}=\frac{-2+5}{2}=\frac{3}{2}. $$ The fitted vector is $(8/3)\mathbf{1}+(3/2)x=(7/6,8/3,25/6)^\mathsf{T}$, and the residual vector is $$ y-P_Dy=\left(\frac56,-\frac53,\frac56\right)^\mathsf{T} =\frac{5}{\sqrt{6}}c. $$ Its inner products with the design columns are $5/6+(-5/3)+5/6=0$ and $-5/6+5/6=0$, confirming the orthogonality directly.

Now suppose the responses are random, $Y=D\beta+\varepsilon$, with $\varepsilon\sim N_3(0,\sigma^2I_3)$. The fitted vector depends on Gaussian error coordinates along the intercept and trend directions; the residual depends only on the coordinate along $c$. Those coordinates are independent, so the fitted vector and residual are independent. The numerical vector above illustrates the geometry; independence is a statement about the distribution across repeated random responses, not about a single observed decomposition.

Why the Geometry Matters—and a Common Pitfall

Orthogonal projection separates variation into components that can be analyzed without modeling dependence between them when the underlying vector is standard Gaussian. In regression, this explains why fitted values and residuals are independent under the stated spherical Gaussian error model. The conclusion relies on both orthogonality and Gaussianity, as well as on the specified covariance structure.

Pitfall: treating zero covariance as sufficient for independence. Orthogonal projections have zero cross-covariance, but zero covariance alone does not imply independence for general random variables.
Correction: Here the projected components are jointly Gaussian because they are linear transformations of a Gaussian vector. For jointly Gaussian vectors, zero cross-covariance implies independence, including when a covariance matrix is singular. The projection proof above establishes independence directly from the independent Gaussian coordinates.
Pitfall: assuming a projection has a full-dimensional density. A projection onto a proper subspace is concentrated on that subspace and has a singular covariance matrix in the ambient coordinates.
Correction: Describe its distribution using coordinates in an orthonormal basis for its range. Those coordinates are ordinary independent standard normals.

The regression conclusion also depends on the error covariance being $\sigma^2I_n$. If the errors have a different covariance structure, perpendicularity in the ordinary Euclidean geometry does not by itself yield independent projected components. The projection theorem applies directly to standard Gaussian vectors; other Gaussian settings require the relevant covariance structure to be accounted for.

Check Your Understanding

Use the definitions and projection theorem to answer the following.

Why must the eigenvalues of an orthogonal projection matrix be either 0 or 1?
If $P$ has rank $r$, how many independent standard normal coordinates describe $PZ$ within its range?
For $Z\sim N_p(0,I_p)$, what are the covariance matrices of $PZ$ and $(I_p-P)Z$?
In least squares with fixed full-rank design and errors $N_n(0,\sigma^2I_n)$, why are the fitted vector and residual independent?
Why does zero covariance between two arbitrary random vectors not suffice to conclude independence?