Orthogonal Decomposition of Gaussian Samples

Why Project a Gaussian Sample?

A sample of $n$ multivariate observations contains two different kinds of variation: the overall location of the sample and the deviations of individual observations from that location. A projection onto the constant-vector direction extracts the location component; the orthogonal projection extracts the centered deviations. For a Gaussian sample, these two components are not merely uncorrelated: they are independent.

The key geometric idea is that these projections act on the observation index, not on the coordinates within each observation. The sample mean is repeated in all $n$ rows of the projected matrix, whereas the residual rows sum to zero.

Projection Matrices for the Mean and Residuals

Let $X$ be an $n\times p$ data matrix whose rows $X_1',\ldots,X_n'$ are independent and identically distributed as $N_p(\mu,\Sigma)$, where $\Sigma$ is positive definite. Write $\mathbf 1$ for the $n$-vector of ones, and define the sample mean vector by $\bar X=n^{-1}\sum_{i=1}^n X_i$.

Definition: The projection onto the constant-vector direction in observation space and its complementary centering projection are $$ P=\frac{1}{n}\mathbf 1\mathbf 1',\qquad M=I_n-P. $$ The projected mean component and centered residual component of the sample are $PX$ and $MX$, respectively.

Both matrices are symmetric and idempotent: $P'=P$, $P^2=P$, $M'=M$, and $M^2=M$. They are orthogonal projections because $PM=MP=0$, and they add to the identity, $P+M=I_n$. Since $P\mathbf 1=\mathbf 1$, direct multiplication gives

PX=\mathbf 1\bar X',\qquad MX=X-\mathbf 1\bar X'.

Thus every row of $PX$ equals $\bar X'$. Each column of $MX$ sums to zero, because $\mathbf 1'M=0$. The exact sample decomposition is

X=PX+MX=\mathbf 1\bar X'+(X-\mathbf 1\bar X').

Worked Example: Decomposing Four Observations

Consider the four observed row vectors $(2,1),(4,3),(6,5),(8,7)$. Here $n=4$, and

\bar X=(5,4)',\quad P=\frac14\begin{pmatrix}1&1&1&1\\1&1&1&1\\1&1&1&1\\1&1&1&1\end{pmatrix}.

The mean component repeats $(5,4)$ four times. Subtracting it from each row gives

PX=\begin{pmatrix}5&4\\5&4\\5&4\\5&4\end{pmatrix},\qquad MX=\begin{pmatrix}-3&-3\\-1&-1\\1&1\\3&3\end{pmatrix}.

For either column, the residuals sum to zero: $-3-1+1+3=0$. Adding the two components recovers every original row. This example illustrates the algebraic decomposition; it does not by itself assert anything probabilistic about these particular observations.

Covariances Under the Gaussian Sampling Model

For the independence result, it is useful to specify the vectorization convention. In this tutorial, $\operatorname{vec}(X)$ stacks the columns of $X$ in order. Under the sampling model above, $X$ is matrix normal with row covariance $I_n$ and column covariance $\Sigma$, so

E[X]=\mathbf 1\mu',\qquad \operatorname{Cov}(\operatorname{vec}(X))=\Sigma\otimes I_n.

The order of the Kronecker factors follows from column-wise stacking: the covariance between columns $j$ and $k$ is $\Sigma_{jk}I_n$. For any fixed $n\times n$ matrix $A$, column-wise vectorization satisfies $\operatorname{vec}(AX)=(I_p\otimes A)\operatorname{vec}(X)$.

Theorem: If the rows of $X$ are independent $N_p(\mu,\Sigma)$ and $P,M$ are the projections defined above, then $$ E[PX]=\mathbf 1\mu',\qquad E[MX]=0, $$ $$ \operatorname{Cov}(\operatorname{vec}(PX))=\Sigma\otimes P,\qquad \operatorname{Cov}(\operatorname{vec}(MX))=\Sigma\otimes M, $$ and $PX$ and $MX$ are independent. These statements include the case $n=1$, when $MX=0$ with probability one.

Proof. The means follow by linearity: $E[PX]=P\mathbf 1\mu'=\mathbf 1\mu'$, while $E[MX]=M\mathbf 1\mu'=0$. Let $U=\operatorname{vec}(PX)$ and $V=\operatorname{vec}(MX)$. Both are linear transformations of the jointly Gaussian vector $\operatorname{vec}(X)$, so their joint distribution is Gaussian, possibly singular. By the vectorization identity,

U=(I_p\otimes P)\operatorname{vec}(X),\qquad V=(I_p\otimes M)\operatorname{vec}(X).

Using $\operatorname{Cov}(A Z)=A\operatorname{Cov}(Z)A'$, the symmetry and idempotence of $P,M$, and the mixed-product rule for Kronecker products gives

\begin{aligned} \operatorname{Cov}(U) &=(I_p\otimes P)(\Sigma\otimes I_n)(I_p\otimes P) =\Sigma\otimes P,\\ \operatorname{Cov}(V) &=(I_p\otimes M)(\Sigma\otimes I_n)(I_p\otimes M) =\Sigma\otimes M,\\ \operatorname{Cov}(U,V) &=(I_p\otimes P)(\Sigma\otimes I_n)(I_p\otimes M) =\Sigma\otimes PM=0. \end{aligned}

To conclude independence, consider the joint characteristic function of $(U,V)$. For a jointly Gaussian pair with means $m_U,m_V$, covariance blocks $C_U,C_V$, and cross-covariance zero, its value at vectors $s,t$ is

\exp\!\left(i s'm_U+i t'm_V-\frac12s'C_Us-\frac12t'C_Vt\right).

This factors into the characteristic function of $U$ at $s$ and that of $V$ at $t$. Factorization of the joint characteristic function implies independence. This argument remains valid for singular Gaussian distributions, so no density for $U$ or $V$ is required. Therefore $PX$ and $MX$ are independent. $\square$

What the Covariance Formulas Say

The repeated mean component has only one direction of variation across the observations: every column is proportional to $\mathbf 1$. In particular, the theorem also yields the familiar distribution of the sample mean by taking any row of $PX$:

\bar X\sim N_p\left(\mu,\frac{\Sigma}{n}\right).

The residual component, by contrast, lies in the $n-1$ dimensional subspace perpendicular to $\mathbf 1$, provided $n>1$. Its rows are generally not independent. Indeed, the covariance between residual rows $i$ and $j$, viewed as $p$-vectors, is $M_{ij}\Sigma$. Their dependence is consistent with the exact constraint that the residual rows sum to zero.

The Kronecker order also records the vectorization convention. The displayed covariance formulas use column-wise stacking. If instead the rows were stacked consecutively, the corresponding covariance matrices would be $P\otimes\Sigma$ and $M\otimes\Sigma$. In either convention, the blocks associated with two observation indices must reflect their covariance multiplier from the projection matrix.

Worked Example: Independent Mean and Contrasts in a Scalar Sample

Let $Z_1,Z_2,Z_3$ be independent $N(2,4)$ variables, and write $z=(Z_1,Z_2,Z_3)'$. Define three coordinates

A=\frac{Z_1+Z_2+Z_3}{\sqrt3},\quad B=\frac{Z_1-Z_2}{\sqrt2},\quad C=\frac{Z_1+Z_2-2Z_3}{\sqrt6}.

The coefficient vectors are orthonormal: each has squared length one, and every pair has inner product zero. Thus the transformation from $z$ to $(A,B,C)'$ is orthogonal. Since $E[z]=(2,2,2)'$ and $\operatorname{Cov}(z)=4I_3$, the transformed vector has mean $(2\sqrt3,0,0)'$ and covariance $4I_3$. It is Gaussian, so its zero cross-covariances imply that $A,B,C$ are mutually independent.

The first coordinate is $A=\sqrt3\,\bar Z$, so it contains the mean direction. The other two coordinates are contrasts: their coefficient vectors sum to zero. Consequently the centered vector is $Mz=q_1B+q_2C$, where $q_1=(1,-1,0)'/\sqrt2$ and $q_2=(1,1,-2)'/\sqrt6$. The pair $(B,C)$, and hence the residual vector, is independent of the sample mean coordinate $A$.

Worked Example: Checking Covariance Blocks for Two Bivariate Observations

Suppose $n=2$, $p=2$, $\mu=(1,-1)'$, and

\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix},\quad P=\frac12\begin{pmatrix}1&1\\1&1\end{pmatrix},\quad M=\frac12\begin{pmatrix}1&-1\\-1&1\end{pmatrix}.

For column-wise vectorization, the coordinate order is $(X_{11},X_{21},X_{12},X_{22})'$. The mean component has expectation $\mathbf 1\mu'$, and the residual component has expectation zero. Their covariance matrices are

\operatorname{Cov}(\operatorname{vec}(PX)) =\Sigma\otimes P =\begin{pmatrix} 2&2&1/2&1/2\\ 2&2&1/2&1/2\\ 1/2&1/2&1&1\\ 1/2&1/2&1&1 \end{pmatrix}, $$ $$ \operatorname{Cov}(\operatorname{vec}(MX)) =\Sigma\otimes M =\begin{pmatrix} 2&-2&1/2&-1/2\\ -2&2&-1/2&1/2\\ 1/2&-1/2&1&-1\\ -1/2&1/2&-1&1 \end{pmatrix}.

For example, the first $2\times2$ block of the first matrix is $4P$, while the upper-right block is $1P$; this matches the observation-index covariance $P$ and the coordinate covariance $\Sigma$. The cross-covariance is $\Sigma\otimes PM=0$, so the mean and residual matrices are independent even though the residual covariance is singular.

Common Mistakes and Why the Decomposition Matters

Assuming uncorrelated always means independent. Zero covariance alone does not generally establish independence. Here the implication holds because the two components are jointly Gaussian.
Reversing the Kronecker factors without changing the convention. With column-wise vectorization, the covariance matrices are $\Sigma\otimes P$ and $\Sigma\otimes M$. With row-wise stacking, the factors appear in the opposite order.
Calling the residual rows independent. The residual rows sum to zero, and their covariance is $M_{ij}\Sigma$; they are generally dependent even though the entire residual matrix is independent of the mean component.
Expecting a nonsingular residual distribution. The residuals are constrained to an $(n-1)$-dimensional subspace of the $n$-dimensional observation space. For $n>1$, this makes their joint Gaussian distribution singular as a vector of $np$ entries.

This decomposition matters because it isolates two sources of sample variation in a form that makes their relationship exact. The mean component describes location, while the residual component records deviations that cannot change that location. Their independence is a special and useful consequence of Gaussianity combined with orthogonality in observation space.

Check Your Understanding

Use the projection identities and Gaussian covariance argument to answer the following.

Why does every row of $PX$ equal the sample mean vector?
Show that $\mathbf 1'M=0$, and explain what this implies about the column sums of $MX$.
For column-wise vectorization, why is the cross-covariance between $\operatorname{vec}(PX)$ and $\operatorname{vec}(MX)$ equal to zero?
Why does zero cross-covariance imply independence in this setting, and what assumption is essential?
Are the residual rows generally independent of one another? Use their sum constraint or their covariance formula to justify your answer.