Orthogonal Projections And Cochran's Wishart Theorem

Why Orthogonal Projections Produce Wishart Matrices

A Gaussian data matrix has many possible sums of squares. Some summarize variation around a fitted component; others summarize variation left in a residual component. When the corresponding sample-space subspaces are orthogonal, these summaries have a particularly useful structure: their random components are independent, and each sum of squares has a Wishart distribution. The multivariate version of Cochran's theorem makes this relationship precise.

We use $Y$ for an $n\times p$ data matrix, with observations in rows. Its rows are independent $p$-variate normal vectors with common covariance matrix $\Sigma$. A projection acts on the rows, so the associated sum-of-squares matrix is of the form $Y'PY$, a $p\times p$ matrix.

Definition: A matrix $P\in\mathbb R^{n\times n}$ is an orthogonal projection if $P'=P$ and $P^2=P$. Its range is the subspace onto which it projects, and its rank is the dimension of that subspace. Two orthogonal projections $P_i,P_j$ have orthogonal ranges exactly when $P_iP_j=0$.

For an integer $\nu\geq 0$, $W_p(\nu,\Sigma)$ denotes the distribution of a sum $\sum_{a=1}^{\nu}V_aV_a'$, where the $V_a$ are independent $N_p(0,\Sigma)$ vectors. The sum is the zero matrix when $\nu=0$. When $\nu<p$, this distribution is supported on singular matrices; the notation does not imply that a density exists.

The Multivariate Cochran Theorem

First take a centered Gaussian data matrix $Y$ whose rows are independent $N_p(0,\Sigma)$ vectors, where $\Sigma$ is positive definite. The theorem concerns a collection of projections that splits the whole sample space into mutually orthogonal parts.

Theorem (Cochran's Wishart theorem): Let $P_1,\ldots,P_k$ be symmetric idempotent $n\times n$ matrices such that $P_iP_j=0$ for $i\ne j$ and $\sum_{j=1}^kP_j=I_n$. Write $\nu_j=\operatorname{rank}(P_j)$. If the rows of $Y$ are independent $N_p(0,\Sigma)$, then $$ Y'P_jY\sim W_p(\nu_j,\Sigma),\qquad j=1,\ldots,k, $$ and the matrices $Y'P_1Y,\ldots,Y'P_kY$ are mutually independent.

The key is to examine the projected Gaussian data themselves, not only their sums of squares. To see this, write $Y=Z\Sigma^{1/2}$, where the entries of $Z$ are independent $N(0,1)$ variables and $\Sigma^{1/2}$ is a symmetric square root of $\Sigma$. Choose an orthonormal basis of $\mathbb R^n$ by first taking bases for the ranges of the $P_j$. If $U$ is the orthogonal matrix with these basis vectors as columns, then each $P_j$ becomes a diagonal block projection in these coordinates: $U'P_jU$ has an identity block of size $\nu_j$ and zeros elsewhere.

The matrix $U'Z$ has the same distribution as $Z$. Indeed, each column of $Z$ has density proportional to $\exp(-z'z/2)$; an orthogonal change of coordinates preserves $z'z$ and has Jacobian with absolute value one. The transformed columns therefore remain independent standard normal vectors, and their entries are again independent standard normal variables. In particular, the rows of $U'Z$ are independent.

Y'P_jY =\Sigma^{1/2}Z'P_jZ\Sigma^{1/2} =\Sigma^{1/2}(U'Z)'(U'P_jU)(U'Z)\Sigma^{1/2}.

The middle expression selects a block of $\nu_j$ rows of $U'Z$. Its contribution is a sum of $\nu_j$ outer products of independent $N_p(0,I_p)$ vectors. Multiplication on both sides by $\Sigma^{1/2}$ turns these into independent $N_p(0,\Sigma)$ vectors, giving the stated Wishart law. Different projections select disjoint row blocks, so the corresponding projected Gaussian data are independent. Each matrix $Y'P_jY$ is a function of its own projected block, which proves mutual independence of the matrices as well. $\square$

Sample Mean and Centered Scatter

A familiar decomposition is obtained by projecting onto the direction of the constant vector and its orthogonal complement. Let $\mathbf 1$ be the $n$-vector of ones and define $$ H=\frac{1}{n}\mathbf 1\mathbf 1',\qquad C=I_n-H. $$ Here $H$ projects onto the constant-vector direction, while $C$ projects onto the subspace orthogonal to that direction. They are symmetric idempotents, $HC=0$, and $H+C=I_n$. Their ranks are $1$ and $n-1$, respectively.

Write $\bar Y=Y'\mathbf 1/n$ for the $p$-vector of column means. Direct multiplication gives $Y'HY=n\bar Y\bar Y'$, while $Y'CY$ is the centered scatter matrix. For centered Gaussian observations, the theorem therefore gives the following result.

Corollary: If the rows of $Y$ are independent $N_p(0,\Sigma)$ vectors, then $Y'CY\sim W_p(n-1,\Sigma)$, $Y'HY\sim W_p(1,\Sigma)$, and these matrices are independent. More strongly, the projected data $CY$ and $HY$ are independent. Since $\bar Y$ is a function of $HY$, the sample mean is independent of the centered scatter.

The stronger statement matters: independence of two matrices alone would not establish independence between one of them and a vector that is not determined by the other matrix. Here $\bar Y$ is obtained from the projected Gaussian component $HY$, so its independence from $Y'CY$ follows from independence of $CY$ and $HY$.

If the rows instead have mean $\mu$, write $Y=\mathbf 1\mu'+E$, where $E$ has centered Gaussian rows. Since $C\mathbf 1=0$, $Y'CY=E'CE$ still has the central law $W_p(n-1,\Sigma)$. Also, $CY=CE$ and $HY=\mathbf 1\mu'+HE$ are independent because $CE$ and $HE$ are independent. Thus the centered scatter is independent of $\bar Y$ in this case too. However, $Y'HY=n\bar Y\bar Y'$ is not generally a central Wishart matrix when $\mu\ne0$; its distribution is affected by the nonzero mean. This distinction is important when applying the central theorem.

Worked Example: The Mean and Scatter for Four Observations

Suppose $Y$ has $n=4$ independent centered bivariate normal rows with covariance $\Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}$. Determine the distributions, expected values, and independence of the fitted and residual sums of squares.

The projection $H=\mathbf 1\mathbf 1'/4$ has rank $1$, and $C=I_4-H$ has rank $3$. Since $H$ and $C$ are orthogonal projections summing to $I_4$, Cochran's theorem gives $$ Y'HY\sim W_2(1,\Sigma),\qquad Y'CY\sim W_2(3,\Sigma), $$ independently. Also, $Y'HY=4\bar Y\bar Y'$.

From the sum-of-outer-products definition of the Wishart distribution, $E[W_p(\nu,\Sigma)]=\nu\Sigma$: each of the $\nu$ outer products has expectation $\Sigma$. Thus $$ E[Y'HY]=\begin{pmatrix}4&1\\1&2\end{pmatrix},\qquad E[Y'CY]=3\begin{pmatrix}4&1\\1&2\end{pmatrix} =\begin{pmatrix}12&3\\3&6\end{pmatrix}. $$ The mean matrix is independent of the centered scatter because the projected Gaussian data $HY$ and $CY$ are independent, and both summaries are functions of their respective projected data.

General Orthogonal Decompositions in Practice

Projections need not correspond only to the sample mean and residuals. Any orthogonal split of the sample-space directions gives independent Wishart components for centered Gaussian data. The ranks determine the degrees of freedom; the choice of basis within any one projected subspace does not change its sum-of-squares matrix.

Worked Example: A Non-Coordinate Rank-One Projection

Let $Z$ be a $4\times2$ matrix with independent standard normal entries, and let $a=(1,1,0,0)'/\sqrt 2$. Set $P=aa'$ and $Q=I_4-P$. Find the distributions of $Z'PZ$ and $Z'QZ$, and determine whether they are independent.

Because $a'a=1$, $P'=P$ and $P^2=a(a'a)a'=P$. Its rank is $1$. In coordinates, $$ P=\frac12\begin{pmatrix} 1&1&0&0\\ 1&1&0&0\\ 0&0&0&0\\ 0&0&0&0 \end{pmatrix}. $$ Consequently, $Q$ is also a symmetric idempotent, has rank $3$, and satisfies $PQ=0$. Further, $$ Z'PZ=(a'Z)'(a'Z). $$ The two entries of the row vector $a'Z$ are independent $N(0,1)$ variables: each is the normalized sum of the corresponding first two independent standard normal entries of $Z$, and the two columns of $Z$ are independent. Hence $Z'PZ\sim W_2(1,I_2)$. The theorem gives $Z'QZ\sim W_2(3,I_2)$, independent of $Z'PZ$. The projection direction here is a linear combination of observation coordinates, not one of the original coordinate axes.

Worked Example: Splitting Four Directions Into Two Groups

Again let $Z$ be a $4\times2$ standard Gaussian data matrix. Define the orthonormal vectors $$ u_1=\frac{(1,1,1,1)'}2,\quad u_2=\frac{(1,-1,0,0)'}{\sqrt2},\quad u_3=\frac{(1,1,-2,0)'}{\sqrt6},\quad u_4=\frac{(1,1,1,-3)'}{\sqrt{12}}. $$ Use the first two directions for one component and the last two for another.

Each vector has squared length $1$. Their pairwise dot products are zero; for instance, the unnormalized dot products of $u_3$ with $u_1,u_2,u_4$ have numerators $1+1-2+0=0$, $1-1+0+0=0$, and $1+1-2+0=0$, respectively. The other pairs likewise have zero dot products. Thus the four vectors form an orthonormal basis. Set $P_1=u_1u_1'+u_2u_2'$ and $P_2=u_3u_3'+u_4u_4'$. These are orthogonal projections of rank $2$, with $P_1P_2=0$ and $P_1+P_2=I_4$.

In the orthogonal coordinates defined by these vectors, $U'Z$ has four independent standard normal rows. The first component uses the first two rows and the second uses the remaining two. Therefore $Z'P_1Z\sim W_2(2,I_2)$, $Z'P_2Z\sim W_2(2,I_2)$, and the components are independent. Each has expected value $2I_2$, so each expected trace is $4$. The trace calculation follows by summing the two diagonal entries of $2I_2$.

Conditions, Interpretation, and a Common Mistake

The central Wishart conclusions require centered Gaussian rows with common covariance, and the projections must be symmetric idempotents with mutually orthogonal ranges. For a complete decomposition their sum must be $I_n$. The theorem does not say that arbitrary quadratic summaries of Gaussian data are independent merely because their expected values or ranks look compatible. The orthogonality condition is what allows the data to be transformed into disjoint blocks of independent Gaussian rows.

Mistake: “The fitted matrix $Y'HY$ is independent of the residual matrix $Y'CY$, so the sample mean $\bar Y$ must be independent of the residual matrix.” This inference is invalid in general: a matrix-valued function of a vector need not determine that vector.
Correction: Establish that the projected Gaussian data $HY$ and $CY$ are independent. Then $\bar Y=Y'\mathbf 1/n$ is a function of $HY$, and $Y'CY$ is a function of $CY$, so these quantities are independent.
Another pitfall: Calling $Y'HY$ central Wishart for nonzero $\mu$. The central theorem applies to centered Gaussian data; with a nonzero mean, the fitted component is shifted and does not in general have the central Wishart law.

Orthogonal projection decompositions are useful because they separate a multivariate sum of squares into interpretable pieces without losing distributional information. The projection ranks supply the Wishart degrees of freedom, while independence of the projected Gaussian data justifies independent-component calculations. In applications, checking the mean condition and the projection geometry before assigning a Wishart law prevents both incorrect distributions and invalid independence claims.

Check Your Understanding

Use the projection ranks and the Gaussian components to answer the following questions.

For centered Gaussian rows and an orthogonal projection $P$ of rank $r$, what is the distribution of $Y'PY$?
Why does the proof of independence examine $PY$ and $QY$, rather than only the matrices $Y'PY$ and $Y'QY$?
For $H=\mathbf 1\mathbf 1'/n$, state the ranks of $H$ and $I_n-H$, and identify the corresponding sums of squares.
If the row mean is nonzero, why does the centered scatter still have a central Wishart law?
What additional condition on a collection of orthogonal projections makes their ranges a decomposition of the full sample space?