Tutorials › Statistical Theory III - Theories of Inference, Multivariate Methods, and Contingency Tables › Worked Matrix Example: Verify an Orthogonal Sum-of-Squares Partition

Home  /  Tutorials  /  Cochran's Theorem and Sums-of-Squares Decompositions

Cochran's Theorem and Sums-of-Squares Decompositions · Tutorial 220 of 972

Worked Matrix Example: Verify an Orthogonal Sum-of-Squares Partition

Work through explicit orthogonal projection matrices to check a sum-of-squares partition and determine the distributions and independence of its Gaussian components.

Intermediate 12 min read

What You'll Learn

  • Check symmetry, idempotence, mutual orthogonality, and the identity-sum condition for explicit matrices.
  • Find projection ranks and relate them to the degrees of freedom of Gaussian quadratic forms.
  • Calculate the sum-of-squares components from a data vector using projection coordinates.
  • Derive central or noncentral chi-square distributions from a spherical Gaussian model.
  • Explain why orthogonality gives independence and why spherical covariance is essential.

Why Verify a Matrix Partition?

A sum-of-squares decomposition is more than an algebraic identity: under a Gaussian model, its matrix components can describe independent random quantities with known distributions. To use that conclusion, however, the matrices must have the right structure. This tutorial checks that structure in explicit examples: symmetry, idempotence, mutual orthogonality, and a sum equal to the identity.

The central objects are orthogonal projection matrices. A projection extracts the part of a vector lying in a specified subspace; orthogonal projections onto perpendicular subspaces produce components that do not overlap. Under a Gaussian distribution with spherical covariance, the corresponding projected components are independent.

Definition: A real matrix \(P\) is an orthogonal projection matrix if \(P'=P\) and \(P^2=P\). A collection \(P_1,\ldots,P_k\) is an orthogonal partition of the identity if each \(P_i\) is an orthogonal projection matrix, \(P_iP_j=0\) whenever \(i\ne j\), and \(\sum_{i=1}^k P_i=I_n\). The rank \(r_i=\operatorname{rank}(P_i)\) is the dimension of the subspace onto which \(P_i\) projects.

If \(y\in\mathbb R^n\) and \(P_i\) is symmetric and idempotent, the projected sum of squares is \(y'P_i y\). The partition identity gives an exact decomposition:

$$ y'y=y'\left(\sum_{i=1}^k P_i\right)y=\sum_{i=1}^k y'P_i y. $$

Each term is nonnegative. In fact, \(y'P_i y=(P_i y)'(P_i y)=\lVert P_i y\rVert^2\), because \(P_i'P_i=P_i^2=P_i\). Thus the decomposition divides the squared length of \(y\) among perpendicular projected components.

A Gaussian Result for the Components

The matrix checks become especially useful when \(Y\) is Gaussian. The next result states precisely how the ranks and projected means determine the distributions.

Theorem: Let \(Y\sim N(\mu,\sigma^2 I_n)\), where \(\sigma^2>0\). Suppose \(P_1,\ldots,P_k\) are symmetric idempotent matrices with \(P_iP_j=0\) for \(i\ne j\) and \(\sum_iP_i=I_n\). Then the random variables \(Y'P_1Y,\ldots,Y'P_kY\) are mutually independent, and $$ \frac{Y'P_iY}{\sigma^2}\sim\chi^2_{r_i}(\lambda_i), \qquad r_i=\operatorname{rank}(P_i), \qquad \lambda_i=\frac{\mu'P_i\mu}{\sigma^2}. $$ Here \(\chi^2_r(\lambda)\) denotes the distribution of the sum of squares of \(r\) independent \(N(a_j,1)\) random variables with \(\sum_j a_j^2=\lambda\). When \(\lambda=0\), this is the central chi-square distribution \(\chi^2_r\).

To see why the theorem holds, choose an orthonormal basis for the range of each \(P_i\). The ranges are perpendicular: if \(x=P_i a\) and \(z=P_j b\), then \(x'z=a'P_iP_jb=0\). Their dimensions sum to \(n\), since the ranges are mutually perpendicular and their projections sum to \(I_n\). Joining the bases therefore gives an orthonormal basis of \(\mathbb R^n\).

In those coordinates, each \(P_i\) keeps the coordinates belonging to its range and sets the others to zero. The transformed Gaussian vector still has covariance \(\sigma^2 I_n\); its coordinates are independent normal random variables. Consequently, each quadratic form is the sum of squares of the coordinates in one block, and different blocks are independent.

1
Choose orthonormal bases.
For each projection, take an orthonormal basis of its range. Mutual orthogonality makes the ranges perpendicular.
2
Use the partition.
The ranges together span \(\mathbb R^n\), so their bases form one orthonormal coordinate system.
3
Read off the distributions.
In the new coordinates, each quadratic form is a block sum of squared independent normal coordinates; its block size is the rank, and its noncentrality is the squared projected mean divided by \(\sigma^2\).

The rank statement follows directly from these coordinates: the \(i\)th block contains \(r_i\) coordinates. The mean vector in that block is the coordinate representation of \(P_i\mu\), whose squared length is \(\mu'P_i\mu\). Dividing the coordinates by \(\sigma\) gives independent unit-variance normals, so the sum of their squared means is \(\mu'P_i\mu/\sigma^2\), as claimed.

This argument also proves a useful identity for the ranks. Since the ranges form an orthogonal decomposition of \(\mathbb R^n\), their dimensions satisfy \(\sum_i r_i=n\). Equivalently, in a basis adapted to the ranges, the projections are block diagonal with an identity block of size \(r_i\) for \(P_i\) and zeros elsewhere.

Worked Matrix Checks

Worked Example: Three One-Dimensional Projections in \(\mathbb R^3\)

Consider the orthonormal vectors \(u_1=(1,1,1)'/\sqrt{3}\), \(u_2=(1,-1,0)'/\sqrt{2}\), and \(u_3=(1,1,-2)'/\sqrt{6}\). Their inner products are \(u_1'u_2=0\), \(u_1'u_3=0\), and \(u_2'u_3=0\), and each has squared length 1. Set \(P_i=u_i u_i'\).

The matrices are

$$ P_1=\frac13\begin{pmatrix}1&1&1\\1&1&1\\1&1&1\end{pmatrix}, \qquad P_2=\frac12\begin{pmatrix}1&-1&0\\-1&1&0\\0&0&0\end{pmatrix}, \qquad P_3=\frac16\begin{pmatrix}1&1&-2\\1&1&-2\\-2&-2&4\end{pmatrix}. $$

Each matrix is symmetric. Also \(P_i^2=u_i(u_i'u_i)u_i'=u_iu_i'=P_i\), so each is idempotent. For \(i\ne j\), \(P_iP_j=u_i(u_i'u_j)u_j'=0\). Finally, the columns \(u_1,u_2,u_3\) are an orthonormal basis, so their outer products sum to the identity: \(P_1+P_2+P_3=I_3\). Each range is one-dimensional, hence \(r_1=r_2=r_3=1\).

For a direct numerical check, let \(y=(2,1,0)'\). Its coordinates along the three basis vectors are \(u_1'y=\sqrt{3}\), \(u_2'y=1/\sqrt{2}\), and \(u_3'y=3/\sqrt{6}\). Therefore,

$$ y'P_1y=3,\qquad y'P_2y=\frac12,\qquad y'P_3y=\frac32, \qquad 3+\frac12+\frac32=5=y'y. $$

If \(Y\sim N((2,0,1)',I_3)\), its three projected mean coordinates are \(\sqrt{3}\), \(\sqrt{2}\), and \(0\). The theorem gives independent components \(Y'P_1Y\sim\chi^2_1(3)\), \(Y'P_2Y\sim\chi^2_1(2)\), and \(Y'P_3Y\sim\chi^2_1\). The last component is central because the mean has zero projection onto the \(u_3\) direction.

Worked Example: A Rank-Two Projection and a Rank-One Projection

Let \(v=(1,1,1)'/\sqrt{3}\), and take \(Q=vv'\) and \(P=I_3-Q\). Explicitly,

$$ Q=\frac13\begin{pmatrix}1&1&1\\1&1&1\\1&1&1\end{pmatrix}, \qquad P=\frac13\begin{pmatrix}2&-1&-1\\-1&2&-1\\-1&-1&2\end{pmatrix}. $$

Because \(v'v=1\), \(Q'=Q\) and \(Q^2=Q\). Thus \(P'=P\), \(P^2=(I_3-Q)^2=I_3-2Q+Q^2=I_3-Q=P\), and \(PQ=QP=0\). The matrices sum to \(I_3\). The range of \(Q\) is the span of \(v\), so its rank is 1; the range of \(P\) is the two-dimensional subspace perpendicular to \(v\), so its rank is 2.

For \(y=(0,2,4)'\), the projection \(Qy\) is \((2,2,2)'\), while \(Py=(-2,0,2)'\). Hence \(y'Qy=12\) and \(y'Py=8\), which add to \(y'y=0^2+2^2+4^2=20\).

Now let \(Y\sim N((1,2,3)',2I_3)\). The projected mean is \(Q\mu=(2,2,2)'\), with squared length 12; \(P\mu=(-1,0,1)'\), with squared length 2. The scaled components are independent and satisfy

$$ \frac{Y'QY}{2}\sim\chi^2_1(6), \qquad \frac{Y'PY}{2}\sim\chi^2_2(1). $$

The divisors are the variance \(2\), and the noncentrality parameters are \(12/2=6\) and \(2/2=1\). In particular, the rank-two component has two degrees of freedom, not one.

Worked Example: An Orthogonal Partition Not Aligned With the Coordinate Axes

In \(\mathbb R^2\), set \(u=(3,4)'/5\) and \(v=(-4,3)'/5\). Both vectors have squared length 1 and \(u'v=(-12+12)/25=0\). Their projection matrices are

$$ P=uu'=\frac1{25}\begin{pmatrix}9&12\\12&16\end{pmatrix}, \qquad Q=vv'=\frac1{25}\begin{pmatrix}16&-12\\-12&9\end{pmatrix}. $$

They are symmetric. The unit lengths give \(P^2=P\) and \(Q^2=Q\), while \(PQ=uu'vv'=u(u'v)v'=0\). Their sum is

$$ P+Q=\frac1{25}\begin{pmatrix}25&0\\0&25\end{pmatrix}=I_2. $$

Both ranks are 1. With \(y=(1,2)'\), the projection coordinates are \(u'y=11/5\) and \(v'y=2/5\). Thus \(y'Py=121/25\) and \(y'Qy=4/25\); their sum is \(125/25=5=y'y\).

For \(Y\sim N((1,0)',I_2)\), the projected mean coordinates are \(u'\mu=3/5\) and \(v'\mu=-4/5\). Consequently \(Y'PY\) and \(Y'QY\) are independent, with distributions \(\chi^2_1(9/25)\) and \(\chi^2_1(16/25)\), respectively. This example shows that the projection subspaces need not align with the original coordinate axes.

What the Checks Establish—and a Common Pitfall

For explicit matrices, checking the conditions in an organized order prevents algebraic identities from being mistaken for distributional conclusions. Symmetry and idempotence establish that each matrix is an orthogonal projection. The zero products establish that the projected subspaces are perpendicular. The identity sum establishes that no part of the vector is omitted. Only after these checks, together with the Gaussian spherical model, can the independent chi-square conclusion be applied.

A common mistake is to infer independence merely because two quadratic forms add to the total sum of squares. The equality \(Y'P_1Y+Y'P_2Y=Y'Y\) alone does not establish independence. The relevant conditions here are orthogonal projection matrices in a jointly Gaussian vector with covariance \(\sigma^2I_n\). The orthogonality gives disjoint coordinates in an orthonormal basis, and the spherical Gaussian covariance makes those coordinates independent. With a nonspherical covariance matrix, orthogonality in the ordinary Euclidean inner product does not by itself give this argument.

Another frequent error is to use the rank as the noncentrality parameter or to forget the variance scaling. The rank determines the degrees of freedom; the noncentrality is \(\mu'P_i\mu/\sigma^2\); and the chi-square statement applies to \(Y'P_iY/\sigma^2\), not to the unscaled sum of squares.

Key takeaway: For a verified orthogonal partition of the identity, the component sums of squares add exactly to the total. Under \(Y\sim N(\mu,\sigma^2I_n)\), they are independent; the rank gives each component's degrees of freedom, and the projected mean gives its noncentrality.

Check Your Understanding

Use the definitions and theorem to answer each question.

  1. What two matrix equalities define an orthogonal projection matrix?
  2. Why does \(P_iP_j=0\) imply that the ranges of \(P_i\) and \(P_j\) are perpendicular?
  3. If a projection in an orthogonal partition has rank 3, what is the degrees-of-freedom parameter for its scaled Gaussian sum of squares?
  4. For \(Y\sim N(\mu,\sigma^2I_n)\), what condition makes the component associated with \(P_i\) central rather than noncentral chi-square?
  5. Why does a sum-of-squares identity alone not justify claiming that its components are independent?

Previous

Next

Back to Tutorials
Clinical Biostats