Why Gaussianity and Spherical Covariance Matter

Why Orthogonal Pieces May Still Be Dependent

Orthogonal projection matrices split a vector into geometrically separate components. For sums of squares formed from those components, orthogonality guarantees an additive decomposition of the total squared length. It does not by itself guarantee that the component sums of squares are statistically independent. Exact independence in the familiar normal-theory setting comes from the error distribution as well as the geometry: the errors must be Gaussian with spherical covariance.

This distinction matters whenever independent sums of squares are used to justify exact probability calculations. The examples below isolate two different ways the conclusion can fail: non-Gaussian errors even with identity covariance, and Gaussian errors with nonspherical covariance.

Definition: A matrix \(P\) is an orthogonal projection matrix if \(P=P'\) and \(P^2=P\). Two such matrices \(P\) and \(Q\) are orthogonal if \(PQ=0\). For a centered random error vector \(\varepsilon\), the associated Euclidean projection sums of squares are \(\varepsilon'P\varepsilon=\|P\varepsilon\|^2\) and \(\varepsilon'Q\varepsilon=\|Q\varepsilon\|^2\). The covariance is spherical if \(\operatorname{Cov}(\varepsilon)=\sigma^2 I\) for some \(\sigma^2>0\).

When \(PQ=0\), the projected vectors \(P\varepsilon\) and \(Q\varepsilon\) lie in orthogonal subspaces. In particular, their inner product is zero, and the squared lengths add:

\|(P+Q)\varepsilon\|^2 = \|P\varepsilon\|^2+\|Q\varepsilon\|^2.

That is an algebraic statement, valid for every vector \(\varepsilon\). Independence is a distributional statement about how often different pairs of lengths occur. It does not follow from this identity alone.

The Gaussian Spherical Case

Here is the exact independence fact behind the usual normal-theory argument. It concerns centered errors; adding a nonzero mean can change the distributions of the projected components and is not covered by this statement.

Theorem: Let \(\varepsilon\sim N(0,\sigma^2 I)\), with \(\sigma^2>0\), and let \(P\) and \(Q\) be orthogonal projection matrices satisfying \(PQ=0\). Then \(P\varepsilon\) and \(Q\varepsilon\) are independent. Consequently, the projection sums of squares \(\varepsilon'P\varepsilon\) and \(\varepsilon'Q\varepsilon\) are independent.

Proof. The pair \((P\varepsilon,Q\varepsilon)\) is jointly Gaussian because it is a linear transformation of a Gaussian vector. Its cross-covariance matrix is

\operatorname{Cov}(P\varepsilon,Q\varepsilon) = P\operatorname{Cov}(\varepsilon)Q' = \sigma^2 PQ = 0.

For jointly Gaussian random vectors, zero cross-covariance implies independence: their joint characteristic function has no cross term and factors into the product of the two marginal characteristic functions. Thus \(P\varepsilon\) and \(Q\varepsilon\) are independent. Each sum of squares is a function of one of these independent vectors, so the two sums of squares are independent as well. \(\square\)

This proof identifies the roles of the assumptions. Orthogonality gives \(PQ=0\), and spherical covariance makes the cross-covariance of the projected vectors a multiple of \(PQ\). Gaussianity turns zero cross-covariance into independence. Without Gaussianity, that final implication is not generally valid. Without spherical covariance, even the projected vectors may have nonzero cross-covariance.

Worked Examples: How Independence Fails

Worked Example: Rademacher Errors With Identity Covariance

Let \(\varepsilon=(\varepsilon_1,\varepsilon_2)'\), where \(\varepsilon_1\) and \(\varepsilon_2\) are independent and each equals \(1\) or \(-1\) with probability \(1/2\). These errors have mean zero and covariance \(I\), but they are not Gaussian. Take the orthogonal projections onto the directions \((1,1)'/\sqrt{2}\) and \((1,-1)'/\sqrt{2}\), respectively:

P=\frac12 \begin{pmatrix} 1&1\\ 1&1 \end{pmatrix}, \qquad Q=\frac12 \begin{pmatrix} 1&-1\\ -1&1 \end{pmatrix}.

The matrices are symmetric and idempotent, and multiplication gives \(PQ=0\). The projection sums of squares are

\varepsilon'P\varepsilon = \frac{(\varepsilon_1+\varepsilon_2)^2}{2}, \qquad \varepsilon'Q\varepsilon = \frac{(\varepsilon_1-\varepsilon_2)^2}{2}.

If the two signs agree, the first sum is \(2\) and the second is \(0\). This happens with probability \(1/2\). If the signs disagree, the first sum is \(0\) and the second is \(2\), also with probability \(1/2\). Each sum has expectation \(1\), but their product is always zero. Therefore,

\operatorname{Cov}(\varepsilon'P\varepsilon,\varepsilon'Q\varepsilon) = E[(\varepsilon'P\varepsilon)(\varepsilon'Q\varepsilon)]-1\cdot1 = 0-1 = -1.

The sums are dependent. This example shows that even independent, identically distributed errors with identity covariance do not guarantee independence of orthogonal projection sums of squares when the errors are non-Gaussian.

Worked Example: Gaussian Errors With Correlated Coordinates

Let \(Z_1,Z_2\) be independent \(N(0,1)\) random variables, and define \(X=Z_1\) and \(Y=\tfrac12 Z_1+\tfrac{\sqrt{3}}{2}Z_2\). Then \(\varepsilon=(X,Y)'\) is Gaussian, with

\operatorname{Cov}(\varepsilon) = \begin{pmatrix} 1&1/2\\ 1/2&1 \end{pmatrix}.

This covariance is not spherical. Use the orthogonal coordinate projections \(P=\operatorname{diag}(1,0)\) and \(Q=\operatorname{diag}(0,1)\). The two projection sums of squares are \(X^2\) and \(Y^2\). Since \(X\) and \(Y\) are standard normal with correlation \(1/2\), the Gaussian fourth-moment formula gives \(E[X^2Y^2]=1+2(1/2)^2=3/2\). Also, \(E[X^2]=E[Y^2]=1\), so

\operatorname{Cov}(X^2,Y^2) = E[X^2Y^2]-E[X^2]E[Y^2] = \frac32-1 = \frac12.

Nonzero covariance rules out independence. The coordinate subspaces are Euclidean-orthogonal and the errors are Gaussian, but nonspherical covariance makes the projected components correlated.

Worked Example: A Continuous Spherical Distribution That Is Not Gaussian

Let \(\Theta\) be uniform on \([0,2\pi)\), and set \(X=\sqrt{2}\cos\Theta\), \(Y=\sqrt{2}\sin\Theta\). The error vector \(\varepsilon=(X,Y)'\) has mean zero. Since \(E[\cos^2\Theta]=E[\sin^2\Theta]=1/2\) and \(E[\sin\Theta\cos\Theta]=0\), its covariance is \(I\). It is not Gaussian: its length is always \(\sqrt{2}\).

For the coordinate projections, the sums of squares are \(X^2\) and \(Y^2\). Their sum is exactly \(2\), so knowing one determines the other. More explicitly, \(E[X^2]=E[Y^2]=1\), while \(E[\cos^2\Theta\sin^2\Theta]=1/8\), giving

E[X^2Y^2]=4\cdot\frac18=\frac12, \qquad \operatorname{Cov}(X^2,Y^2)=\frac12-1=-\frac12.

Thus these projection sums are dependent despite spherical covariance and orthogonal coordinate projections. This example makes clear that spherical covariance alone does not replace Gaussianity.

A Gaussian Covariance Calculation

A useful general calculation quantifies the effect of nonspherical covariance for centered Gaussian errors. It also highlights the difference between the independence result above and a covariance calculation.

Result: If \(\varepsilon\) is centered Gaussian with covariance matrix \(\Sigma\), and \(P,Q\) are symmetric matrices, then

\operatorname{Cov}(\varepsilon'P\varepsilon,\varepsilon'Q\varepsilon) = 2\operatorname{tr}(P\Sigma Q\Sigma).

Proof. Write the two quadratic forms as \(\sum_{i,j}P_{ij}\varepsilon_i\varepsilon_j\) and \(\sum_{k,l}Q_{kl}\varepsilon_k\varepsilon_l\). For centered Gaussian coordinates, differentiation of the moment-generating function \(\exp(t'\Sigma t/2)\) four times at zero gives

E[\varepsilon_i\varepsilon_j\varepsilon_k\varepsilon_l] = \Sigma_{ij}\Sigma_{kl} +\Sigma_{ik}\Sigma_{jl} +\Sigma_{il}\Sigma_{jk}.

In the expectation of the product of the quadratic forms, the first pairing contributes \(E[\varepsilon'P\varepsilon]E[\varepsilon'Q\varepsilon]\), which cancels when the covariance is formed. The second and third pairings each contribute \(\operatorname{tr}(P\Sigma Q\Sigma)\); symmetry of \(P,Q,\Sigma\) gives the same trace for both. Adding those two contributions proves the formula. \(\square\)

For spherical covariance, the formula becomes \(2\sigma^4\operatorname{tr}(PQ)\), which is zero for orthogonal projections. But zero covariance of two random variables does not by itself imply independence. In the Gaussian spherical theorem, independence came from the stronger fact that the projected vectors are jointly Gaussian with zero cross-covariance. In the nonspherical Gaussian example, the covariance formula detects dependence directly.

Common Mistake and Why the Distinction Matters

Wrong reasoning: “The projection subspaces are orthogonal, so the two projection sums of squares must be independent.” This confuses orthogonality of vectors or subspaces with statistical independence. Orthogonality ensures the algebraic squared-length decomposition; it says nothing by itself about the joint distribution of the lengths.

Correction: Check the error distribution and covariance as well as the projection geometry. For centered Gaussian errors with covariance \(\sigma^2 I\), orthogonal projections produce independent projected vectors and hence independent sums of squares. If the errors are non-Gaussian, zero cross-covariance is not enough. If the Gaussian covariance is nonspherical, Euclidean-orthogonal projections need not even be uncorrelated.

These distinctions are important because independence is what permits separate probability calculations for components of a sum-of-squares decomposition. The decomposition itself may remain algebraically correct when independence fails, but probability statements that treat its pieces as independent need additional justification.

Check Your Understanding

For each question, distinguish what follows from geometry alone from what requires assumptions on the error distribution.

What does \(PQ=0\) guarantee about the projected vectors and their squared lengths, and what does it not guarantee?
In the Rademacher example, verify from the two possible sign patterns that each sum has expectation \(1\) and their product is always zero.
Why does zero cross-covariance imply independence for the projected vectors in the Gaussian spherical theorem?
In the Gaussian nonspherical example, what covariance between the two squared projections rules out independence?
Can spherical covariance alone guarantee independence of orthogonal projection sums of squares? Explain using the continuous example.