Nonorthogonal Sums of Squares and Loss of Independence

Why an Exact Identity Is Not an Independence Result

An equality between sums of squares is an algebraic statement: it holds for each possible data vector. Independence is a probability statement about the joint distribution of random components. One does not automatically imply the other. In normal-theory decompositions, orthogonality provides the crucial bridge: for a spherical normal vector, orthogonal projections are independent. When the relevant spaces overlap in nonorthogonal directions, their sums of squares generally share random variation.

We work first with a centered spherical normal vector \(Y\sim N(0,\sigma^2 I_n)\), where \(\sigma^2>0\). For a subspace \(S\subseteq\mathbb R^n\), let \(P_S\) denote the orthogonal projection matrix onto \(S\). Its projection sum of squares is \(Y'P_SY=\|P_SY\|^2\). These definitions let us separate the geometry of the subspaces from the probability calculation.

Definition: Two subspaces \(S\) and \(T\) are orthogonal if \(s't=0\) for every \(s\in S\) and \(t\in T\). Their projection sums of squares are \(Q_S=Y'P_SY\) and \(Q_T=Y'P_TY\). The subspaces may have a nonzero intersection, or may have directions that are not orthogonal even if their intersection is trivial.

Covariance Measures More Than the Shared Dimension

For centered spherical normal data, the covariance between projection sums of squares has a compact matrix expression. It depends on the full geometric overlap between the spaces, not just their intersection.

Theorem: If \(Y\sim N(0,\sigma^2 I_n)\) and \(A,B\) are symmetric matrices, then \(\operatorname{Cov}(Y'AY,Y'BY)=2\sigma^4\operatorname{tr}(AB)\). In particular, for subspaces \(S,T\), \(\operatorname{Cov}(Q_S,Q_T)=2\sigma^4\operatorname{tr}(P_SP_T)\).

Proof. The centered normal fourth-moment identity is \(E[Y_iY_jY_kY_l]=\sigma^4(\delta_{ij}\delta_{kl}+\delta_{ik}\delta_{jl}+\delta_{il}\delta_{jk})\), where \(\delta_{ij}\) is 1 when \(i=j\) and 0 otherwise. It follows, for example, by differentiating the normal moment-generating function \(E[\exp(t'Y)]=\exp(\sigma^2t't/2)\) four times at zero. Expanding both quadratic forms by their entries and applying this identity gives

E[(Y'AY)(Y'BY)] =\sigma^4\{\operatorname{tr}(A)\operatorname{tr}(B) +\operatorname{tr}(AB)+\operatorname{tr}(AB')\}.

Because \(B\) is symmetric, \(B'=B\), so the last two terms sum to \(2\operatorname{tr}(AB)\). Also \(E[Y'AY]=\sigma^2\operatorname{tr}(A)\) and \(E[Y'BY]=\sigma^2\operatorname{tr}(B)\). Subtracting the product of these expectations proves the stated covariance formula. Projection matrices are symmetric and idempotent, so substituting \(A=P_S\) and \(B=P_T\) gives the particular result. \(\square\)

For projection matrices, \(\operatorname{tr}(P_SP_T)\geq 0\); indeed it equals \(\operatorname{tr}(P_TP_SP_T)\), the sum of squared lengths of the projections under consideration. Shared orthonormal directions contribute 1 each to this trace, but nonorthogonal directions can contribute as well. Thus an intersection dimension alone does not determine the covariance.

Worked Example: An Intersection Does Not Determine the Covariance

In \(\mathbb R^3\), let \(S=\operatorname{span}(e_1,e_2)\) and \(T=\operatorname{span}(e_1,v)\), where \(v=(e_2+e_3)/\sqrt{2}\). The vectors \(e_1,e_2,v\) are unit vectors, and \(e_1\) is orthogonal to both \(e_2\) and \(v\). The spaces intersect exactly in \(\operatorname{span}(e_1)\): a vector in their intersection must have zero \(e_3\) coordinate, forcing its coefficient on \(v\) to be zero.

The projections are \(P_S=e_1e_1'+e_2e_2'\) and \(P_T=e_1e_1'+vv'\). Since \(e_1\) and \(e_2\) form an orthonormal basis for \(S\),

\operatorname{tr}(P_SP_T) =e_1'P_Te_1+e_2'P_Te_2 =1+\left(e_2'e_1\right)^2+\left(e_2'v\right)^2 =1+0+\frac12=\frac32.

Therefore, if \(Y\sim N(0,\sigma^2 I_3)\), then \(\operatorname{Cov}(Q_S,Q_T)=2\sigma^4(3/2)=3\sigma^4\). The shared direction contributes \(2\sigma^4\), while the nonorthogonal directions \(e_2\) and \(v\) contribute another \(\sigma^4\). If the spaces were orthogonal apart from their shared direction, the trace would instead be 1 and the covariance would be \(2\sigma^4\). The intersection by itself does not settle the answer.

Orthogonality and Independence

The projection geometry also gives a direct condition for independence. The vectors \(P_SY\) and \(P_TY\) are jointly normal, with cross-covariance matrix \(E[(P_SY)(P_TY)']=\sigma^2P_SP_T\). For jointly normal vectors, zero cross-covariance implies independence: their joint normal characteristic function factors into the product of their marginal characteristic functions when the cross-covariance block is zero. Consequently, if \(S\perp T\), then \(P_SP_T=0\), and the projected vectors—and hence their squared lengths \(Q_S\) and \(Q_T\)—are independent. Conversely, for these projected vectors, independence requires zero cross-covariance and therefore \(P_SP_T=0\).

For sums of squares, the covariance theorem gives a useful diagnostic in the centered case. If \(\operatorname{tr}(P_SP_T)>0\), then \(\operatorname{Cov}(Q_S,Q_T)>0\), so the sums of squares cannot be independent. A zero covariance alone is not, in general, a proof of independence; the orthogonality argument above supplies that conclusion for these normal projections.

Worked Example: Two Nonorthogonal Contrast Sums

Let \(u_1=e_1\) and \(u_2=(e_1+e_2)/\sqrt{2}\), and define the contrasts \(C_1=u_1'Y\) and \(C_2=u_2'Y\), with \(Y\sim N(0,\sigma^2 I_2)\). Both vectors have unit length, but \(u_1'u_2=1/\sqrt{2}\), so the contrast directions are not orthogonal.

Their variances and covariance are \(\operatorname{Var}(C_1)=\sigma^2\), \(\operatorname{Var}(C_2)=\sigma^2\), and \(\operatorname{Cov}(C_1,C_2)=\sigma^2u_1'u_2=\sigma^2/\sqrt{2}\). Since \(C_1,C_2\) are jointly normal and their covariance is nonzero, they are not independent. Their squares are not independent either: applying the covariance formula to \(A=u_1u_1'\) and \(B=u_2u_2'\) gives

\operatorname{Cov}(C_1^2,C_2^2) =2\sigma^4\operatorname{tr}(u_1u_1'u_2u_2') =2\sigma^4(u_1'u_2)^2 =\sigma^4.

Thus the nonorthogonality is visible directly in the squared components. Merely naming two contrasts or writing their squared values separately does not make those values independent.

An Algebraic Sum-of-Squares Identity With Dependent Terms

A particularly clear demonstration that an identity is not enough uses three nonorthogonal contrasts. The terms in the identity below are weighted squared contrasts, not orthogonal projection sums of squares. That distinction matters: an orthogonal projection partition has additional geometric structure, whereas a general quadratic-form identity need not.

Worked Example: A Tight-Frame Identity With Correlated Components

In \(\mathbb R^2\), take \(u_1=(1,0)'\), \(u_2=(1/2,\sqrt{3}/2)'\), and \(u_3=(-1/2,\sqrt{3}/2)'\). Each vector has unit length, and each pair has inner product either \(1/2\) or \(-1/2\). Directly summing their outer products gives

u_1u_1'+u_2u_2'+u_3u_3' = \begin{pmatrix}1&0\\0&0\end{pmatrix} +\begin{pmatrix}1/4&\sqrt{3}/4\\\sqrt{3}/4&3/4\end{pmatrix} +\begin{pmatrix}1/4&-\sqrt{3}/4\\-\sqrt{3}/4&3/4\end{pmatrix} =\frac32 I_2.

For any \(y\in\mathbb R^2\), this matrix equality implies the exact identity \(\|y\|^2=(2/3)\sum_{i=1}^3(u_i'y)^2\). Now let \(Y\sim N(0,\sigma^2 I_2)\) and set \(T_i=\sqrt{2/3}\,u_i'Y\). Then \(\sum_i T_i^2=Y'Y\) for every value of \(Y\), not merely on average.

Yet the components are dependent. For example, \(\operatorname{Var}(T_i)=(2/3)\sigma^2\), and \(\operatorname{Cov}(T_1,T_2)=(2/3)\sigma^2(u_1'u_2)=\sigma^2/3\). More decisively, the normal-square covariance formula yields \(\operatorname{Cov}(T_1^2,T_2^2)=2\{\operatorname{Cov}(T_1,T_2)\}^2=2\sigma^4/9>0\). The exact sum-of-squares identity therefore coexists with dependent terms. The matrices \((2/3)u_iu_i'\) are not idempotent projections, so this example is an algebraic quadratic-form decomposition, not an orthogonal projection partition.

Common Pitfall and Practical Check

Wrong reasoning: “These components add up exactly to the total sum of squares, so they must be independent.” The identity only establishes equality of quadratic forms. Independence requires a distributional argument, and in the normal projection setting the relevant spaces must be orthogonal for the projected vectors to be independent.

A second version of the same mistake is to infer the covariance from the dimension of \(S\cap T\) alone. The first worked example shows why this fails: even with a one-dimensional intersection, other directions can be nonorthogonal and increase \(\operatorname{tr}(P_SP_T)\). Compute the trace, or establish orthogonality directly; do not substitute intersection dimension for geometric overlap.

Identify the objects.
Check whether each component is a projection sum of squares or a more general quadratic form.

Check the geometry.
For projection sums, determine whether the subspaces are orthogonal; if not, compute \(\operatorname{tr}(P_SP_T)\).

Separate identity from probability.
Use the algebra to verify a sum-of-squares equality, then use the distribution and covariance structure to assess independence.

Key takeaway: Under centered spherical normality, projection sums with nonorthogonal spaces have covariance \(2\sigma^4\operatorname{tr}(P_SP_T)\). An algebraic sum-of-squares identity does not replace the orthogonality condition needed for independence.

Check Your Understanding

Use the geometric and probabilistic arguments in this tutorial to answer the following.

For \(Y\sim N(0,\sigma^2 I_n)\), what quantity determines the covariance of two projection sums of squares?
Why does knowing only the dimension of \(S\cap T\) fail to determine \(\operatorname{Cov}(Q_S,Q_T)\)?
If two subspaces are orthogonal, what argument establishes independence of their projected vectors?
In the three-contrast example, what does the matrix identity establish, and what does it not establish?
Can nonzero covariance between two random variables be consistent with independence? Apply your answer to the projection sums discussed here.