Quadratic Sums as Orthogonal Projection Lengths

Why a Quadratic Sum Has a Geometric Meaning

A sum of squares is often introduced as a calculation on coordinates. But a quadratic form such as \(y^\mathsf{T}Py\) also has a geometric interpretation: when \(P\) is an orthogonal projection matrix, the form is the squared length of the component of \(y\) lying in a particular subspace. This viewpoint explains why certain sums of squares add exactly, before any probability distribution or inference is considered.

Throughout, vectors belong to \(\mathbb{R}^n\), with the usual inner product \(x^\mathsf{T}y\) and squared length \(\|x\|^2=x^\mathsf{T}x\). A matrix \(P\) is an orthogonal projection when it maps each vector to its nearest point in a subspace, using this inner product.

Quadratic Forms as Projection Lengths

The two matrix conditions have distinct geometric roles. Symmetry makes the projection orthogonal, so the leftover vector is perpendicular to the projected subspace. Idempotence means that applying the projection a second time changes nothing. Together, they give a direct identity between the quadratic form and a squared length.

Proof. First, \(Py\) belongs to \(\mathcal{C}(P)\). For any \(x\in\mathbb{R}^n\), idempotence gives \(P(y-Py)=Py-P^2y=0\). Therefore, for every \(x\), \((Px)^\mathsf{T}(y-Py)=x^\mathsf{T}P^\mathsf{T}(y-Py)=x^\mathsf{T}P(y-Py)=0\). Thus \(y-Py\) is orthogonal to every vector in the range of \(P\), including \(Py\). Also, \(\|Py\|^2=(Py)^\mathsf{T}(Py)=y^\mathsf{T}P^\mathsf{T}Py=y^\mathsf{T}P^2y=y^\mathsf{T}Py\). Finally, \(y=Py+(y-Py)\) is a sum of orthogonal vectors, so the Pythagorean identity gives \(\|y\|^2=\|Py\|^2+\|y-Py\|^2\). \(\square\)

The same reasoning also shows why the conditions matter. An arbitrary matrix may define a quadratic form, but without symmetry and idempotence it need not be an orthogonal projection, and its quadratic form need not equal the squared length of the transformed vector.

Orthogonal Subspaces Give Additive Sums of Squares

The one-projection identity extends to several subspaces. Suppose a vector is split into components lying in mutually orthogonal subspaces. The squared length of the whole vector is then the sum of the squared lengths of those components. Projection matrices express each component as a quadratic form.

Proof. Set \(v_i=P_i y\). For \(i\ne j\), symmetry and the assumed product condition give \(v_i^\mathsf{T}v_j=y^\mathsf{T}P_i^\mathsf{T}P_jy=y^\mathsf{T}P_iP_jy=0\). Thus the vectors \(v_i\) are pairwise orthogonal. Also, by the projection identity, \(\|v_i\|^2=y^\mathsf{T}P_i y\). Expanding the squared length of their sum, every cross-product is zero, so \(\|\sum_i v_i\|^2=\sum_i\|v_i\|^2=\sum_i y^\mathsf{T}P_i y\). If \(\sum_iP_i=I\), then \(\sum_i v_i=y\), proving the complete decomposition.

For the remaining case, write \(S=\sum_iP_i\). The matrices \(P_i\) are symmetric, so \(S\) is symmetric. The pairwise product conditions and idempotence imply \(S^2=\sum_iP_i^2+\sum_{i\ne j}P_iP_j=\sum_iP_i=S\). Consequently \(R=I-S\) is symmetric and idempotent, and \(P_iR=P_i-P_iS=0\) for every \(i\). The projection \(Ry\) is therefore orthogonal to each \(P_i y\). Since \(y=Sy+Ry=\sum_iP_i y+Ry\), the Pythagorean identity gives the claimed decomposition including \(y^\mathsf{T}Ry=\|Ry\|^2\). \(\square\)

From Projection Lengths to Total Variation

A total sum of squares is a squared length measured from a chosen reference vector. If \(b\) is that reference and \(z=y-b\) is the deviation vector, then the total squared deviation is \(\|z\|^2\). Whenever \(z\) is resolved into orthogonal components, this total is the sum of their squared lengths. For example, if \(P_1,\ldots,P_k\) project onto orthogonal spaces and \(R\) projects onto the remaining orthogonal space, then \(\|y-b\|^2=\sum_i (y-b)^\mathsf{T}P_i(y-b)+(y-b)^\mathsf{T}R(y-b)\).

The reference \(b\) must be stated: changing it changes the deviation vector and therefore the total. Likewise, an additive identity is justified by orthogonality, not simply by naming several pieces of a calculation “components.” In applications, projection spaces encode which directions of variation are being separated; the geometric identity is what guarantees that their squared lengths add without cross-terms.

Common Mistakes and Why the Geometry Matters

• Assuming every quadratic form is a squared length. The equality \(y^\mathsf{T}Py=\|Py\|^2\) uses both \(P^\mathsf{T}=P\) and \(P^2=P\). Without these, it is not generally true.
• Adding components that are not orthogonal. For arbitrary \(a\) and \(b\), \(\|a+b\|^2=\|a\|^2+\|b\|^2+2a^\mathsf{T}b\). The cross-term disappears only when \(a^\mathsf{T}b=0\).
• Assuming the displayed pieces exhaust the total. A collection of orthogonal projections partitions the entire vector only if their projection matrices sum to \(I\). Otherwise, the remaining projection contributes another nonnegative squared length.
• Mixing a reference point with a projection origin. The projection formula applies to the vector being projected. To decompose deviations from \(b\), apply the projections to \(y-b\), not automatically to \(y\).

These facts matter because they let a sums-of-squares calculation be checked geometrically: verify the projection conditions, verify orthogonality of the component spaces, and identify any leftover space. This establishes the algebraic decomposition without requiring a distributional assumption.