Decomposing Total Variation Around the Sample Mean

Why the Sample Mean Separates Two Kinds of Variation

A sum of squared observations measures distance from zero. A sum of squared deviations from the sample mean instead measures how spread out the observations are around their own center. These are different quantities: the first also includes the contribution of the location of the data, while the second does not. The difference between them is an exact identity, not an approximation or a probabilistic statement.

The identity has a useful geometric form. A data vector splits into its projection onto the constant-vector direction and its projection onto the perpendicular space of centered vectors. The mean contribution lies in a one-dimensional space; centered variation lies in a space of dimension \(n-1\). These dimensions are the ranks of the corresponding projection matrices.

The Algebraic Decomposition

For each observation, write \(x_i=(x_i-\bar{x})+\bar{x}\). Squaring and summing produces a cross term, but that term vanishes because deviations from the sample mean sum to zero:

This is the corrected-versus-uncorrected total-sum-of-squares identity. It applies to any real-valued observations and any positive integer \(n\). No independence, normality, or sampling assumption is needed.

The Projection Form

Write the data as the column vector \(\mathbf{x}=(x_1,\ldots,x_n)^\mathsf{T}\), and let \(\mathbf{1}\) denote the \(n\)-vector of ones. Define

The matrix \(J\) maps a vector to the constant vector whose entries are its mean: \(J\mathbf{x}=\bar{x}\mathbf{1}\). The matrix \(M\) subtracts that mean from every entry: \(M\mathbf{x}=\mathbf{x}-\bar{x}\mathbf{1}\). Both are symmetric and idempotent, so each is an orthogonal projection.

Proof. Since \(\mathbf{1}^\mathsf{T}\mathbf{1}=n\),

Thus \(J\) is a symmetric idempotent matrix. Its image is exactly \(\operatorname{span}(\mathbf{1})\): every output \(J\mathbf{v}\) is a multiple of \(\mathbf{1}\), and \(J\mathbf{1}=\mathbf{1}\). This image has dimension one, so \(\operatorname{rank}(J)=1\).

The matrix \(M=I_n-J\) is symmetric, and

Its image is the null space of \(J\), which is \(\{\mathbf{v}:\mathbf{1}^\mathsf{T}\mathbf{v}=0\}\), since \(J\mathbf{v}=\mathbf{1}(\mathbf{1}^\mathsf{T}\mathbf{v})/n\). This space has dimension \(n-1\): it is defined by one nonzero linear constraint. Therefore \(\operatorname{rank}(M)=n-1\). Also \(JM=J(I_n-J)=0\), so the two projected components are orthogonal. Finally, \(\mathbf{x}=M\mathbf{x}+J\mathbf{x}\), and orthogonality gives

Here the last equality uses symmetry and idempotence of each projection. Moreover, \(\mathbf{x}^\mathsf{T}M\mathbf{x}=\sum_i(x_i-\bar{x})^2\), while \(J\mathbf{x}=\bar{x}\mathbf{1}\), so \(\mathbf{x}^\mathsf{T}J\mathbf{x}=n\bar{x}^2\). This proves the stated decomposition and ranks. \(\square\)

A Related Identity Around a Fixed Center

The same vanishing-cross-term argument gives a useful variant. For any fixed real number \(a\),

Indeed, \(x_i-a=(x_i-\bar{x})+(\bar{x}-a)\); after squaring and summing, the cross term is \(2(\bar{x}-a)\sum_i(x_i-\bar{x})=0\). The fixed center \(a\) need not be zero. In particular, taking \(a=0\) recovers the original identity, and taking \(a\) to be a hypothesized population mean separates sample spread from the squared displacement of the sample mean from that value.

What the Ranks Mean Under Normality

The rank counts the number of independent directions contributing to a projected squared length. This is why the mean contribution has one degree of freedom and the corrected sum of squares has \(n-1\). For an exact normal-theory illustration, suppose \(X_1,\ldots,X_n\) are independent and each has distribution \(N(\mu,\sigma^2)\), with \(\sigma^2>0\). The centered vector \(M\mathbf{X}\) has no component in the constant-vector direction, regardless of \(\mu\); the mean vector \(\mu\mathbf{1}\) is annihilated by \(M\).

Choose an orthonormal basis whose first vector is \(\mathbf{1}/\sqrt{n}\). In these coordinates, \(\mathbf{X}\) has independent normal coordinates with variance \(\sigma^2\); the first coordinate is \(\sqrt{n}\bar{X}\), and the other \(n-1\) coordinates make up the centered component. Consequently,

and these two quantities are independent. Here \(\chi^2_k(\lambda)\) denotes a noncentral chi-square distribution with \(k\) degrees of freedom and noncentrality parameter \(\lambda\); \(\chi^2_k=\chi^2_k(0)\) is central. The first result follows because the \(n-1\) centered orthonormal coordinates, divided by \(\sigma\), are independent standard normal variables whose squared sum has a \(\chi^2_{n-1}\) distribution. The first coordinate divided by \(\sigma\) is normal with mean \(\sqrt{n}\mu/\sigma\) and variance one, so its square has the stated noncentral chi-square distribution. Independence follows from independence of the coordinates in this orthonormal basis. These distributional statements require the stated independent normal model; the algebraic decomposition and ranks do not.

Common Pitfall and Practical Importance

A common mistake is to treat \(\sum_i x_i^2\) as if it were automatically the variation among the observations. It is variation about zero. If the observations share a large location, \(n\bar{x}^2\) can dominate even when they are tightly clustered. The corrected sum of squares measures their spread about the sample mean; the uncorrected sum includes both that spread and the mean contribution.

A second error is to replace \(n\bar{x}^2\) by \(n(\bar{x}-a)^2\) in the decomposition of \(\sum_i x_i^2\). That replacement is valid only when the left side is changed to \(\sum_i(x_i-a)^2\). Keeping track of the center is essential: the identity is a Pythagorean decomposition relative to a specified origin or fixed center.

The projection viewpoint makes the degrees of freedom transparent. There is one direction for the overall mean, and imposing the centering constraint removes that direction from the \(n\)-dimensional observation space, leaving \(n-1\) directions. This geometry is the foundation for exact normal-theory sums-of-squares calculations.