Pooling Within-Group Sums of Squares

Why Pool Within-Group Variation?

In a one-way layout, each group has its own mean and its own residual variation around that mean. When the groups have a common error variance, their within-group sums of squares can be added to summarize variation about the separate group means. Pooling does not mean averaging the group sums of squares: the contribution from each group is added, and its degrees of freedom are added as well.

The geometric reason is that centering within a group is an orthogonal projection onto a residual subspace. Different groups occupy separate coordinate blocks, so their residual subspaces are orthogonal. Under independent normal sampling, the resulting squared projection lengths are independent. A common variance then makes their sum a single scaled chi-square variable.

Definition: For group \(g\), let \(X_g=(X_{g1},\ldots,X_{gn_g})^\mathsf{T}\) be its observation vector and let \(\bar X_g=n_g^{-1}\sum_{i=1}^{n_g}X_{gi}\). The within-group sum of squares is \(S_g=\sum_{i=1}^{n_g}(X_{gi}-\bar X_g)^2\). For \(m\) groups, the pooled within-group sum of squares is \(W=\sum_{g=1}^{m}S_g\). Its residual degrees of freedom are \(\sum_{g=1}^{m}(n_g-1)\).

The degrees of freedom count the dimensions left after estimating one group mean in each group. A group of size \(n_g\) has \(n_g\) observation coordinates, but its centered residuals sum to zero, leaving \(n_g-1\) dimensions. A group of size one therefore contributes zero residual degrees of freedom and a within-group sum of squares of zero.

Projection Form and Pooled Rank

Write \(\mathbf 1_g\) for the \(n_g\)-vector of ones and define \(R_g=I_{n_g}-\mathbf 1_g\mathbf 1_g^\mathsf{T}/n_g\). This matrix subtracts the group mean from every observation. It is symmetric and idempotent, so it is the orthogonal projection onto the subspace perpendicular to \(\mathbf 1_g\).

S_g=X_g^\mathsf{T}R_gX_g,\qquad \operatorname{rank}(R_g)=\operatorname{tr}(R_g)=n_g-1.

Stack the group vectors into \(X=(X_1^\mathsf{T},\ldots,X_m^\mathsf{T})^\mathsf{T}\), and let \(R\) be the block-diagonal matrix with blocks \(R_1,\ldots,R_m\). Then \(W=X^\mathsf{T}RX\). Because the blocks act on separate coordinates, the rank of \(R\) is the sum of their ranks.

Rank result: The pooled residual projection has rank \(r=\sum_{g=1}^{m}(n_g-1)=N-m\), where \(N=\sum_{g=1}^{m}n_g\). Thus pooling adds the residual dimensions; it does not create an extra residual dimension for each group.

Exact Distribution Under a Common Variance

Assume each group has a fixed mean \(\mu_g\), all observations are normal with common variance \(\sigma^2>0\), and the group observation vectors are independent: \(X_g\sim N_{n_g}(\mu_g\mathbf 1_g,\sigma^2 I_{n_g})\). The means may differ across groups. The common-variance condition concerns the within-group variation, not equality of the group means.

Theorem: Under the preceding assumptions, the group sums \(S_1,\ldots,S_m\) are independent and \[ \frac{S_g}{\sigma^2}\sim\chi^2_{n_g-1} \quad\text{for each }g. \] Consequently, if \(r=\sum_g(n_g-1)>0\), then \[ \frac{W}{\sigma^2}\sim\chi^2_r. \] If \(r=0\), then \(W=0\) with probability one.

Proof. For a group with \(d_g=n_g-1>0\), choose an \(n_g\times d_g\) matrix \(Q_g\) whose columns form an orthonormal basis for the subspace perpendicular to \(\mathbf 1_g\). Then \(R_g=Q_gQ_g^\mathsf{T}\), and \(Q_g^\mathsf{T}\mathbf 1_g=0\). Therefore \(S_g=\|Q_g^\mathsf{T}X_g\|^2\). The vector \(Q_g^\mathsf{T}X_g\) is multivariate normal with mean zero, because \(Q_g^\mathsf{T}\mu_g\mathbf 1_g=0\), and covariance matrix \(\sigma^2 I_{d_g}\), because \(Q_g^\mathsf{T}Q_g=I_{d_g}\). Its \(d_g\) coordinates, divided by \(\sigma\), are independent standard normal variables. The sum of their squares is \(\chi^2_{d_g}\), proving the stated distribution for \(S_g/\sigma^2\). When \(d_g=0\), the residual subspace contains only the zero vector and \(S_g=0\).

Each \(S_g\) is a measurable function of \(X_g\). Independence of the group vectors therefore gives independence of these functions. Finally, independent chi-square variables add their degrees of freedom: this follows, for example, by multiplying their moment-generating functions \((1-2t)^{-d_g/2}\) for \(t<1/2\). Thus \(W/\sigma^2\sim\chi^2_{\sum_g d_g}\) when that sum is positive. If it is zero, every group contribution is zero. \(\square\)

Worked Examples

Worked Example: Pooling Two Observed Within-Group Sums

Suppose group A contains \(2,4,6\), and group B contains \(1,3,5,7\). Calculate each within-group sum of squares and the pooled residual degrees of freedom.

For group A, \(\bar X_A=(2+4+6)/3=4\), so \(S_A=(2-4)^2+(4-4)^2+(6-4)^2=4+0+4=8\).

For group B, \(\bar X_B=(1+3+5+7)/4=4\), so \(S_B=(1-4)^2+(3-4)^2+(5-4)^2+(7-4)^2=9+1+1+9=20\). The pooled sum is \(W=8+20=28\).

The residual degrees of freedom are \((3-1)+(4-1)=2+3=5\). If these observations are modeled as independent normal observations with a common variance \(\sigma^2\), then \(W/\sigma^2\sim\chi^2_5\). The numerical value 28 is the observed pooled sum; it is not itself the degrees of freedom or a chi-square variable after scaling is omitted.

Worked Example: Seeing the Three Residual Dimensions

Consider two independent normal groups with sizes two and three, means \(\mu_1\) and \(\mu_2\), and common variance \(\sigma^2\). Use orthonormal residual contrasts to verify the pooled distribution.

For group one, the residual contrast is \(q_1=(1,-1)^\mathsf{T}/\sqrt{2}\). Its coordinate \(q_1^\mathsf{T}X_1=(X_{11}-X_{12})/\sqrt{2}\) has mean zero and variance \(\sigma^2\). The group sum is \(S_1=(q_1^\mathsf{T}X_1)^2\).

For group two, use the two orthonormal contrasts \(q_{21}=(1,-1,0)^\mathsf{T}/\sqrt{2}\) and \(q_{22}=(1,1,-2)^\mathsf{T}/\sqrt{6}\). Both are perpendicular to \((1,1,1)^\mathsf{T}\). The two coordinates \(q_{21}^\mathsf{T}X_2\) and \(q_{22}^\mathsf{T}X_2\) have mean zero and variance \(\sigma^2\), and their covariance is \(\sigma^2q_{21}^\mathsf{T}q_{22}=0\). They are jointly normal, so they are independent. Their squared sum is \(S_2\).

The first group contributes one standardized squared normal and the second contributes two. The group vectors are independent, so all three standardized contrast coordinates are independent standard normals. Hence \((S_1+S_2)/\sigma^2\sim\chi^2_3\). The rank count gives the same answer: \((2-1)+(3-1)=3\).

Worked Example: Unequal Variances and a Zero-Degree Group

Take independent normal groups with sizes two and three, but let their variances be \(1\) and \(4\), respectively. The group means may be arbitrary. Find the distribution, expectation, and variance of the pooled sum \(W\), and compare it with a common-scale \(\chi^2_3\) law.

The first group has one residual dimension, so \(S_1\sim\chi^2_1\). The second has two residual dimensions and variance \(4\), so \(S_2/4\sim\chi^2_2\), or \(S_2\sim4\chi^2_2\). Independence gives \(W\overset{d}{=}\chi^2_1+4\chi^2_2\), with independent terms.

Using \(E[\chi^2_k]=k\) and \(\operatorname{Var}(\chi^2_k)=2k\), \[ E[W]=1+4(2)=9, \qquad \operatorname{Var}(W)=2(1)^2+4^2\{2(2)\}=2+64=66. \] If \(W\) were a common-scale multiple \(c\chi^2_3\), matching its expectation would require \(3c=9\), hence \(c=3\). That law would have variance \(2c^2(3)=54\), not 66. Thus this pooled sum does not have the usual single-scale chi-square distribution with the total of three residual degrees of freedom.

Unequal variances do not invalidate every pooled chi-square law: if a group has size one, its residual degrees of freedom are zero and its contribution is identically zero, whatever its variance. For example, a size-two group with variance \(1\) and an independent size-one group with variance \(4\) give \(W\sim\chi^2_1\). It is the variances of groups that actually contribute residual dimensions that matter.

When Are the Contributions Independent?

The independence in the theorem comes from independence of the group observation vectors. Since each sum \(S_g\) depends only on its own vector, independent vectors give independent sums, whether or not the data are normal. Normality is needed for the exact chi-square distributions, not for this elementary independence argument.

There is also a geometric route to independence when all observations are jointly normal: orthogonal projected coordinates are independent if their covariance is spherical, \(\sigma^2 I\). But orthogonal residual subspaces alone do not guarantee independence under arbitrary dependence or covariance. In the setting here, independent group vectors with a common variance ensure both the needed separation across groups and the common scale for the pooled chi-square result.

A useful consequence under the common-variance model is that \(E[S_g]=(n_g-1)\sigma^2\), so \(E[W]=r\sigma^2\). When \(r>0\), \(W/r\) is therefore an unbiased estimator of the common variance. The denominator is the total residual rank, not the number of groups and not the total sample size.

Common Mistakes

Using the wrong degrees of freedom. Estimating one mean in each group removes one dimension per group. The pooled residual degrees of freedom are \(\sum_g(n_g-1)=N-m\), not \(N-1\).
Pooling by averaging the sums of squares. The pooled sum is \(S_1+\cdots+S_m\). If a variance estimate is desired, divide that sum by the total residual degrees of freedom afterward.
Assuming orthogonality alone gives independence. Independence of group contributions follows here from independent group vectors; the chi-square conclusion additionally uses normality and a common variance for contributing dimensions.
Claiming unequal variances always prevent a chi-square law. A group with one observation contributes no residual dimensions. More carefully: when groups with positive residual degrees of freedom have unequal variances, the pooled sum is generally a weighted sum of chi-square variables rather than one common-scale chi-square.

Key takeaway: Under independent normal groups with common variance, pooling adds both the within-group sums of squares and their projection ranks. The result satisfies \(W/\sigma^2\sim\chi^2_{N-m}\) when \(N-m>0\); independence and a common scale are essential parts of this exact statement.

Check Your Understanding

Use the projection and distribution arguments above to answer these questions.

Three groups have sizes \(4,5,\) and \(1\). What is the pooled within-group residual rank, and which group contributes no within-group sum of squares?
Why does independence of the group observation vectors imply independence of their within-group sums of squares?
Under the common-variance normal model, what is the distribution of \(W/\sigma^2\) if the group sizes are \(2,3,\) and \(6\)?
Which assumption gives the exact chi-square distribution, and which assumption ensures independence across groups?
For independent normal groups with unequal variances, explain why a group of size one does not change the distribution of the pooled within-group sum.