Worked One-Way Example: Compute Independent Components

From Grouped Observations to Independent Components

A one-way layout records a quantitative response in several groups. Its total variation separates into variation among the group means and variation among observations within their groups. In a Gaussian model, these components are not merely an accounting identity: under specified conditions, they are independent quadratic forms. A reliable calculation therefore checks both the arithmetic decomposition and the geometry of the projections that produce it.

Let there be $k$ nonempty groups, with $n_j$ observations in group $j$, and let $n=\sum_{j=1}^k n_j$. Write the observations as $Y_{ji}$, where $i=1,\ldots,n_j$. Define the group mean $\bar Y_j=n_j^{-1}\sum_iY_{ji}$, the grand mean $\bar Y=n^{-1}\sum_j\sum_iY_{ji}$, and the following sums of squares.

Definition: The corrected total sum of squares, between-group sum of squares, and within-group sum of squares are $$ SS_T=\sum_{j=1}^k\sum_{i=1}^{n_j}(Y_{ji}-\bar Y)^2,\qquad SS_B=\sum_{j=1}^k n_j(\bar Y_j-\bar Y)^2,\qquad SS_W=\sum_{j=1}^k\sum_{i=1}^{n_j}(Y_{ji}-\bar Y_j)^2. $$ The first measures variation about the grand mean, the second variation of the group means about the grand mean (weighted by group size), and the third variation about each group’s own mean.

The group-size weights in $SS_B$ are essential when group sizes differ. The projection view makes the same definition precise in matrix form. Stack the observations in a vector $Y\in\mathbb R^n$. Let $P_G$ project onto the space of vectors that are constant within each group, and let $P_0=\mathbf 1\mathbf 1'/n$ project onto the constant-vector space. Then set $P_B=P_G-P_0$ and $P_W=I-P_G$.

The Decomposition and Its Projection Ranks

The group-constant space has dimension $k$, since one value can be chosen independently for each nonempty group. The constant-vector space is a one-dimensional subspace of it. Thus the between-group space has dimension $k-1$, and the within-group space, the orthogonal complement of the group-constant space, has dimension $n-k$.

Result: The matrices $P_0$, $P_B$, and $P_W$ are mutually orthogonal projection matrices. Their ranks are $1$, $k-1$, and $n-k$, respectively, and $$ I=P_0+P_B+P_W,\qquad I-P_0=P_B+P_W. $$ Consequently, $SS_T=Y'(I-P_0)Y$, $SS_B=Y'P_BY$, $SS_W=Y'P_WY$, and $SS_T=SS_B+SS_W$.

Here is a direct verification of the key identities. Both $P_G$ and $P_0$ are symmetric idempotent matrices, and $P_GP_0=P_0P_G=P_0$, since every constant vector is group-constant. It follows that $P_B$ is symmetric and $(P_G-P_0)^2=P_G-2P_0+P_0=P_B$, so it too is a projection. The matrix $P_W=I-P_G$ is a projection as well. Their pairwise products vanish: $P_0P_B=P_0P_G-P_0^2=0$, $P_0P_W=P_0-P_0P_G=0$, and $P_BP_W=(P_G-P_0)(I-P_G)=0$. Their ranks follow from the dimensions just given; in particular, $\operatorname{rank}(P_B)=k-1$ and $\operatorname{rank}(P_W)=n-k$.

For the sum-of-squares identity, insert and subtract each observation’s group mean: $Y_{ji}-\bar Y=(Y_{ji}-\bar Y_j)+(\bar Y_j-\bar Y)$. On squaring and summing, the cross term is zero because $\sum_i(Y_{ji}-\bar Y_j)=0$ within every group. The two remaining terms are exactly $SS_W$ and $SS_B$. This also verifies that the matrix quadratic forms match the grouped-data definitions: $P_GY$ replaces each observation by its group mean, while $P_0Y$ replaces every observation by the grand mean.

When the Components Are Independent

The arithmetic identity holds for any observations. Independence is a separate distributional claim and requires Gaussian errors with spherical covariance. Consider the fixed one-way Gaussian model $Y\sim N(\mu,\sigma^2I)$, where $\mu$ is constant within each group, $\sigma^2>0$, and all groups are nonempty. This is the common-variance model with independent normal observations; the group means in $\mu$ need not be equal.

Theorem: Under that model, $SS_B/\sigma^2$ and $SS_W/\sigma^2$ are independent. The within-group component has a central chi-square distribution with $n-k$ degrees of freedom. The between-group component has a noncentral chi-square distribution with $k-1$ degrees of freedom and noncentrality parameter $\lambda=\mu'P_B\mu/\sigma^2$. If all group means are equal, then $\lambda=0$, so the between-group component is central as well.

To justify the result, choose orthonormal bases for the ranges of $P_0$, $P_B$, and $P_W$. Their ranges are mutually orthogonal and together span $\mathbb R^n$, so combining these bases gives an orthogonal matrix $U$. The coordinates of $U'Y$ are independent normal random variables with variance $\sigma^2$: their covariance matrix is $U'(\sigma^2I)U=\sigma^2I$, and jointly normal coordinates with zero covariance are independent. The coordinates corresponding to $P_B$ and $P_W$ therefore form independent blocks. The squared lengths of those blocks, divided by $\sigma^2$, are $SS_B/\sigma^2$ and $SS_W/\sigma^2$.

Because $\mu$ is group-constant, $P_W\mu=0$. The within-group coordinates consequently have zero means, giving a central chi-square with one degree of freedom for each of the $n-k$ basis vectors. The between-group coordinates have squared standardized means summing to $\mu'P_B\mu/\sigma^2$, giving the stated noncentral chi-square. Under equal group means, $\mu$ is constant overall, so $P_B\mu=0$ and $\lambda=0$. This establishes the distributional claims and their independence. $\square$

Worked Calculations

Worked Example: Equal Group Sizes

Three groups have observations $A=(4,6)$, $B=(5,7)$, and $C=(6,8)$. Compute all three sums of squares and check the ranks.

The group means are $\bar Y_A=5$, $\bar Y_B=6$, and $\bar Y_C=7$. The total is $4+6+5+7+6+8=36$, so $n=6$ and $\bar Y=36/6=6$. Thus $SS_B=2(5-6)^2+2(6-6)^2+2(7-6)^2=2+0+2=4$.

Within each group the two deviations from its mean are $-1$ and $1$, so each group contributes $1+1=2$, giving $SS_W=2+2+2=6$. Directly from the six observations, $SS_T=(-2)^2+0^2+(-1)^2+1^2+0^2+2^2=10$. The check is $SS_B+SS_W=4+6=10=SS_T$.

Here $k=3$, so the projection ranks are $1$ for the grand mean, $k-1=2$ for between groups, and $n-k=3$ for within groups. They sum to $1+2+3=6=n$; the corrected total has rank $n-1=5$, matching the two component ranks $2+3=5$. These data calculations do not by themselves establish a Gaussian model. If the observations arise from independent normal errors with common variance and group-specific means, however, the theorem gives independent between- and within-group components.

Worked Example: Unequal Group Sizes

Take $A=(2,4)$, $B=(5,6,7)$, and $C=(8,10)$, with group sizes $2,3,2$. Find the sums of squares and confirm the rank calculation.

The group means are $3,6,9$. The total sum is $6+18+18=42$, so $n=7$ and $\bar Y=42/7=6$. Weighting by the actual group sizes gives $SS_B=2(3-6)^2+3(6-6)^2+2(9-6)^2=18+0+18=36$.

The within-group contributions are $SS_{W,A}=(2-3)^2+(4-3)^2=2$, $SS_{W,B}=(5-6)^2+(6-6)^2+(7-6)^2=2$, and $SS_{W,C}=(8-9)^2+(10-9)^2=2$. Hence $SS_W=6$. Directly, the squared deviations from 6 are $16,4,1,0,1,4,16$, which sum to $42$. Therefore $SS_T=42=36+6=SS_B+SS_W$.

With $k=3$ and $n=7$, the between- and within-group ranks are $2$ and $4$; together with the grand-mean rank $1$, they sum to $7$. The observed group means differ, but this does not determine the unknown model means. Under the Gaussian group-means model, the within-group component is central chi-square after division by $\sigma^2$; the between-group component is central only if the model means are equal.

Worked Example: Equal Observed Means Do Not Remove Within-Group Variation

Suppose $A=(2,4)$, $B=(1,5)$, and $C=(0,6)$. Calculate the components and explain what their values do and do not show.

Each group mean is $3$, so the grand mean is also $3$. The between-group sum is $SS_B=2(3-3)^2+2(3-3)^2+2(3-3)^2=0$. The within-group contributions are $2$ for $A$, $8$ for $B$, and $18$ for $C$, giving $SS_W=28$. The squared deviations of the six observations from the grand mean are $1,1,4,4,9,9$, so $SS_T=28=0+28$.

The ranks remain $2$ between groups and $3$ within groups, because ranks depend on the group layout, not on the observed values. In particular, an observed between-group sum of zero does not mean that the between-group projection has rank zero. Nor does it imply that the random between-group component is always zero when the population group means are equal: under that null model it is a central chi-square component with $k-1=2$ degrees of freedom, which can take positive values in other samples.

Checking the Conditions and Avoiding a Common Pitfall

Before interpreting the components as independent, check the setup: group membership is fixed, every group is represented, observations have a joint normal distribution with mean constant within each group, and their covariance is $\sigma^2I$ for a common $\sigma^2>0$. The projections must also be the mutually orthogonal one-way projections above. Under these conditions the distributional result applies. The algebraic identity $SS_T=SS_B+SS_W$ alone does not imply independence.

A common error is to argue, “The between-group and within-group sums add to the total, so they must be independent.” That reasoning is wrong: an additive identity is deterministic, while independence is a property of a joint probability distribution. The correction is to verify the Gaussian spherical model and orthogonal projection structure. Under those conditions the two projected Gaussian coordinate blocks are independent; without them, the same numerical decomposition can still hold, but the independence conclusion need not follow.

Key takeaway: Calculate the group means and weighted between-group sum carefully, check the within-group residual sum, and verify that the ranks add correctly. Then assess independence from the model and orthogonality conditions—not from the sum-of-squares identity.

Check Your Understanding

Use the definitions, projection ranks, and Gaussian conditions above to answer the following.

Why must the between-group sum use $n_j$ as the weight for group $j$?
For $k=4$ groups and $n=10$ observations, what are the ranks of $P_0$, $P_B$, and $P_W$?
If all model group means are equal, what is the noncentrality parameter for $SS_B/\sigma^2$, and what is its degrees of freedom?
Does $SS_T=SS_B+SS_W$ by itself prove that $SS_B$ and $SS_W$ are independent? Explain.
In a one-way layout with unequal group sizes, which feature of the between-group formula must be retained to calculate it correctly?