Proving the ANOVA Variation Decomposition

Why Total Variation Splits Into Two Parts

In a one-way ANOVA, a quantitative response is recorded for individuals in several groups. The total spread of the observations around the grand mean can be described in two complementary ways: how far the group means are from the grand mean, and how far observations are from their own group means. The key result is that these two sources of variation add exactly to the total.

This equality is not an approximation and does not depend on a probability model. It follows from algebra and the definition of each group mean. Understanding the derivation makes clear why the decomposition works, why group sizes matter, and why the between-group and within-group sums of squares measure distinct parts of the observed variation.

Definition: Suppose there are $k$ groups. Group $j$ contains $n_j$ observations, written $x_{ij}$ for observation $i$ in group $j$, where $i=1,\ldots,n_j$. The group mean is $\bar{x}_j$, the total sample size is $N=\sum_{j=1}^{k}n_j$, and the grand mean is $\bar{x}=\frac{1}{N}\sum_{j=1}^{k}\sum_{i=1}^{n_j}x_{ij}$. The total sum of squares is $SS_{\text{Total}}=\sum_{j=1}^{k}\sum_{i=1}^{n_j}(x_{ij}-\bar{x})^2$. The between-group sum of squares is $SS_{\text{Between}}=\sum_{j=1}^{k}n_j(\bar{x}_j-\bar{x})^2$. The within-group sum of squares is $SS_{\text{Within}}=\sum_{j=1}^{k}\sum_{i=1}^{n_j}(x_{ij}-\bar{x}_j)^2$.

The between-group expression weights each squared distance by the size of its group. That weight reflects how many observations share that group mean: a group mean farther from the grand mean contributes more, and a group containing more observations contributes more as well. The within-group expression adds the squared distances from individual observations to their own group means.

These definitions describe a finite data set partitioned into $k$ non-overlapping groups, with a quantitative observation for each individual. They require no assumption of normality, equal variances, random sampling, or equal group sizes. Such assumptions can matter for later inference using ANOVA, but the variation identity itself is purely algebraic.

The Decomposition: A Complete Derivation

For every observation, add and subtract its group mean. Its deviation from the grand mean can then be written as the sum of two deviations: its deviation from its group mean, and the deviation of its group mean from the grand mean.

x_{ij}-\bar{x}=(x_{ij}-\bar{x}_j)+(\bar{x}_j-\bar{x})

Squaring this expression creates two squared terms and a cross-product term. Summing over all observations gives the total sum of squares:

\begin{aligned} SS_{\text{Total}} &=\sum_{j=1}^{k}\sum_{i=1}^{n_j} \left[(x_{ij}-\bar{x}_j)+(\bar{x}_j-\bar{x})\right]^2\\ &=\sum_{j=1}^{k}\sum_{i=1}^{n_j}(x_{ij}-\bar{x}_j)^2 +\sum_{j=1}^{k}\sum_{i=1}^{n_j}(\bar{x}_j-\bar{x})^2\\ &\quad+2\sum_{j=1}^{k}\sum_{i=1}^{n_j} (x_{ij}-\bar{x}_j)(\bar{x}_j-\bar{x}). \end{aligned}

The first term is $SS_{\text{Within}}$ by definition. In the second term, $\bar{x}_j-\bar{x}$ is constant across the $n_j$ observations in group $j$, so the group’s contribution is $n_j(\bar{x}_j-\bar{x})^2$. Summing those contributions gives $SS_{\text{Between}}$. It remains to show that the cross-product term is zero.

\begin{aligned} 2\sum_{j=1}^{k}\sum_{i=1}^{n_j} (x_{ij}-\bar{x}_j)(\bar{x}_j-\bar{x}) &=2\sum_{j=1}^{k}(\bar{x}_j-\bar{x}) \sum_{i=1}^{n_j}(x_{ij}-\bar{x}_j)\\ &=2\sum_{j=1}^{k}(\bar{x}_j-\bar{x})\cdot 0\\ &=0. \end{aligned}

The inner sum is zero for every group because deviations from a group’s own mean sum to zero: $\sum_{i=1}^{n_j}(x_{ij}-\bar{x}_j)=\sum_i x_{ij}-n_j\bar{x}_j=0$. Thus the cross-product terms cancel within each group, and the decomposition follows.

Theorem: For observations partitioned into $k$ groups, the total sum of squares equals the between-group sum of squares plus the within-group sum of squares: $$ SS_{\text{Total}}=SS_{\text{Between}}+SS_{\text{Within}}. $$ The identity holds for balanced or unequal group sizes.

Worked Examples

Worked Example: Two Equal-Sized Groups

Suppose a hypothetical data set has response values 4 and 6 in group A, and 8 and 10 in group B. Calculate all three sums of squares and verify the decomposition.

The group means are $\bar{x}_A=(4+6)/2=5$ and $\bar{x}_B=(8+10)/2=9$. The grand mean is $\bar{x}=(4+6+8+10)/4=7$.

The total sum of squares measures each observation’s squared distance from 7:

SS_{\text{Total}}=(4-7)^2+(6-7)^2+(8-7)^2+(10-7)^2 =9+1+1+9=20.

For the between-group sum of squares, each group mean’s squared distance from 7 is multiplied by that group’s size, 2:

SS_{\text{Between}}=2(5-7)^2+2(9-7)^2 =2(4)+2(4)=16.

Within group A, the squared deviations from 5 are $(4-5)^2+(6-5)^2=2$. Within group B, the squared deviations from 9 are $(8-9)^2+(10-9)^2=2$. Therefore $SS_{\text{Within}}=2+2=4$. The two parts add to $16+4=20=SS_{\text{Total}}$.

Worked Example: Unequal Group Sizes

Consider three hypothetical groups: A has values 2, 4, 6; B has values 8, 10; and C has values 5, 7, 9, 11. Verify the decomposition when the group sizes differ.

The group sizes are 3, 2, and 4, and their means are $\bar{x}_A=4$, $\bar{x}_B=9$, and $\bar{x}_C=8$. There are $N=9$ observations with total $62$, so the grand mean is $\bar{x}=62/9$, approximately 6.889.

The total sum of squares can be calculated directly. The sum of the squared observations is $4+16+36+64+100+25+49+81+121=496$. Expanding the definition of $SS_{\text{Total}}$ gives $\sum x_{ij}^2-(\sum x_{ij})^2/N$, so:

SS_{\text{Total}}=496-\frac{62^2}{9} =496-\frac{3844}{9} =\frac{620}{9}\approx68.89.

Now calculate the between-group sum of squares, using the actual size of each group as its weight:

\begin{aligned} SS_{\text{Between}} &=3\left(4-\frac{62}{9}\right)^2 +2\left(9-\frac{62}{9}\right)^2 +4\left(8-\frac{62}{9}\right)^2\\ &=3\left(\frac{-26}{9}\right)^2 +2\left(\frac{19}{9}\right)^2 +4\left(\frac{10}{9}\right)^2\\ &=\frac{2028+722+400}{81} =\frac{3150}{81} =\frac{350}{9}\approx38.89. \end{aligned}

The within-group sum of squares is $8$ for A, $2$ for B, and $20$ for C:

SS_{\text{Within}}= [(2-4)^2+(4-4)^2+(6-4)^2] +[(8-9)^2+(10-9)^2] +[(5-8)^2+(7-8)^2+(9-8)^2+(11-8)^2] =8+2+20=30.

The components sum to $350/9+30=350/9+270/9=620/9$, exactly the total sum of squares. The unequal sizes do not alter the identity, but they do determine the weights in the between-group calculation.

Worked Example: Seeing the Cross-Product Cancel

Consider hypothetical group A values 1, 3, 5 and group B values 4, 8. This example makes the cross-product cancellation visible: individual cross-products need not be zero, even though their sum within each group is.

The group means are $\bar{x}_A=3$ and $\bar{x}_B=6$; the grand mean is $\bar{x}=21/5=4.2$. The total sum of squares is:

SS_{\text{Total}}=(1-4.2)^2+(3-4.2)^2+(5-4.2)^2 +(4-4.2)^2+(8-4.2)^2 =10.24+1.44+0.64+0.04+14.44=26.8.

Within group A, the residuals $x_{ij}-\bar{x}_A$ are $-2,0,2$, and the group-mean offset $\bar{x}_A-\bar{x}$ is $-1.2$. The sum of the cross-products before doubling is $(-2)(-1.2)+(0)(-1.2)+(2)(-1.2)=2.4+0-2.4=0$. For group B, the residuals are $-2,2$ and the offset is $1.8$, giving $(-2)(1.8)+(2)(1.8)=-3.6+3.6=0$.

The within-group sum is $SS_{\text{Within}}=(4+0+4)+(4+4)=16$. The between-group sum is $SS_{\text{Between}}=3(3-4.2)^2+2(6-4.2)^2=3(1.44)+2(3.24)=10.8$. Hence $SS_{\text{Within}}+SS_{\text{Between}}=16+10.8=26.8=SS_{\text{Total}}$. The cancellation occurs across observations in each group, not necessarily observation by observation.

What the Identity Means

The decomposition separates the observed squared deviation of every response from the grand mean into two sources. The between-group component describes variation in the group means around the grand mean. The within-group component describes variation among individuals around their respective group means. Their sum accounts for all of the total squared variation; no residual term is missing.

A sum of squares has units equal to the square of the response’s units. If response values are measured in minutes, each sum of squares is in squared minutes. These quantities are useful building blocks for the later ANOVA calculations, but the identity alone is not a test and does not establish that groups differ in a population. It simply organizes the variation in the observed data.

Common Mistakes

Assuming the result follows just because deviations add. Deviations do add, but squaring a sum produces a cross-product term. The proof must account for it; it disappears because deviations from each group mean sum to zero within that group.
Leaving out group-size weights. The between-group sum is $\sum_j n_j(\bar{x}_j-\bar{x})^2$, not just the unweighted sum of squared distances between the group means and the grand mean.
Using an unweighted average of group means as the grand mean when sizes differ. The grand mean is the mean of all observations, so it weights each group mean by $n_j$: $\bar{x}=\sum_j n_j\bar{x}_j/N$.
Calling the identity a modeling assumption. It is an exact algebraic equality for the data as partitioned. Distributional assumptions may be needed for an inferential procedure, but not for this decomposition.

Key takeaway: Split each observation’s deviation into its within-group deviation and its group’s deviation from the grand mean. After squaring and summing, the cross-product is zero because the within-group deviations sum to zero. Therefore $SS_{\text{Total}}=SS_{\text{Between}}+SS_{\text{Within}}$.

Check Your Understanding

Answer these questions using the definitions and derivation above.

For one observation $x_{ij}$, write its deviation from the grand mean as the sum of a within-group deviation and a group-mean deviation.
Why does the cross-product term sum to zero within group $j$?
A group contains 8 observations and its mean is 2 units above the grand mean. What is this group’s contribution to $SS_{\text{Between}}$?
Does the variation decomposition require equal group sizes or normal distributions? Explain.
Explain why the between-group sum of squares uses $n_j$ as a weight for group $j$.