A Worked Balanced Nested ANOVA

Why a Nested ANOVA Needs More Than One Error Term

Suppose each treatment is applied to several experimental units, and multiple observations are recorded on each unit. Observations from the same unit share that unit, so they do not play the same role as observations from different units. A balanced nested ANOVA separates treatment differences, variation among units receiving the same treatment, and variation among observations on a unit. That separation determines which mean square belongs in each F statistic.

Consider $a$ treatments, $b$ units within each treatment, and $n$ observations per unit. The design is balanced when all three counts are the same across their respective levels. The units are nested because each unit belongs to just one treatment. In the model below, treatments are fixed effects, while units are sampled from populations of units within treatments.

Model and assumptions: Let $Y_{ijk}$ be observation $k$ on unit $j$ receiving treatment $i$, where $i=1,\ldots,a$, $j=1,\ldots,b$, and $k=1,\ldots,n$. Use the model $$ Y_{ijk}=\mu+\alpha_i+u_{j(i)}+\varepsilon_{ijk}. $$ Here $\mu$ is the overall mean, $\alpha_i$ is the fixed effect of treatment $i$, $u_{j(i)}$ is the random effect of unit $j$ nested within treatment $i$, and $\varepsilon_{ijk}$ is residual error. Assume $\sum_i\alpha_i=0$, independent $u_{j(i)}\sim N(0,\sigma^2_u)$, and independent $\varepsilon_{ijk}\sim N(0,\sigma^2)$, with the unit effects independent of the residuals. The observations are independent conditional on their unit effects, and the variances are common across treatments. The balanced layout and these distributional assumptions support the classical exact F tests described below.

The treatment hypothesis is $H_0:\alpha_1=\cdots=\alpha_a=0$, equivalent to equal treatment means, against the alternative that at least one treatment mean differs. The nested-unit hypothesis is $H_0:\sigma^2_u=0$, against $H_a:\sigma^2_u>0$. The first asks about treatment means; the second asks whether units within treatments contribute variation beyond residual variation among observations on a unit.

Partitioning the Variation

Write $\bar{Y}_{i..}$ for the mean of all $bn$ observations under treatment $i$, $\bar{Y}_{ij.}$ for the mean of the $n$ observations on unit $j$ within treatment $i$, and $\bar{Y}_{...}$ for the grand mean. The total variation around the grand mean separates into three sums of squares.

Definition: The treatment sum of squares measures variation among treatment means; the nested-unit sum of squares measures variation among unit means within treatments; and the residual sum of squares measures variation among observations within units. $$ SS_A=bn\sum_{i=1}^{a}(\bar{Y}_{i..}-\bar{Y}_{...})^2,\qquad SS_{B(A)}=n\sum_{i=1}^{a}\sum_{j=1}^{b}(\bar{Y}_{ij.}-\bar{Y}_{i..})^2, $$ $$ SS_E=\sum_{i=1}^{a}\sum_{j=1}^{b}\sum_{k=1}^{n}(Y_{ijk}-\bar{Y}_{ij.})^2,\qquad SS_T=SS_A+SS_{B(A)}+SS_E. $$

The total number of observations is $abn$, so total degrees of freedom are $abn-1$. The treatment, nested-unit, and residual degrees of freedom are respectively $a-1$, $a(b-1)$, and $ab(n-1)$. Their sum is $abn-1$, as required. Each mean square is its sum of squares divided by its degrees of freedom: $MS_A=SS_A/(a-1)$, $MS_{B(A)}=SS_{B(A)}/[a(b-1)]$, and $MS_E=SS_E/[ab(n-1)]$.

Why the F Tests Use Different Denominators

Under the model, the expected mean squares are $$ E(MS_E)=\sigma^2,\qquad E(MS_{B(A)})=\sigma^2+n\sigma^2_u, $$ $$ E(MS_A)=\sigma^2+n\sigma^2_u+ \frac{bn\sum_{i=1}^{a}\alpha_i^2}{a-1}. $$ These expressions explain the denominators. When treatment means are equal, the extra treatment term in $E(MS_A)$ is zero, so $MS_A$ and $MS_{B(A)}$ have the same expected value. When the nested-unit variance is zero, $MS_{B(A)}$ and $MS_E$ have the same expected value.

Test rule: For the classical balanced model with normal random unit effects and residuals, use $$ F_A=\frac{MS_A}{MS_{B(A)}}\quad\text{with }df=(a-1,\ a(b-1)) $$ to test equal treatment means, and $$ F_{B(A)}=\frac{MS_{B(A)}}{MS_E}\quad\text{with }df=(a(b-1),\ ab(n-1)) $$ to test whether the nested-unit variance is zero. Each p-value is the upper-tail probability under its stated F distribution.

A short algebraic argument shows why the three sums of squares partition the total. For any observation, add and subtract its unit mean and treatment mean: $$ Y_{ijk}-\bar{Y}_{...}=(Y_{ijk}-\bar{Y}_{ij.})+(\bar{Y}_{ij.}-\bar{Y}_{i..})+(\bar{Y}_{i..}-\bar{Y}_{...}). $$ When this expression is squared and summed, cross-products vanish. Within each unit the residuals sum to zero; within each treatment the deviations of unit means from the treatment mean sum to zero; and across treatments the treatment deviations from the grand mean sum to zero. The remaining squared terms are exactly $SS_E$, $SS_{B(A)}$, and $SS_A$. This also clarifies why observations within a unit contribute to residual variation rather than counting as additional independent units.

Worked ANOVA Examples

Worked Example: Equal Treatment Means but Variable Units

A hypothetical study has $a=3$ treatments, $b=3$ units per treatment, and $n=2$ observations per unit. The unit means, in measurement units, are $9,10,11$ for each treatment. For every listed unit mean $m$, the two observations are $m-1$ and $m+1$. Thus each treatment mean and the grand mean are 10.

Treatment variation. Since all three treatment means are 10, $SS_A=2(3)[(10-10)^2+(10-10)^2+(10-10)^2]=0$.

Nested-unit variation. Within each treatment, the unit means differ from the treatment mean by $-1,0,1$. Therefore $SS_{B(A)}=2[3(1^2+0^2+1^2)]=12$.

Residual variation. Each unit contributes $(-1)^2+1^2=2$, and there are nine units. Hence $SS_E=9(2)=18$. The total is $SS_T=0+12+18=30$.

The degrees of freedom are $df_A=2$, $df_{B(A)}=3(3-1)=6$, $df_E=3(3)(2-1)=9$, and $df_T=17$. The mean squares are $MS_A=0/2=0$, $MS_{B(A)}=12/6=2$, and $MS_E=18/9=2$. Thus $F_A=0/2=0$, with $df=(2,6)$ and $p=1$. Also, $F_{B(A)}=2/2=1$, with $df=(6,9)$ and $p\approx0.480$. At $\alpha=0.05$, neither test is significant: these data do not provide evidence of treatment-mean differences or of a positive nested-unit variance.

Worked Example: A Treatment Result Just Below 0.05

In a second hypothetical study, there are $a=2$ treatments, $b=4$ units per treatment, and $n=2$ observations per unit. Let $c=\sqrt{3}/2$ measurement units. The four unit means under treatment 1 are $c-\sqrt{3}/2,\ c-\sqrt{3}/2,\ c+\sqrt{3}/2,\ c+\sqrt{3}/2$; under treatment 2 they are the negatives of those four means. Each unit has two observations equal to its mean minus 1 and its mean plus 1. Treatment means are therefore $c$ and $-c$, and the grand mean is zero. This construction specifies concrete hypothetical observations and gives equal replication throughout.

The treatment sum of squares is $SS_A=bn[c^2+(-c)^2]=8(2c^2)=12$, since $c^2=3/4$. Within each treatment, the four unit means deviate from their treatment mean by two values of $-\sqrt{3}/2$ and two of $+\sqrt{3}/2$. Thus $SS_{B(A)}=2[8(3/4)]=12$. Each of the eight units has two observations one unit below and above its mean, so $SS_E=8(1^2+1^2)=16$. The total is 40.

The degrees of freedom are $1$, $6$, and $8$, respectively, for treatment, nested units, and residual error. The mean squares are $MS_A=12$, $MS_{B(A)}=12/6=2$, and $MS_E=16/8=2$. The treatment statistic is $F_A=12/2=6$, with $df=(1,6)$ and upper-tail $p\approx0.0498$. Since $0.0498<0.05$, reject equal treatment means at $\alpha=0.05$. The nested-unit statistic is $F_{B(A)}=2/2=1$, with $df=(6,8)$; its upper-tail p-value is greater than 0.05, so these observations do not provide evidence that the nested-unit variance is positive.

This example shows why rounding a p-value too early can change a decision. The value is just below 0.05, not above it.

Worked Example: A Large F Statistic That Is Not Significant

A third hypothetical study has two treatments, two units per treatment, and two observations per unit. Under treatment 1, the unit observations are $(4,6)$ and $(5,7)$; under treatment 2, they are $(6,8)$ and $(7,9)$. The unit means are 5, 6, 7, and 8; treatment means are 5.5 and 7.5, and the grand mean is 6.5.

The treatment sum of squares is $SS_A=bn[(5.5-6.5)^2+(7.5-6.5)^2]=4(1+1)=8$. Each unit mean is 0.5 from its treatment mean, so $SS_{B(A)}=n[4(0.5^2)]=2$. Each unit's two observations are one unit below and above its mean, giving $SS_E=4(1^2+1^2)=8$. Therefore $SS_T=18$.

The degrees of freedom are $df_A=1$, $df_{B(A)}=2$, $df_E=4$, and $df_T=7$. The mean squares are $MS_A=8$, $MS_{B(A)}=2/2=1$, and $MS_E=8/4=2$. Thus $F_A=8/1=8$, with $df=(1,2)$ and $p\approx0.1056$; $F_{B(A)}=1/2=0.5$, with $df=(2,4)$ and $p=0.640$. Neither test is significant at 0.05. In particular, the treatment statistic is numerically large, but its p-value must be interpreted using its degrees of freedom; here it does not cross the 0.05 threshold.

Reading the Table and Avoiding Common Errors

An ANOVA table makes the calculations easy to audit. Check that the sums of squares add to the total, that the degrees of freedom add to $abn-1$, and that every mean square equals its sum of squares divided by the corresponding degrees of freedom.

Source	Sum of squares	Degrees of freedom	Mean square	F denominator
Treatment	$SS_A$	$a-1$	$SS_A/(a-1)$	$MS_{B(A)}$
Units within treatment	$SS_{B(A)}$	$a(b-1)$	$SS_{B(A)}/[a(b-1)]$	$MS_E$
Residual	$SS_E$	$ab(n-1)$	$SS_E/[ab(n-1)]$	Not applicable
Total	$SS_T$	$abn-1$	Not applicable	Not applicable

A frequent mistake is to use $MS_E$ as the denominator for the treatment test merely because it is called the error mean square. That comparison ignores variation among units within treatments. In this model, the treatment statistic uses $MS_{B(A)}$; the unit-within-treatment statistic uses $MS_E$. Another mistake is to treat all $abn$ observations as independent treatment replicates. Treatment assignment occurs at the unit level in this design, so the unit count—not the number of observations within units—determines the treatment denominator and its degrees of freedom.

Finally, the model assumptions require attention. The layout should truly have the same number of units per treatment and observations per unit. Units should be sampled independently, residuals should be reasonably normal with a common variance, and residuals within a unit should not have unmodelled dependence beyond the shared unit effect. A severe departure from these conditions can make the classical F reference distributions unreliable. A nonsignificant result is not proof that all means or variances are identical; it means the data do not provide sufficient evidence against the specified null at the chosen significance level.

Key takeaway: In a balanced nested ANOVA with fixed treatments and random units nested within them, use $MS_{B(A)}$ to test treatment means and $MS_E$ to test nested-unit variation. Correct sums of squares, degrees of freedom, and denominators are all essential to a valid conclusion.

Check Your Understanding

Use the model and test rules in this tutorial to answer the following.

For $a=4$, $b=3$, and $n=2$, find the treatment, nested-unit, residual, and total degrees of freedom.
Why is $MS_{B(A)}$, rather than $MS_E$, the denominator for the treatment test in this model?
If $SS_{B(A)}=24$ and $a=3,b=4$, calculate $MS_{B(A)}$.
A treatment test has $F_A=0.8$ and $df=(2,9)$. What null hypothesis does it test, and which mean square is its denominator?
Explain why recording more observations on each unit does not create more independent experimental units.

Source	Sum of squares	Degrees of freedom	Mean square	F denominator
Treatment	\(SS_A\)	\(a-1\)	\(SS_A/(a-1)\)	\(MS_{B(A)}\)
Units within treatment	\(SS_{B(A)}\)	\(a(b-1)\)	\(SS_{B(A)}/[a(b-1)]\)	\(MS_E\)
Residual	\(SS_E\)	\(ab(n-1)\)	\(SS_E/[ab(n-1)]\)	Not applicable
Total	\(SS_T\)	\(abn-1\)	Not applicable	Not applicable

What You'll Learn