Why the Error Term Matters
A factorial ANOVA compares variation attributed to specified factors and their interactions with variation not explained by those terms. To interpret an \(F\) statistic, it matters which variation supplies the denominator: the denominator estimates the background error variability against which a factorial effect is judged. In a replicated design, observations within the same factor-level combination provide information about that variability. With only one observation per combination, a full factorial model has no such within-cell information.
This tutorial focuses on choosing that denominator in a replicated fixed-effects factorial model and on the limitation created by no replication. It uses a complete, balanced design: every combination of factor levels is observed the same number of times. This makes the source of residual variation especially clear.
The Model and Its Assumptions
Consider a three-factor fixed-effects design. Factor A has \(a\) levels, factor B has \(b\) levels, and factor C has \(c\) levels. Each combination of levels is observed \(r\) times. Write \(Y_{ijkl}\) for observation \(l\) at A level \(i\), B level \(j\), and C level \(k\). The full model includes all three main effects, all three two-factor interactions, and the three-factor interaction:
The factor levels are treated as fixed: the effects describe the particular levels included in the design. The errors \(\varepsilon_{ijkl}\) are assumed to be independent, normally distributed with mean zero and common variance \(\sigma^2\). Equivalently, each observation has a mean specified by its cell and an error with distribution \(N(0,\sigma^2)\). The design must also have independent experimental units; repeated measurements on one unit do not become independent replicates just because they are recorded more than once.
The full model can be written in terms of the \(abc\) cell means, one for each combination of levels. Each replicate in a cell varies around that cell's mean. The residual error is the pooled variation of these observations around their respective cell means. It is not the variation among cell means: that variation is what the factorial terms describe.
How the Residual Mean Square Is Obtained
Let \(\bar{x}_{ijk\cdot}\) denote the sample mean of the \(r\) observations in cell \((i,j,k)\). For the complete balanced design, the residual sum of squares and its degrees of freedom are
There are \(r\) observations in each cell, and estimating that cell's mean leaves \(r-1\) degrees of freedom for its within-cell deviations. Summing over \(abc\) cells gives \(abc(r-1)\). The residual mean square is therefore
For a factorial term \(T\), the usual test statistic is \(F=MS_T/MS_{\text{res}}\), where \(MS_T=SS_T/df_T\). The numerator degrees of freedom depend on the term; for example, A has \(a-1\) degrees of freedom and the A-by-B interaction has \((a-1)(b-1)\). The denominator degrees of freedom are \(abc(r-1)\) for each of these tests in this full model. The hypotheses for each test concern that term, with the other factorial terms included in the model.
Why This Denominator Works
The numerator measures variation among fitted cell means associated with a particular factorial term. The residual mean square measures within-cell variation. In a balanced, full factorial model, these are distinct components of variation. Under the null hypothesis for a term, its mean square and the residual mean square estimate the same error variance after accounting for their degrees of freedom.
The reason can be seen from the within-cell residuals. In any one cell, subtracting the cell mean gives deviations whose sum is zero. Under the model, the cell mean captures that cell's fixed mean, leaving the random errors as the source of within-cell deviations. With independent normal errors, the residual sum of squares divided by \(\sigma^2\) has a chi-squared distribution with \(abc(r-1)\) degrees of freedom. In the balanced design, it is also independent of the sums of squares for the factorial terms.
Under the null hypothesis for term \(T\), \(MS_T/\sigma^2\) is a chi-squared quantity divided by \(df_T\); \(MS_{\text{res}}/\sigma^2\) is an independent chi-squared quantity divided by \(df_{\text{res}}\). Their ratio therefore has an \(F\) distribution:
This is the justification for using the residual mean square—not a different factorial term's mean square—as the denominator in these tests. The conclusion depends on the model assumptions, including independent, normal errors with a common variance. If those assumptions are not reasonable, the usual \(F\) reference distribution may not be appropriate.
Worked Example: Calculating a Residual Mean Square and an A Test
Suppose a hypothetical balanced \(2\times2\) design has three independent observations in every A-by-B cell. The observations, in measurement units, are:
A1B1: 8, 9, 10; A1B2: 10, 11, 12; A2B1: 12, 13, 14; A2B2: 14, 15, 16. Assume the observations are independent and that a normal, equal-variance error model is reasonable. The repeated observations within cells allow the residual variation to be estimated.
The four cell means are 9, 11, 13, and 15. In each cell, the deviations from its mean are \(-1,0,1\), so that cell contributes \(1+0+1=2\) to the residual sum of squares. Thus
The two A marginal means are 10 and 14, and the grand mean is 12. Each A level contains \(2\times3=6\) observations. The A sum of squares is
A has one degree of freedom, so \(MS_A=48/1=48\). Its test statistic is
At significance level \(\alpha=0.05\), the critical value for \(F(1,8)\) is about 5.318. Since \(48>5.318\), reject the null hypothesis of no A effect in this model. The test uses within-cell variation as its error estimate; it does not use the B or interaction mean square as a substitute denominator.
Worked Example: Using the Same Denominator for Different Terms
Consider a hypothetical balanced \(2\times2\times2\) fixed-effects design with two independent observations per cell. Assume independent errors with a common variance and a normal error distribution. Suppose its full-model ANOVA calculations give the following sums of squares. These values are illustrative; the purpose is to identify and use the error term.
| Source | Sum of squares | Degrees of freedom |
|---|---|---|
| A | 18 | 1 |
| B | 8 | 1 |
| C | 2 | 1 |
| A-by-B | 12 | 1 |
| A-by-C | 6 | 1 |
| B-by-C | 4 | 1 |
| A-by-B-by-C | 20 | 1 |
| Residual | 16 | 8 |
There are \(abc=8\) cells, with \(r=2\) observations per cell. The residual degrees of freedom are \(8(2-1)=8\), so \(MS_{\text{res}}=16/8=2\). Each listed factorial term has one degree of freedom, making its mean square equal to its sum of squares. For example,
All three tests use \(F(1,8)\) as their reference distribution under their respective null hypotheses. At \(\alpha=0.05\), the critical value is about 5.318, so each of these three statistics exceeds it. This comparison does not establish that the effects have the same meaning: each test addresses its own factorial term. It shows that the residual mean square supplies the common estimate of error variance for these tests in this balanced full model.
What Changes Without Replication?
If there is only one observation in each cell, \(r=1\), and the full factorial model has
There is no within-cell deviation to sum, so the full model provides no residual mean square. This is not a matter of choosing a different denominator: the data contain no independent within-cell estimate of random error for that saturated model. Each cell mean is just its single observed value.
For a two-factor design, this problem can be especially visible. With one observation per A-by-B cell, a model containing A, B, and their interaction has enough fitted cell means to reproduce all the observations. A model that omits the interaction may leave residual degrees of freedom, but those residuals combine random error with any unmodelled interaction. They do not independently measure pure within-cell error unless additional assumptions are made. An external estimate of error variance or a justified model restriction can change what analysis is possible, but neither creates replication in the original full factorial model.
Worked Example: Why a One-Observation-Per-Cell Model Has No Error Estimate
A hypothetical \(2\times2\) experiment records one response in each cell: \(Y_{11}=10\), \(Y_{12}=12\), \(Y_{21}=13\), and \(Y_{22}=19\). Assume the intended model includes A, B, and their interaction. There are four observations and four cell means, one for each cell.
The full model fits each cell mean to its single observation. Consequently every fitted residual is zero, and the residual degrees of freedom are \(ab(r-1)=2(2)(1-1)=0\). Dividing a residual sum of squares by zero degrees of freedom cannot produce an estimate of \(\sigma^2\), so the usual full-model \(F\) tests have no independent error denominator.
One way to see what may be lost by omitting the interaction is to calculate the difference of differences:
An additive model without an interaction cannot represent this nonzero difference of differences. Its leftover variation would therefore reflect the omitted interaction as well as random error. Treating that leftover variation as pure error would require an assumption that the interaction is absent; the data with one observation per cell cannot separately check that assumption and estimate the random error.
Common Mistake and Practical Check
Wrong reasoning: “The interaction mean square is available, so I can use it as the error term for the main effects,” or “With no replication, I can simply use the leftover variation after dropping a term as the error estimate.” These choices change the model assumptions rather than selecting the usual denominator.
Correction: In the replicated, balanced, full fixed-effects model described here, use the pooled within-cell residual mean square for tests of factorial terms. It is based on independent variation among replicates within cells. In an unreplicated full model, that source of variation has zero degrees of freedom. A reduced model may produce residual degrees of freedom, but interpreting them as pure error requires a defensible restriction on the terms omitted.
Before reporting a factorial \(F\) test, check that the model has replication, identify the source and degrees of freedom of its residual mean square, and verify that the independence, common-variance, and normal-error assumptions are reasonable for the design and data. State the term being tested and its numerator and denominator degrees of freedom so the reference distribution is clear.
Check Your Understanding
Answer each question using the full fixed-effects model unless the question says otherwise.
- A balanced \(3\times2\times2\) design has four independent replicates in each cell. What are the residual degrees of freedom?
- In a replicated balanced factorial model, what variation does the residual sum of squares measure, and why is it used as the error denominator?
- A factorial term has \(df_T=2\), \(MS_T=15\), and the residual mean square is 3 with 24 degrees of freedom. Calculate the test statistic and state its reference \(F\) distribution under the null.
- Why can a full factorial model with one observation per cell fit the observed cell means but still lack an estimate of error variance?
- In an unreplicated two-factor design, what additional assumption is needed before residual variation from a model omitting the interaction could be treated as pure error?