Why Audit an ANOVA Table?
An ANOVA table is a compact report of a model comparison, not a complete description of the analysis. A row labelled “A” might test A before any other term is entered, after other main effects, or conditional on the remaining terms in the model. With unbalanced data, those tests can differ. Coding can also affect what a row labelled “main effect” means when an interaction is present.
A careful audit therefore asks more than whether a reported F statistic looks plausible. It checks the fitted model, the sums-of-squares convention, the order of terms where relevant, the coding and contrast definitions, and the degrees of freedom. Together, these reveal the comparison and null hypothesis behind each test.
State the Model and Its Assumptions First
For a two-factor fixed-effects model with factors A and B, including their interaction, a cell-means form is
Here, \(Y_{ijk}\) is the response for observation \(k\) in the A-level \(i\), B-level \(j\) cell, and \(\mu_{ij}\) is that cell’s population mean. The model assumes independent errors with mean zero and a common variance \(\sigma^2\); normal errors are assumed for the usual exact small-sample F tests. Equivalently, the fitted means can be represented using main-effect and interaction terms with suitable constraints or coding.
A table alone cannot verify these assumptions. Before interpreting its tests, check that the observations are independent according to the study design, and examine residuals for unusual patterns, unequal spread, and serious departures from normality. Also check that the fitted model matches the question and that its parameters are identifiable. The worked examples below suppose these checks support the stated models and that every relevant factor-level combination has observations.
What a Sum of Squares and an F Row Represent
For a linear model, residual sum of squares (SSE) measures the squared discrepancies between observed and fitted responses. A term’s sum of squares measures a reduction in residual variation attributed to a specified model comparison. Different conventions define that comparison differently. The F statistic for a test of a term with \(q\) degrees of freedom is
The numerator is the term’s mean square; the denominator is the residual mean square from the stated full model. Under the null hypothesis and the model assumptions, this statistic has an F distribution with \(q\) numerator degrees of freedom and \(df_{\text{error}}\) denominator degrees of freedom. The p-value is the probability, under that null model, of an F statistic at least as large as the observed one.
A Short Derivation: The Partial F Test
The logic behind many term tests can be seen by comparing a reduced model with a full model. Suppose the reduced model is nested within the full model, the full model has \(df_{\text{error}}\) residual degrees of freedom, and adding the tested terms adds \(q\) independent parameters. Write their residual sums of squares as \(SSE_R\) and \(SSE_F\), respectively. Because the full model can reproduce every fit available to the reduced model, \(SSE_R\geq SSE_F\). The reduction is \(SS_{\text{added}}=SSE_R-SSE_F\). Under the null that the added parameters do not improve the mean model, this reduction divided by \(q\) and the full-model MSE estimate the same error variance. Their ratio gives
This derivation explains why identifying the reduced model is essential. Two rows can both be labelled A but correspond to different reduced models, and so answer different questions.
Worked Example: Auditing Sequential Sums of Squares
Worked Example: Auditing Sequential Sums of Squares
Suppose a hypothetical experiment compares two assay protocols (A) across three incubation schedules (B), using 30 independent experimental units. All six combinations are represented. The scientific team fits A, B, and A-by-B interaction. Residual checks are satisfactory, and the output specifies Type I (sequential) sums of squares in the order A, B, A:B.
The relevant output is:
| Source | df | SS | F |
|---|---|---|---|
| A | 1 | 90 | 9.00 |
| B | 2 | 40 | 2.00 |
| A:B | 2 | 30 | 1.50 |
| Error | 24 | 240 |
Check the model dimensions. A has two levels, so its degrees of freedom are \(2-1=1\). B has three levels, so its degrees of freedom are \(3-1=2\). Their interaction has \(1\times2=2\) degrees of freedom. A full factorial model therefore uses \(1+2+2=5\) model degrees of freedom. With 30 observations, the residual degrees of freedom are \(30-6=24\), since the full model estimates six cell means. These checks match the table.
Check the denominator. The error mean square is \(MSE=240/24=10\). For A, \(F=(90/1)/10=9.00\). For B, \(F=(40/2)/10=2.00\). For the interaction, \(F=(30/2)/10=1.50\). Each reported F statistic agrees with its SS, df, and the common error mean square.
Read the convention and order. A’s 90 is entered first, so it measures the reduction in residual SS when A is added before B and A:B. B’s 40 measures the additional reduction when B is added after A. The interaction’s 30 measures the additional reduction when A:B is added after both main effects. Thus the A row is not automatically an A test adjusted for B and the interaction. If the term order changed, the sequential sums of squares could change, especially with unequal cell sizes.
The table does not report p-values here, so the F values alone do not tell us whether any test is statistically significant at a chosen \(\alpha\). It does establish the tests’ sequential comparisons, provided the stated order and model are correct.
Type II, Type III, and the Hypothesis Behind a Row
Type II sums of squares test a term after other terms that are not higher-order relatives of it. In a two-factor model, a Type II main-effect test adjusts for the other main effect but does not use the interaction as an adjustment for that main effect. The interaction test is adjusted for both main effects. Type III sums of squares test each term after the other terms in the fitted model. In an unbalanced factorial model with an interaction, these procedures can give different main-effect tests.
The word “adjusted” does not by itself state the full null hypothesis. The factor coding and the contrasts used for a test matter, especially for Type III tests when interactions are present. For example, with treatment coding, a main-effect coefficient for A in a model containing A:B describes an A difference at the reference level of B. A Type III test of that A coefficient tests whether this reference-level difference is zero. With sum-to-zero coding, the main-effect coefficients instead express effects relative to the chosen constraint; a corresponding Type III test can represent an equal-weight marginal comparison across B levels. Always verify the software’s actual contrast definition rather than infer the hypothesis from the row label alone.
Worked Example: Comparing Type II and Type III Output
A second hypothetical study compares two treatment protocols (A) at three clinic settings (B), with unequal numbers of independent participants in the six combinations. The model includes A, B, and A:B. Assume residual checks support the model. The software reports \(SSE=240\), \(df_{\text{error}}=24\), and hence \(MSE=240/24=10\). It gives the following term tests:
| Convention | Term | df | SS | F |
|---|---|---|---|---|
| Type II | A | 1 | 48 | 4.80 |
| Type II | B | 2 | 36 | 1.80 |
| Type II | A:B | 2 | 20 | 1.00 |
| Type III | A | 1 | 18 | 1.80 |
| Type III | B | 2 | 30 | 1.50 |
| Type III | A:B | 2 | 20 | 1.00 |
Verify the calculations. For the Type II A row, \(F=(48/1)/10=4.80\). For Type II B, \(F=(36/2)/10=1.80\). The Type III A row gives \(F=(18/1)/10=1.80\), and Type III B gives \(F=(30/2)/10=1.50\). Both interaction rows give \(F=(20/2)/10=1.00\). The degrees of freedom match two A levels, three B levels, and their interaction.
Identify the change in question. These are unbalanced data, so the main-effect SS need not agree across conventions. The Type II A test adjusts for B without using A:B as an adjustment; the Type III A test adjusts for all other fitted terms. Neither SS is a universal measure of “the A effect.” To state the Type III A null precisely, this example must also specify the coding and contrast tested. If the model uses treatment coding with the first B level as the reference, its one-degree-of-freedom A test concerns the protocol difference at that reference clinic setting. The Type II test instead concerns A under its Type II comparison rule. The two F statistics are not competing calculations of one automatically identical hypothesis.
No p-value or decision is supplied by the table alone. If a p-value were reported, it would be interpreted relative to its own null hypothesis and the chosen \(\alpha\), not as the probability that the null is true. A report should name the convention and coding and describe the comparison in context.
Audit Degrees of Freedom and Coding
Degrees of freedom provide a useful check on what software treated as a factor and what terms it fitted. For a full factorial model with \(a\) levels of A and \(b\) levels of B, with all \(ab\) cells represented, the usual degrees of freedom are \(a-1\) for A, \(b-1\) for B, and \((a-1)(b-1)\) for A:B. In a full-rank cell-means model, the error degrees of freedom are \(N-ab\). A mismatch can indicate a different model, different coding of a variable, a different analysis subset, or a model with insufficient rank. The table does not tell you which explanation applies; check the model specification and data used.
Worked Example: Detecting a Coding Change From Degrees of Freedom
A hypothetical study has 36 independent observations in a design with three ordered dose levels and two settings; all six dose-setting combinations are represented. The planned model treats dose as a categorical factor and includes dose, setting, and their interaction. Residual checks are acceptable.
Work out the expected degrees of freedom. For categorical dose, \(df_{\text{dose}}=3-1=2\); for setting, \(df_{\text{setting}}=2-1=1\); and for the interaction, \(df_{\text{interaction}}=2\times1=2\). The full model has \(2+1+2=5\) model degrees of freedom, so \(df_{\text{error}}=36-6=30\). Output showing these degrees of freedom is consistent with the planned full factorial model.
Now suppose the output instead lists dose with 1 degree of freedom, setting with 1, dose:setting with 1, and error with 32. The arithmetic is \(1+1+1=3\) model degrees of freedom and \(36-4=32\) residual degrees of freedom. These values are consistent with a different model: dose was entered as a single numeric predictor, with a linear dose-by-setting interaction. The dose row then tests a linear trend, not the general two-degree-of-freedom difference among three categorical dose means.
For illustration, suppose that alternative model reports \(MSE=330/32=10.3125\) and a one-degree-of-freedom dose SS of 41.25. Its dose statistic is \(F=(41.25/1)/10.3125=4.00\). The arithmetic is internally consistent, but the test answers the linear-trend question under the numeric-dose model. It is not a substitute for the planned categorical dose test. The investigator should inspect the model formula and variable coding, then refit if the categorical question was intended.
A Practical Audit and Common Pitfall
Before interpreting an ANOVA table, follow the rows back to the analysis that produced them:
Check the factors, interaction terms, and observations included. Verify that the fitted model matches the question.
For Type I output, record the order in which terms entered. For Type II or Type III output, confirm the stated adjustment rule and the software’s definition.
Confirm which variables are factors or numeric predictors, what contrasts are used, and whether each term’s degrees of freedom match its fitted role.
Divide each term’s mean square by the reported error mean square. Investigate discrepancies before interpreting a p-value.
Describe the specific model comparison or contrast tested. Do not rely on a short row label such as “A” to supply the hypothesis.
A common mistake is to see “A” in a table and write, “This tests whether the means for A are equal, adjusted for everything else.” That conclusion does not follow from the label. With Type I SS, the test depends on the term order. With Type II SS, higher-order relatives are not used to adjust a main effect. With Type III SS, coding and contrast definitions can determine the main-effect hypothesis when an interaction is included. The correction is to identify the convention, model, order where applicable, and coding before describing the test.
The audit matters because the same response data can produce different valid tests for different questions. A table can be numerically correct and still be unsuitable for the intended scientific claim. Reporting the model, sums-of-squares convention, term order when relevant, coding, degrees of freedom, and contextual null makes the analysis interpretable and reproducible.
Check Your Understanding
Use the model comparison, term order, and coding to explain what each row tests.
- A Type I table lists B before A. What additional information is needed to identify precisely what the A row tests?
- A full factorial model has four levels of A and two levels of B, with all cells represented. What are the degrees of freedom for A, B, and A:B?
- Why might Type II and Type III sums of squares for a main effect differ in an unbalanced model containing an interaction?
- With treatment coding and an interaction, what does a main-effect coefficient for A typically describe when B’s first level is the reference?
- A three-level dose variable has 1 degree of freedom in a model table. What should you check before interpreting the dose test?