From Model to Conclusion
A factorial ANOVA result is useful only when its test answers the comparison the analysis was meant to address. In an unbalanced design, different sums-of-squares choices can correspond to different comparisons, even when they use the same observations and fitted model. A reliable report therefore connects the study question to the model, the test’s hypothesis, and the conclusion—not just to a software table.
This tutorial follows that workflow for two factors. The examples use hypothetical data and results. In each case, the model and the assumptions are stated before interpreting a test.
Specify the Model and the Intended Comparison
Suppose a hypothetical experiment compares two programs (factor A) across two clinics (factor B), measuring a symptom score in points. The response is quantitative, and each participant contributes one score to one program-by-clinic cell. Assume the four cells are all represented. A fixed-effects model that includes the interaction is
Here \(Y_{ijk}\) is the score for participant \(k\) in program \(i\) and clinic \(j\); \(\mu\) is the overall level; \(\alpha_i\) and \(\beta_j\) are factor effects; and \((\alpha\beta)_{ij}\) is the interaction effect. For identifiability, one possible convention is sum-to-zero constraints on each factor’s effects and on the interaction across each factor. The error terms are assumed independent, normally distributed with mean zero and common variance \(\sigma^2\). This model also treats both factors as fixed.
Before choosing a test, state what comparison matters. For example, is the aim to compare programs after adjusting for clinic, to compare programs in a particular clinic, or to compare equally weighted averages across the two clinics? Those are not automatically the same question when cell sizes differ or an interaction is present. Type I tests depend on term order; Type II and Type III tests represent different adjusted hypotheses. The sums-of-squares label is part of the test specification, not a cosmetic setting.
Worked Example: A Type III Workflow With Unequal Cell Sizes
Worked Example: A Type III Workflow With Unequal Cell Sizes
Suppose the hypothetical study has these cell sizes and sample means. The within-cell sum of squared deviations from the four cell means is 40 score-points squared.
| Program | Clinic | Sample size | Sample mean score (points) |
|---|---|---|---|
| A0 | B0 | 4 | 10 |
| A1 | B0 | 6 | 14 |
| A0 | B1 | 5 | 12 |
| A1 | B1 | 7 | 18 |
There are \(N=22\) participants and four cell means, so the full interaction model has \(22-4=18\) residual degrees of freedom. Its residual mean square is
Assume the hypothetical design kept participants independent, the residual plots showed no systematic pattern, the residual distributions were reasonably compatible with normality, and the within-cell spreads were similar. These checks support using the model’s independent, normal, common-variance errors; they do not prove the assumptions exactly.
With the interaction in the model, a Type III main-effect test for program compares the equally weighted program means across the two clinics. The A0 average is \((10+12)/2=11\) points; the A1 average is \((14+18)/2=16\) points. Their difference is 5 points. This is not a comparison weighted by the observed cell sizes.
For the contrast of those equal-weight averages, the coefficients on the cell means in table order are \((-1/2,1/2,-1/2,1/2)\). Its estimated variance is \(MSE\) times the sum of each squared coefficient divided by its cell size. Thus the contrast sum of squares is
The corresponding Type III test has one numerator degree of freedom. Its statistic is \(F=(SS_A^{(III)}/1)/MSE\), or
The p-value is less than .001. Against the null hypothesis that the equally weighted program means are equal across these clinics, the result is evidence of a difference. In this hypothetical sample, the A1 average is 5 points higher. The test does not establish that the program caused the difference unless the study design and other assumptions support a causal interpretation.
For completeness, the Type III clinic comparison is also an equally weighted average: B1 has mean \((12+18)/2=15\), and B0 has mean \((10+14)/2=12\), a difference of 3 points. The same variance calculation gives
This test also has \(df=(1,18)\) and \(p<.001\). The interaction contrast is the difference in program differences between clinics: \((18-12)-(14-10)=2\) points. Its Type III sum of squares is
With \(df=(1,18)\), this gives \(p\approx .141\). The data do not provide strong evidence against the null hypothesis of no difference in program effects between clinics. This does not prove the interaction is exactly zero.
Why the Sums-of-Squares Choice Changes the Report
For a partial F test, the numerator sum of squares measures the reduction in residual sum of squares when the tested term or comparison is added to a specified reduced model. If the tested hypothesis has \(q\) degrees of freedom and the full model has residual mean square \(MSE\), the statistic is
This statistic is compared with an \(F\) distribution with \(q\) numerator degrees of freedom and the full model’s residual degrees of freedom, under the null hypothesis and model assumptions. The p-value is the probability, under that null and model, of obtaining a statistic at least as large as the observed one. A small p-value is evidence against the tested hypothesis; it is not the probability that the hypothesis is true.
The reason the contrast calculation in the first example gives its sum of squares can be seen directly. For independent cell means, the variance of a linear contrast \(\widehat{L}=\sum_i c_i\bar{x}_i\) is \(\sigma^2\sum_i c_i^2/n_i\). Under the null hypothesis \(L=0\), the squared contrast divided by its estimated variance is an \(F\) statistic with one numerator degree of freedom:
This also explains why unequal cell sizes matter: they change the estimated variance of a contrast. The hypothesis must be identified before interpreting the resulting statistic. A test of equally weighted marginal means is not interchangeable with a test based on the observed sample-size weighting.
Worked Example: Reporting a Sequential Test
Worked Example: Reporting a Sequential Test
Consider a separate hypothetical unbalanced factorial analysis with two-level factor A entered first, followed by factor B and their interaction. Suppose there are 22 observations, all four cells are represented, and the full model has residual sum of squares 36 on 18 degrees of freedom. Therefore \(MSE=36/18=2\). Assume the design supports independent observations and the full-model residuals are reasonably normal with similar spreads across cells.
The software’s Type I sum of squares for A is 24 on one degree of freedom. Since A is entered first, this is the improvement from the intercept-only model to the model containing A; it is not A’s improvement after adjusting for B. Using the full-model residual mean square as denominator,
A suitable report is: “When entered first, factor A reduced the residual variation relative to the intercept-only model, \(F(1,18)=12.00\), \(p=.002769\).” If the scientific question was whether A differs after accounting for B, this sequential result does not by itself answer that question. The order and comparison belong in the report.
Worked Example: Reporting an Adjusted Partial Test
Worked Example: Reporting an Adjusted Partial Test
In another hypothetical analysis, suppose the intended model is additive: two-level factors A and B are included, but no interaction term is included. There are 22 independent observations, all factor combinations are represented, and model diagnostics support approximate normality and a common error variance. The full additive model has 19 residual degrees of freedom. Suppose its residual sum of squares is 38, so \(MSE=38/19=2\). Comparing the full model with the reduced model containing B alone gives an extra sum of squares for A, conditional on B, of 21.58.
The partial test is
A suitable report is: “In the additive model, there was evidence of an A difference after adjusting for B, \(F(1,19)=10.79\), \(p=.003896\).” This conclusion is conditional on the additive model: because no interaction is included, the analysis assumes that the A difference does not vary across levels of B. A report should not imply that this test is an interaction-adjusted comparison if the model did not include an interaction.
A Reporting Checklist and Common Pitfall
A concise report can still be precise. State the model terms, identify the test type or reduced-model comparison, give the tested hypothesis in words, and report the statistic with its degrees of freedom and p-value. Then give the conclusion in context, including the direction or estimated difference when useful. If the test is not significant, say that the analysis did not find sufficient evidence for the tested difference; do not say that the groups are identical.
- Wrong reasoning: “The software reports a significant Type I test for A, so A differs after adjusting for B.”
- Correction: If A was entered first, its Type I test compares the intercept-only model with the A-only model. Identify that sequential comparison; use an adjusted test if adjustment for B is the intended question.
- Wrong reasoning: “The interaction p-value is above .05, so the program effect is identical in both clinics.”
- Correction: A non-small p-value is not proof of equality. Report that the test did not provide strong evidence of an interaction, and retain the uncertainty in the estimated effects.
- Wrong reasoning: “The Type III program test compares the programs in the clinic with more participants.”
- Correction: In the full two-factor model used above, the Type III test concerns equally weighted marginal means. Describe that target explicitly; it is not automatically weighted by observed cell sizes.
Check Your Understanding
For each question, focus on the link between the model, the hypothesis, and the reported conclusion.
- In a full two-factor model with an interaction, what does an equal-weight Type III main-effect comparison average over?
- A factor is entered first in a Type I analysis. What reduced and expanded models define its sequential comparison?
- A partial test has \(SS_{\text{test}}=18\), \(q=2\), and \(MSE=3\). Calculate its F statistic and state what additional information is needed to find its p-value.
- Why should a report identify the sums-of-squares type or the reduced-model comparison in an unbalanced factorial analysis?
- What conclusion is justified when an interaction test has \(p>.05\), and what stronger conclusion is not justified?