Why Test an Interaction?
A factor interaction allows the association between one categorical predictor and the response to depend on the level of another factor. A fitted model may therefore describe the observed groups better after interaction terms are added. The important question is whether the improvement is large enough, relative to the remaining variation, to provide evidence that the interaction terms are needed.
The test is a comparison of nested models: a reduced model that contains the main effects and a full model that also contains the specified interaction. It tests the interaction terms as a group. This is different from asking whether one coefficient, considered alone, has a small p-value.
For example, with factors \(A\) and \(B\), the notation `A * B` in an R model formula includes the main effects of \(A\) and \(B\), as well as their interaction. To test whether the interaction adds evidence beyond the main effects, compare that full model with one containing the main effects but not the interaction. The reduced model must not omit terms that the question intends to retain.
The Joint Hypothesis and Partial F Test
Suppose the full model adds \(q\) interaction coefficients to a reduced model. The null hypothesis says that all \(q\) coefficients are zero; the alternative says that at least one is nonzero. With treatment contrasts, the particular coefficients depend on the reference levels, but the joint test of the interaction is not changed by switching to another full-rank contrast coding.
For an ordinary least-squares linear model, let \(\mathrm{RSS}_R\) and \(\mathrm{RSS}_F\) be the residual sums of squares from the reduced and full models. Let \(p_F\) be the number of estimated coefficients in the full model, including the intercept, and let \(n\) be the number of observations. The partial F statistic is:
Under the null hypothesis and the classical linear-model assumptions, this statistic has an \(F\) distribution with \(q\) and \(n-p_F\) degrees of freedom. The numerator measures the reduction in residual sum of squares per added interaction coefficient. The denominator estimates the residual variance using the full model.
Why the Partial F Formula Works
The exact F reference distribution follows from the geometry of least squares. Assume that the response has a linear mean represented by the full model, the errors are independent and normally distributed with common variance \(\sigma^2\), and the model matrices have full column rank. Under the null, the true mean is also representable by the reduced model. The full model has \(q\) additional independent directions in which it can fit the data.
Under these conditions, the reduction in residual sum of squares, divided by \(\sigma^2\), has a chi-square distribution with \(q\) degrees of freedom under the null. The full-model residual sum of squares divided by \(\sigma^2\) has a chi-square distribution with \(n-p_F\) degrees of freedom. These two quantities are independent because they arise from orthogonal least-squares projections. Dividing each by its degrees of freedom and taking their ratio cancels \(\sigma^2\), giving the stated F statistic.
This derivation also explains the role of the conditions. If the models are not nested, the reduction in residual sum of squares does not represent the addition of \(q\) restrictions in this way. If the models use different observations, their residual sums of squares are not a valid direct comparison. If errors are not independent, normal, or homoskedastic, the classical F reference distribution need not be exact; an appropriate alternative test may be needed.
A Careful Model-Comparison Workflow
Name the factors and retain the main effects in both models. Decide which interaction terms are being tested jointly.
Use the same response, observations, and other included predictors. The full model must contain every term in the reduced model.
The numerator degrees of freedom \(q\) equal the number of independent interaction coefficients added. The denominator degrees of freedom are \(n-p_F\).
Calculate the partial F statistic and its p-value under the null. Relate the evidence to the question; do not equate statistical significance with material value.
For a factor with \(r\) levels and another factor with \(s\) levels, a fully specified interaction generally adds \((r-1)(s-1)\) coefficients under a standard full-rank encoding. This gives the numerator degrees of freedom when all interaction terms are jointly tested. Sparse or empty combinations can reduce the estimable rank, so count independent added parameters rather than merely counting printed coefficient names.
Worked Example: Testing a Two-Level-by-Two-Level Interaction
A hypothetical service team models a continuous satisfaction score using two binary factors: contact type (online or phone) and issue category (billing or account access). There are 24 observations, all four combinations are represented, and the model has an intercept, both main effects, and one interaction coefficient. The reduced model has residual sum of squares 120; the full model has residual sum of squares 80. Test whether the interaction adds evidence at the 5% significance level.
Method. Use the partial F test. With two levels for each factor, there is \((2-1)(2-1)=1\) interaction coefficient, so \(q=1\). The full model has \(p_F=4\) coefficients and therefore \(24-4=20\) residual degrees of freedom.
The numerator mean square for the added interaction is:
The full-model residual mean square is:
Thus:
Under the null, this is compared with an \(F\) distribution with 1 and 20 degrees of freedom. The 5% upper-tail critical value is approximately 4.35, and the p-value is approximately 0.0049. Since \(10>4.35\), equivalently since \(0.0049<0.05\), reject the null hypothesis. The data provide evidence that the interaction improves the model fit, conditional on the two main effects.
Check. Adding a parameter cannot increase the least-squares residual sum of squares, and here the RSS falls from 120 to 80. The positive reduction and positive residual mean square give a positive F statistic. The common mistake to avoid is using 80 rather than 4 as the denominator: the formula uses the full-model residual mean square, not its residual sum of squares.
Worked Example: A Joint Test With More Than One Interaction Term
A hypothetical training study compares three workshop formats and two assessment modes. The outcome is a continuous score. All six format-by-mode combinations are represented, and the full model has six cell-mean parameters, so \(p_F=6\). With 36 observations, the reduced additive model has residual sum of squares 250 and the full interaction model has residual sum of squares 190. Test the full interaction at the 5% level.
Method. There are \((3-1)(2-1)=2\) interaction coefficients, so the joint null restriction has \(q=2\). The full-model residual degrees of freedom are \(36-6=30\).
The reduction in RSS per added coefficient is:
The full-model residual mean square is:
Therefore:
The test has 2 and 30 degrees of freedom. Its upper-tail p-value is approximately 0.0163, so the joint null is rejected at the 5% level. The evidence is against the claim that both interaction coefficients are zero. It does not, by itself, identify which format-mode combinations differ; that requires interpretation of the fitted effects.
Check. The additive model has an intercept, two format coefficients, and one assessment-mode coefficient, for four parameters. Adding two interaction coefficients gives six full-model parameters, consistent with the stated residual degrees of freedom \(36-6=30\). The common mistake is to use one numerator degree of freedom because the interaction is a single concept; here it adds two independent coefficients.
Statistical Evidence Is Not Practical Value
A small p-value indicates that the observed improvement in fit would be unusual under the joint null and the model assumptions. It does not say that the interaction is large, important to a decision, causal, or likely to improve predictions on new data. Conversely, a non-significant test does not prove that the interaction is exactly zero; it may reflect limited data or imprecise estimates.
Worked Example: Significant Interaction, Small Practical Difference
A hypothetical insurer models a continuous administrative-cost score using two binary factors: intake route and claim class. With 24 observations, the full model has 20 residual degrees of freedom. The estimated interaction coefficient is 0.40 score units with standard error 0.1265. The organization considers an interaction contrast smaller than 2 score units immaterial for its operational decision. Assess both statistical evidence and practical size.
Method. With two binary factors, there is one interaction coefficient. Its individual t test is equivalent to the one-degree-of-freedom partial F test, with \(F=t^2\). The test statistic is:
Using 20 degrees of freedom, the two-sided p-value is approximately 0.0049. This is statistically significant at the 5% level. For a practical-scale check, a 95% confidence interval for the interaction coefficient is approximately:
The estimated interaction contrast is 0.40 units, and even the upper confidence limit is below the stated 2-unit operational threshold. Thus the result is statistically detectable but small relative to that decision threshold. This conclusion is specific to the stated threshold and outcome scale; a different decision could assign the effect different value.
Check. The coefficient-to-standard-error ratio is about 3.16, whose square is about 10, consistent with the one-degree-of-freedom F test. The common mistake to avoid is concluding “important” solely from \(p<0.05\). Statistical evidence and practical value answer different questions.
Implementation and Common Traps
In R, fit the models to the same analysis data and compare them with a nested-model comparison such as `anova(reduced, full)` for ordinary `lm` models. Confirm that the full model includes all terms in the reduced model and adds exactly the interaction under investigation. If missing values cause the models to use different rows, explicitly construct a common analysis data set before comparing them.
- Testing coefficients one at a time. A multi-parameter interaction is a joint hypothesis. Individual coefficient tests do not substitute for the joint test.
- Using the wrong numerator degrees of freedom. Use the rank of the added interaction terms, not automatically one degree of freedom.
- Comparing models fitted to different observations. The partial F calculation requires a valid nested comparison on the same response data.
- Omitting main effects from the reduced model. The intended interaction test ordinarily compares a model with both main effects against one with those same main effects plus the interaction.
- Calling fit improvement predictive improvement. The full model cannot fit the training data worse by least squares, but that fact alone does not establish better performance on new data.
- Applying the classical formula without checking assumptions. The exact F result relies on the stated linear-model error assumptions and full-rank nested models. If those conditions are unsuitable, use a test appropriate to the model and data.
Check Your Understanding
For each problem, identify the joint test or calculation requested and check the answer against the model structure.
- A factor with four levels interacts with a binary factor in a full-rank model. How many numerator degrees of freedom does a joint test of the interaction have? (Answer: 3)
- A reduced model has RSS 96. A full model adds two interaction coefficients, has RSS 72, and has 18 residual degrees of freedom. Calculate the partial F statistic. (Answer: \(F=(24/2)/(72/18)=3.000\), with 2 and 18 degrees of freedom)
- For the test in Question 2, the 5% critical value is 3.555. State the decision at the 5% level. (Answer: Do not reject the joint null, since \(3.000<3.555\))
- A binary-by-binary interaction test has \(F=6.25\) with 1 and 25 degrees of freedom. What is the equivalent absolute t statistic, and what is the interpretation of this equivalence? (Answer: \(|t|=\sqrt{6.25}=2.5\); for a single added coefficient, the joint F test equals the squared t test)
- A large data set produces a small p-value for an interaction, but its estimated contrast is below a prespecified operational threshold and its uncertainty interval is also below that threshold. What two distinct conclusions should be reported? (Answer: There is statistical evidence against a zero interaction, but the estimated effect is not practically material under the stated threshold)