Why Fixed-Effect Structure Is a Scientific Choice
A mixed ANOVA model has fixed-effect terms describing systematic comparisons of interest and random-effect terms describing variation associated with sampled or clustered units. Once a defensible random-effects and covariance structure has been specified, there may still be reasonable alternatives for the fixed effects: for example, whether treatment differences vary with a baseline measure, or whether an interaction is needed. Comparing these structures can help assess a scientific assumption or select a model for a stated purpose.
The key caution is that simplifying a model can change the quantity being estimated. A model with a treatment-by-subgroup interaction can estimate subgroup-specific treatment contrasts. A model without that interaction asserts one common treatment contrast. Those are not merely two ways to calculate the same answer. Before comparing fit, state the scientific estimand: the population quantity the analysis is intended to describe.
Set the Comparison Up Correctly
For a linear mixed model, write the response vector as
Here \(\mathbf{X}\boldsymbol{\beta}\) specifies the fixed-effect mean structure, while \(\mathbf{Z}\mathbf{b}\) represents random effects. A usual model assumes \(\mathbf{b}\sim N(\mathbf{0},\mathbf{G})\), \(\boldsymbol{\varepsilon}\sim N(\mathbf{0},\mathbf{R})\), and independence between \(\mathbf{b}\) and \(\boldsymbol{\varepsilon}\). The covariance structure must be appropriate for the design and sampling units. In particular, response errors should follow the specified residual covariance structure; when \(\mathbf{R}\) is diagonal, this implies conditional independence across units under the stated Gaussian model. This requires considering clustering, possible interference, and other sources of dependence. Random assignment by itself does not establish error independence.
For a fixed-effect comparison, keep the response observations, the random-effects structure, and the residual covariance assumptions the same. Then the difference being assessed is the fixed-effect structure, rather than a simultaneous change in several modeling decisions. Also check that the proposed terms are scientifically meaningful and that the design contains information to estimate them.
Two common comparison tools have different purposes. A nested-model likelihood-ratio test assesses whether the additional restrictions are compatible with the data under its assumptions. An information criterion such as AIC compares relative fit with a penalty for model complexity; it does not test a null hypothesis. Neither tool decides which estimand is scientifically appropriate.
Nested-Model Tests
Suppose a reduced model is nested within a full model, with \(q\) additional fixed-effect parameters in the full model. If both are fitted by maximum likelihood (ML), the likelihood-ratio statistic is
where \(\ell\) is the maximized log likelihood. Under the null hypothesis that the additional parameters are zero, and under regularity conditions, \(\Lambda\) is approximately distributed as chi-square with \(q\) degrees of freedom. This large-sample approximation is not exact in every small or complex mixed-model setting. Appropriate small-sample methods may be needed when justified by the design and model.
Use ML, rather than restricted maximum likelihood (REML), when comparing models with different fixed-effect structures. REML likelihoods depend on the fixed-effect design matrix, so their values are not directly comparable across such models. After selecting or specifying a fixed-effect structure, REML may be used for estimation when appropriate.
Worked Example: Testing a Treatment-by-Baseline Interaction
Suppose a hypothetical experiment assigns 36 units from 12 sites to one of three treatments. The response is measured once per unit, and a baseline score is recorded. The scientific question is whether treatment differences vary linearly with baseline score. Both candidate models include fixed treatment and baseline terms and the same random site intercept and residual structure. The reduced model omits treatment-by-baseline terms; the full model includes the two interaction parameters required for three treatments.
Check the assumptions and comparison. The response and 36 units are the same in both fits, as are the random site intercept and residual assumptions. The model assumes site-level dependence is represented by the random intercept and that residual errors are independent across units conditional on that intercept. For this hypothetical design, suppose units are distinct, there is no interference between units, and no additional clustering remains unmodeled. These are assumptions about the data-generating process, not consequences of random assignment. The reduced model is nested in the interaction model: setting both interaction parameters to zero yields the reduced model.
Suppose ML fitting gives \(\ell_{\text{reduced}}=-82.4\) and \(\ell_{\text{full}}=-78.9\). The likelihood-ratio statistic is
There are \(q=2\) added parameters, so the reference distribution is approximately chi-square with 2 degrees of freedom. For this distribution, the upper-tail probability at 7.0 is \(P(\chi^2_2\geq 7.0)=e^{-7.0/2}\approx 0.0302\). At \(\alpha=0.05\), this is evidence against the null hypothesis that both interaction parameters are zero, subject to the model and approximation assumptions.
The result supports retaining the interaction for this analysis; it does not say that a single treatment effect applies at every baseline score. The intended comparisons should be reported at relevant baseline values, or averaged over a stated target distribution if a marginal contrast is desired.
Information Criteria for Plausible Alternatives
AIC is defined by \(\mathrm{AIC}=-2\ell+2k\), where \(\ell\) is the maximized likelihood and \(k\) is the number of estimated parameters counted under the chosen likelihood convention. It trades off fit against model complexity: smaller AIC indicates a better relative score among the candidate models being compared. Use the same response data and comparable likelihood framework for all candidates. For mixed models, count estimated fixed-effect and covariance parameters consistently.
AIC is useful when candidate models need not be nested, or when the goal is relative information-based selection rather than a particular null-hypothesis test. A small AIC difference is not a proof that one model is true, and an AIC value is not a p-value or the probability that a model is correct. The candidate set and the target estimand still matter.
Worked Example: Comparing Linear and Curved Baseline Adjustment
In a hypothetical study, 30 experimental units from 10 sites receive one of three treatments. The response is measured once, and baseline score is quantitative. The candidate fixed-effect structures are (A) treatment plus a linear baseline term, and (B) treatment plus linear and quadratic baseline terms. Both use the same random site-intercept and residual structure. Suppose the data have no additional clustering or interference beyond that represented by the model, and residual errors are independent across units conditional on the random effects. These conditions are assumptions to be checked against the design and diagnostics.
Assume both models are fitted by ML to the same 30 responses. Model A has \(k=6\) estimated parameters: an intercept, two treatment contrasts, one baseline slope, a residual variance, and a site-intercept variance. Model B adds one quadratic coefficient, so \(k=7\). Suppose the maximized log likelihoods are \(-111.2\) and \(-108.6\), respectively.
For model A,
For model B,
The difference is \(234.4-231.2=3.2\), favoring model B within these two candidates. The quadratic term improves the likelihood enough to offset its one-parameter penalty. This is relative support, not a formal declaration that baseline response is truly curved. In addition, the treatment estimand is unchanged only insofar as both models retain the same treatment parameterization and the same way of defining the treatment comparison; the baseline adjustment can still affect its numerical estimate.
Interactions, Hierarchy, and the Meaning of a Treatment Effect
When a model includes an interaction, its lower-order terms are generally retained. For example, a model with treatment-by-baseline interaction ordinarily includes both treatment and baseline main effects. This is the hierarchy principle. Removing a main effect while retaining its interaction can make interpretation depend on how variables are coded and where zero is located. Exceptions may be justified by a specific scientific constraint, but should be stated rather than made as an automatic simplification.
To see how the estimand can change, consider two treatments, A and B, and two baseline subgroups, novice and experienced. The following hypothetical cell counts and mean responses are shown; responses are measured in points.
| Baseline subgroup | Treatment A: n | Treatment A: mean | Treatment B: n | Treatment B: mean |
|---|---|---|---|---|
| Novice | 18 | 40 | 2 | 32 |
| Experienced | 2 | 80 | 18 | 78 |
Within the novice subgroup, the B-minus-A contrast is \(32-40=-8\) points. Within the experienced subgroup, it is \(78-80=-2\) points. Thus an interaction model describes two different subgroup contrasts. By contrast, the observed treatment averages, which weight each subgroup according to its observed treatment-specific counts, are
Their difference is \(73.4-44=29.4\) points, even though B has a lower mean within each subgroup. The discrepancy arises because the treatment groups have very different subgroup compositions. These crude averages do not represent the same comparison as either subgroup-specific contrast.
An additive model without the interaction instead imposes one common treatment contrast across subgroups. In an unbalanced dataset, its fitted treatment coefficient is determined jointly by the model and the observed data; it need not equal the crude difference or either subgroup-specific contrast. If the scientific target is an average over a specified population, define the subgroup weights for that population and standardize the subgroup contrasts using those weights. Do not assume that dropping an interaction automatically produces the desired marginal effect.
Worked Example: Choosing Weights for a Marginal Treatment Contrast
Use the hypothetical subgroup contrasts above, but define the target population to be 60% novice and 40% experienced. The target is the average B-minus-A response contrast standardized to those subgroup proportions. The weighted contrast is
Under these stated weights, the target average treatment contrast is \(-5.6\) points: B has an average response 5.6 points lower than A after standardizing both treatments to the same subgroup composition. If instead the target population were 50% in each subgroup, the corresponding contrast would be \(0.50(-8)+0.50(-2)=-5.0\) points. The weights are part of the estimand, not a cosmetic choice made after fitting the model.
This calculation uses the cell contrasts directly and does not establish that either contrast is estimated precisely. In an actual analysis, uncertainty in the cell means and the resulting standardized contrast must also be accounted for.
A Practical Comparison Sequence
Specify whether the target is a conditional contrast, subgroup-specific contrasts, or a marginal contrast with explicit target weights.
Identify exactly which fixed terms differ. Keep the response data, random-effects structure, and residual covariance assumptions fixed for a focused comparison.
Confirm the terms are estimable and consider the experimental units, clustering, interference, and conditional error dependence. Randomization does not by itself guarantee independent errors.
For nested fixed-effect structures, use an appropriate test with ML likelihoods and stated degrees of freedom. For relative selection among candidates, AIC may be useful; it is not a hypothesis test.
Retain lower-order terms when required by hierarchy, and report treatment contrasts at the values or population weights that define the estimand.
Common Mistakes
- Comparing REML likelihoods across different fixed effects. REML likelihoods depend on the fixed-effect structure. Fit these candidate models with ML for the comparison.
- Treating random assignment as proof of independence. Units may still share a site, experience interference, or have other dependence. State and assess the error-independence assumption in light of the design and covariance model.
- Calling AIC a significance test. A lower AIC is a better relative score among the specified candidates; it does not provide a p-value or prove a model true.
- Dropping an interaction and claiming the same effect. The interaction model permits differing contrasts; the additive model imposes a common contrast. State which target is scientifically intended.
- Dropping a main effect while retaining its interaction without explanation. This can make interpretation coding-dependent. Follow hierarchy unless a substantive constraint justifies another parameterization.
- Choosing a structure only because its fit score is best. Model fit cannot determine the target population, the relevant contrast, or whether the assumptions are credible.
Check Your Understanding
Try these questions using the definitions and examples above.
- Two mixed models differ only by a fixed treatment-by-subgroup interaction. What makes them nested, and what is the null hypothesis for the likelihood-ratio test?
- Why should ML rather than REML be used when comparing models with different fixed-effect structures?
- In a study with strong site clustering, why does random assignment not by itself justify independent response errors?
- What does a lower AIC indicate, and what does it not establish?
- A model includes treatment-by-baseline interaction. Which lower-order terms would ordinarily be retained under the hierarchy principle?
- Why might the crude treatment mean difference differ from a treatment contrast standardized to a specified subgroup distribution?