ML and REML for Fixed-Effect Model Comparisons

Why the Estimation Method Matters

When two mixed models have the same random-effects structure but different fixed effects, their likelihoods must be calculated on a common basis to support a likelihood-ratio test. Maximum likelihood (ML) supplies that basis: each model is fitted by maximizing the likelihood of the observed response data over its own fixed-effect and covariance parameters. Restricted maximum likelihood (REML), by contrast, removes the fixed-effect contribution using contrasts determined by the fixed-effect design. Change that design and the restricted likelihood changes its basis.

This distinction is especially important in generalized linear mixed models. For a non-Gaussian response, ML is based on a marginal likelihood that integrates over the random effects. REML is most directly defined for Gaussian linear mixed models; extensions to non-Gaussian models depend on approximations or particular methods. A fixed-effect likelihood-ratio test should therefore compare ML fits, not REML criteria.

Definition: In a generalized linear mixed model, let $y_i$ denote the response vector for cluster $i$, $X_i$ its fixed-effect design matrix, and $b_i$ its random effects. The marginal likelihood for the observed responses is obtained by integrating the conditional response model over the random-effects distribution: $$ L(\beta,\theta;y)=\prod_i \int f(y_i\mid b_i,\beta)\,g(b_i\mid\theta)\,db_i. $$ Here $\beta$ is the vector of fixed-effect coefficients and $\theta$ denotes the random-effects covariance parameters (and any other model parameters). ML estimates maximize this likelihood over all parameters. In a Gaussian linear mixed model, REML instead forms a likelihood from linear contrasts that eliminate the fixed effects.

ML Likelihood-Ratio Comparisons

Suppose a reduced model is nested within a full model: the reduced model can be obtained by imposing $q$ constraints on the full model's fixed effects, while retaining the same response data, random-effects structure, and likelihood specification. Fit both models by ML. If $\ell_0$ and $\ell_1$ are their maximized marginal log-likelihoods, respectively, then the likelihood-ratio statistic is

G^2=2(\ell_1-\ell_0).

The full model cannot have a smaller maximized likelihood when the reduced model is a genuine restriction of it and both are optimized consistently. Under regularity conditions and the null hypothesis, $G^2$ is approximately distributed as a chi-square random variable with $q$ degrees of freedom. The p-value is the upper-tail probability $P(\chi^2_q\ge G^2)$. This is an asymptotic approximation, not an exact small-sample guarantee.

The comparison requires the same observed responses and likelihood basis in both fits. In practice, use the same observations, response distribution, link, random-effects structure, and likelihood approximation. The fixed-effect terms being tested should be the only intended change. The null value must be an interior point of the parameter space, and the models must be identifiable with a finite, well-behaved maximum. When these conditions fail, the usual chi-square reference may not be reliable.

Why REML Criteria Change with the Fixed-Effect Design

The issue can be seen directly in a Gaussian linear mixed model. Let $y\sim N(X\beta,V_\theta)$, where $X$ is the fixed-effect design matrix and $V_\theta$ is the covariance matrix. REML uses contrasts $K'y$, where $K$ has $n-p$ columns and $K'X=0$; here $p$ is the rank of $X$. These contrasts remove $\beta$, so the restricted likelihood is a likelihood for a transformed response rather than the original $n$-dimensional response.

For a full-rank contrast matrix $K$, its log-likelihood has the form

\ell_R(\theta)=-\frac{1}{2}\left[(n-p)\log(2\pi)+\log|K'V_\theta K|+(K'y)'(K'V_\theta K)^{-1}(K'y)\right].

If $X$ changes, then the constraints $K'X=0$ change. The number and identity of the contrasts can change too. Thus REML likelihoods from models with different fixed effects do not generally describe the same transformed data and cannot be treated as a likelihood ratio on a common sample space.

Remove fixed effects.
Choose contrasts $K'y$ satisfying $K'X=0$, so their distribution does not depend on $\beta$.

Change the fixed-effect model.
A different $X$ generally requires a different contrast space, including potentially a different number of contrasts.

Recognize the comparison problem.
The two restricted likelihoods are based on different transformations of $y$. Their difference is not the ordinary likelihood-ratio statistic for testing the added fixed effects.

This also explains why REML can be useful for estimating covariance parameters when the fixed-effects structure is held constant, but its criterion should not be used to test a change in that structure. In Gaussian models, REML's adjustment accounts for estimating fixed effects when estimating covariance parameters. It does not turn its restricted likelihood into a common likelihood for comparing different $X$ matrices.

Worked ML Comparisons

Worked Example: A Gaussian Illustration of the Difference

Consider hypothetical independent Gaussian observations $y=(1,2,2,5)$, with known common variance structure and no random effects. Compare a reduced model with an intercept only to a full model with an intercept and a linear predictor $x=(0,1,2,3)$. This special case illustrates the likelihood calculation; it does not remove the REML comparability issue in mixed models.

The reduced-model fitted mean is $\bar{y}=2.5$, giving $$ SSE_0=(1-2.5)^2+(2-2.5)^2+(2-2.5)^2+(5-2.5)^2=9. $$ For the full model, the slope estimate is $6/5=1.2$, and the intercept is $0.7$. Its fitted values are $0.7,1.9,3.1,4.3$, so $$ SSE_1=(1-0.7)^2+(2-1.9)^2+(2-3.1)^2+(5-4.3)^2=1.8. $$

With an estimated common variance, the maximized Gaussian log-likelihood yields $G^2=n\log(SSE_0/SSE_1)$. Substituting the values, $$ G^2=4\log(9/1.8)=4\log(5)=6.438. $$ There is one added slope, so the asymptotic reference has $df=1$, giving $p\approx0.0112$. This calculation favors the model with the predictor under that approximation. However, only four observations are available, so the chi-square approximation is not dependable evidence on its own. Comparing REML criteria would not provide a valid likelihood-ratio test here because the intercept-only and intercept-plus-slope designs define different residual-contrast spaces.

Worked Example: Testing One Fixed Effect in a Binary GLMM

Suppose hypothetical clustered binary data are fitted with a random-intercept logistic model. The reduced model has an intercept, and the full model adds a treatment indicator. Both models use the same clusters and observations, the same random-intercept distribution, and the same marginal-likelihood approximation. Assume the fitted ML log-likelihoods are $\ell_0=-46.1$ and $\ell_1=-42.8$.

The treatment adds one fixed-effect parameter, so $q=1$. The statistic is $$ G^2=2[-42.8-(-46.1)]=2(3.3)=6.6. $$ For a chi-square distribution with one degree of freedom, the upper-tail p-value is approximately $0.0102$. Under the regularity and approximation conditions, this is evidence against the null that the treatment coefficient is zero, relative to these two models. It does not show that treatment caused an outcome change, nor does it give the treatment effect's size; those require a coefficient estimate and a contextual interpretation.

The comparison is valid only if both log-likelihoods are ML values on the same data and based on compatible calculations. If one fit used a different set of observations, or if one value were a REML criterion, the subtraction would not give the stated test.

Worked Example: Testing a Two-Parameter Categorical Effect

Consider a hypothetical GLMM in which a three-level categorical predictor is represented by two coefficients relative to a reference level. The reduced and full models have identical random-effects structures and use the same data and ML method. Suppose their maximized log-likelihoods are $-78.9$ and $-73.6$, respectively.

The full model adds two parameters, so $q=2$. Therefore $$ G^2=2[-73.6-(-78.9)]=2(5.3)=10.6. $$ For $df=2$, the chi-square upper-tail probability is approximately $0.0050$. Subject to the regularity conditions, this provides evidence that the categorical predictor's fixed effects are not both zero. It is a joint test: it does not by itself establish which level contrasts differ or give their directions. Those questions require estimates and appropriate follow-up inference.

Using REML values for these two fits would not repair or improve the test. The change from the reduced to the full model changes the fixed-effect design and hence the restricted contrasts.

Assumptions, Diagnostics, and Alternatives

Before interpreting a fixed-effect ML comparison, verify that the models are nested and that the intended restriction is the only change. Confirm that they use exactly the same response records, distribution, link, offset, and random-effects structure. Check that both optimizations converged, that parameter estimates are finite, and that the full-model log-likelihood is no smaller than the reduced-model value apart from minor numerical tolerance. A substantial decrease in the full-model likelihood is a warning to check coding, optimization, and model definitions.

The chi-square calibration also assumes regularity: parameters are identifiable, the null constraints are smooth, and the tested parameter values are not on a boundary. Fixed-effect tests typically meet the interior-point condition when testing coefficients equal to zero, but sparse data, near separation, or weakly identified parameters can still make asymptotic inference inaccurate. Boundary issues for variance parameters are a distinct problem; they are not resolved by switching ML and REML.

Approximate GLMM likelihoods deserve particular attention. Laplace approximation or quadrature may be used to evaluate the marginal likelihood. Both models should use settings accurate enough for the difference in log-likelihoods to be stable. If the test result changes materially with reasonable integration settings, the numerical calculation is not yet a dependable basis for inference.

Approach	What it uses	Role in fixed-effect comparisons
ML likelihood-ratio test	Maximized marginal likelihoods for the observed responses	Appropriate for nested fixed-effect models fitted on the same basis
REML	Likelihood of contrasts chosen to eliminate fixed effects	Not a valid likelihood-ratio comparison when the fixed-effect design changes
Wald test	Coefficient estimate and estimated standard error or covariance	Tests fixed-effect constraints without subtracting model likelihoods; it has its own approximation limitations
Information criteria	Fit and a complexity penalty	Can support model selection, but are not the same as a null-hypothesis likelihood-ratio test

A Wald test may be useful when the desired question concerns a coefficient or a linear constraint, but it can behave differently from an ML likelihood-ratio test, particularly with limited information or nonlinear likelihoods. Information criteria such as AIC address a fit-versus-complexity trade-off, not whether a specified nested null is rejected at a chosen significance level. Neither is a reason to substitute REML likelihoods into the ML test.

Common Pitfall and Practical Rule

Wrong reasoning: “REML often gives better covariance estimates, so I should compare the REML criteria for the models with and without a fixed predictor.” This overlooks that the restricted contrasts are defined using the fixed-effect design. The two criteria are not generally likelihoods for the same transformed response.

Correction: For a likelihood-ratio test of different fixed-effect structures, fit both models by ML, on the same observations and with the same random-effects and likelihood specifications. Then calculate $2(\ell_1-\ell_0)$ and use the appropriate degrees of freedom only when the nesting and regularity conditions justify the chi-square approximation. REML has a different role, principally when estimating covariance parameters with the fixed-effects structure held fixed.

Key takeaway: REML's fixed-effect adjustment is exactly why its criteria depend on the fixed-effect design. Use ML—not REML—for likelihood-ratio comparisons that add, remove, or otherwise change fixed effects.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

Two nested models differ by one fixed-effect coefficient. Their ML log-likelihoods are $-30.2$ and $-27.7$. Calculate the likelihood-ratio statistic and identify its asymptotic degrees of freedom.
Why can the REML criteria of an intercept-only model and an intercept-plus-slope model not generally be subtracted to form the fixed-effect likelihood-ratio statistic?
A full model has a lower maximized ML log-likelihood than its nested reduced model. Give two checks to make before interpreting a test.
Two models use different observations because one predictor has missing values. Can their ML log-likelihoods be used for the ordinary nested likelihood-ratio comparison? Explain.

Solutions. (1) $G^2=2[-27.7-(-30.2)]=5.0$, with $df=1$; the chi-square approximation is appropriate only if its regularity conditions hold. (2) REML uses contrasts that eliminate the fixed effects, and those contrasts depend on the design matrix; changing the fixed effects changes the basis of the restricted likelihood. (3) Check that the models are actually nested and fitted to the same response data; also inspect convergence, parameter estimates, and likelihood calculation settings. (4) No. The log-likelihoods are based on different response records, so their difference is not the ordinary likelihood-ratio comparison on a common data set.