Nested Fixed-Effect Models and Likelihood-Ratio Tests

Why Nesting Matters

A fixed-effect test often asks whether a collection of terms improves the model after accounting for clustering. The likelihood-ratio test (LRT) answers this by comparing a reduced GLMM with a larger one that contains it. The comparison is meaningful only when the reduced model is genuinely a special case of the larger model, and when both likelihoods refer to the same data and model structure.

For this tutorial, the focus is on identifying that relationship and writing the comparison correctly. The central question is not simply whether one model has more predictors. It is whether the smaller model can be obtained from the larger model by imposing restrictions on its fixed effects while leaving the rest of the model unchanged.

Definition of Nested Fixed-Effect Models

Let \(Y\) denote the observed responses, \(X\) the fixed-effect design matrix, and \(b\) the random effects. A GLMM specifies a conditional response distribution given \(b\), a link connecting its conditional mean to fixed and random effects, and a distribution for \(b\). For example, a common linear predictor is \(X\beta+Zb\), where \(Z\) is the random-effect design matrix.

The likelihood used for ordinary fixed-effect comparisons is the marginal likelihood: it averages the conditional response likelihood over the random-effects distribution. Thus, when the random-effects distribution has density \(f(b;\theta)\), the marginal likelihood has the form below. Here \(\beta\) denotes fixed-effect parameters and \(\theta\) denotes random-effect distribution parameters.

The product notation assumes conditional independence of the responses given the random effects; the integral accounts for their dependence after the random effects are integrated out. A model’s maximized likelihood is the largest value of this marginal likelihood over its allowed fixed-effect and random-effect parameters.

For example, if the full model has fixed effects \(\beta_0,\beta_1,\beta_2\) and the reduced model omits the term associated with \(\beta_2\), then the reduced model is obtained by the restriction \(H_0:\beta_2=0\), provided the other model components are unchanged. More generally, the null hypothesis may impose several independent linear restrictions on the full model’s fixed-effect parameters.

The same idea can be expressed through design spaces: the reduced fixed-effect design must span a subspace of the full model’s fixed-effect design. Different contrast codings do not by themselves make models different if they span the same space. Conversely, merely using fewer named terms does not establish nesting if the models’ design spaces are not contained one within the other.

Formulating the Likelihood-Ratio Comparison

Write \(L_R\) for the likelihood maximized under the reduced model and \(L_F\) for the likelihood maximized under the full model. Because the reduced parameter space is contained in the full parameter space, the full model can reproduce every fit available to the reduced model. Therefore \(L_F\geq L_R\).

The statistic \(D\) is nonnegative; larger values indicate that the full model improves the maximized likelihood more than would be expected from the restrictions in the reduced model. Under the null hypothesis and suitable regularity conditions, \(D\) is commonly compared with a chi-square reference distribution. Its degrees of freedom are the number of independent restrictions imposed on the full model, equivalently the difference in the effective dimensions of the two fixed-effect parameter spaces.

This reference is an asymptotic result, not an automatic guarantee for every fitted GLMM. It requires, among other conditions, identifiable parameters, a null value that is not on a parameter-space boundary, and a sufficiently informative sample for the approximation to be useful. Testing a fixed effect while keeping the random-effects structure fixed is distinct from testing a variance component, where boundary issues can change the reference distribution.

Worked Comparisons

Justification: Why the Statistic Cannot Be Negative

The nonnegativity follows directly from nesting, not from a special property of GLMM software. Let \(\Theta_R\) and \(\Theta_F\) be the reduced and full parameter spaces, including all parameters allowed to vary in the comparison. Nesting means that every reduced-model parameter value corresponds to an allowed full-model value, so \(\Theta_R\subseteq\Theta_F\). Taking a supremum over a larger set cannot yield a smaller maximum:

Since the logarithm is increasing, \(\ell_F\geq\ell_R\), hence \(D=2(\ell_F-\ell_R)\geq0\). If software reports a noticeably negative value, the models may not actually be nested, the likelihoods may not be comparable, or numerical optimization may have failed to find the relevant maxima.

Checks, Alternatives, and Limitations

Before formulating an LRT, verify that both fits use the same response values and rows, the same conditional response family and link, and the same random-effects structure and likelihood basis. The full model must add only the fixed-effect terms being tested. In particular, changing the random-effects structure at the same time makes it unclear which change accounts for a likelihood difference.

For an ordinary fixed-effect LRT, the likelihoods should be maximized on the same marginal-likelihood basis. Restricted maximum likelihood (REML) criteria are not ordinary likelihoods for direct comparisons across different fixed-effect design spaces. The distinction between ML and REML comparisons is treated separately in the course.

Wald tests are a different approach: they use estimated coefficients and their estimated covariance matrix rather than comparing the two maximized likelihoods. A Wald test may be convenient for a single coefficient, while an LRT naturally assesses a joint block of restrictions. Neither method removes the need to specify the model and null hypothesis correctly. Information criteria can compare candidate models for relative fit with a complexity penalty, but they do not provide the same null-hypothesis test.

Several warning signs deserve attention: the full model’s likelihood is lower than the reduced model’s; one or both fits fail to converge; estimates are extreme or effectively unidentified; or the response rows differ because missing values were handled differently. Sparse binary outcomes or separation can also undermine estimation and the usual large-sample calibration. In such cases, do not treat a computed chi-square probability as automatically reliable; inspect model fitting and consider whether the proposed inferential method is appropriate.

A common mistake is to say, “The full model has more predictors, so it is nested, and the likelihood-ratio test is valid.” More terms alone are not enough. The reduced model must be obtainable from the full model by restrictions, and all other components and data must be held fixed. Another mistake is to interpret a significant joint test as proof that every added coefficient is nonzero. The test addresses the block of restrictions as a whole.