Restricted Likelihood-Ratio Tests for Random Effects

Why restrict the likelihood?

When testing whether a random-effect term is needed, the question is whether the data support a positive variance for that term. In a Gaussian linear mixed model, restricted maximum likelihood (REML) provides a likelihood for contrasts that eliminate the fixed effects. A restricted likelihood-ratio test compares that likelihood under a model with the variance fixed at zero against a model in which the variance is estimated subject to being nonnegative.

The restriction has an important consequence: the comparison is meaningful only when the models use the same fixed-effect design. It also does not make every likelihood-ratio statistic follow an ordinary chi-square distribution. For a single variance component tested at zero, the null value lies on the boundary of the allowable parameter space, so the usual asymptotic reference distribution is a mixture.

What the restricted likelihood compares

Let \(K\) be an \(n\) by \(n-p\) matrix with full column rank and \(K'X=0\). Then \(K'y\) has mean zero and covariance \(K'V(\theta)K\): the fixed-effect contribution disappears because \(K'X\beta=0\). The restricted likelihood is therefore based on variation in these contrasts rather than on a likelihood that estimates the fixed effects as part of the comparison. An equivalent expression for the restricted log likelihood, up to constants that do not depend on \(\theta\), is

The term \(y'P_Vy\) measures the response variation after accounting for the fixed-effect space under the covariance model. The determinant terms account for the covariance of the data and the fixed-effect design. For two models with the same \(X\), a change of valid contrast basis changes the restricted likelihood by a factor that does not depend on the covariance parameters, so it does not change the likelihood-ratio statistic.

Constructing the test and its reference law

For a single variance component \(\tau^2\), the null model fixes \(\tau^2=0\); the alternative estimates \(\tau^2\geq 0\). Let \(\widehat{\ell}_{R,0}\) and \(\widehat{\ell}_{R,1}\) be the maximized restricted log likelihoods under the null and alternative, respectively. The statistic is

Under the null, \(\tau^2=0\) is the edge of the parameter space rather than an interior value. In the standard single-component setting, the asymptotic null law is \(0.5\chi^2_0+0.5\chi^2_1\), where \(\chi^2_0\) denotes a point mass at zero. This is the same half-at-zero, half-chi-square-one boundary law introduced in the earlier material on deriving the half-at-zero, half-chi-square reference law. For an observed \(D_R>0\), the upper-tail p-value is

The usual justification requires a correctly specified Gaussian linear mixed model, the same full-rank \(X\) and the same observations in both fits, nested covariance models, and a single newly tested scalar variance component. Other covariance parameters must be identifiable and interior under the null; the tested variance must be identifiable under the alternative. The usual asymptotic approximation also relies on adequate information, commonly provided by a growing number of independent clusters. If the null makes additional covariance parameters unidentified, the simple mixture need not apply.

A brief way to see why a mixture arises is to consider the local likelihood improvement in the direction of the tested variance. After accounting for nuisance parameters, the corresponding standardized score is approximately \(Z\sim N(0,1)\). Because the variance cannot be negative, a negative score cannot move the estimate below zero: the local improvement is proportional to \(\max(0,Z)^2\). This quantity is zero when \(Z\leq 0\), an event with probability one half, and otherwise has the positive part of a chi-square-one distribution. This explains both the mass at zero and the halved upper tail.

When the comparison is not valid

The fixed-effect design is not a bookkeeping detail. REML removes the fixed-effect space, and changing that space changes the contrasts whose likelihood is being evaluated. Therefore, restricted log likelihoods from models with different fixed-effect designs are not directly comparable as a likelihood-ratio test. When fixed effects differ, ML-based comparison is the standard likelihood approach; REML is used to compare covariance structures while holding the fixed-effect design fixed.

A further limitation matters in a GLMM. For a non-Gaussian response, the marginal likelihood integrates over random effects, but there is generally no exact Gaussian error-contrast construction that removes fixed effects and yields the restricted likelihood above. Some methods and software define approximate restricted-likelihood criteria for GLMMs. Their meaning and accuracy depend on the approximation and implementation; an approximate REML difference should not automatically be treated as having the Gaussian restricted-likelihood reference law. Check the model family, fitting method, and software documentation before reporting such a test.

Diagnostics and practical safeguards

Before interpreting the statistic, verify that both models use the same observations, response representation, fixed-effect design, and compatible covariance assumptions. Confirm that the null is obtained by fixing the tested variance to zero and that the alternative permits nonnegative values. The reference law also requires that no extra covariance parameter becomes unidentified under the null. A random-effect term with additional covariance or correlation parameters may not satisfy the simple single-component conditions.

• Check convergence and the optimum. Inspect optimizer warnings, gradients, and whether the alternative fit reached a constrained maximum. A higher null likelihood than alternative can indicate numerical failure, since the nested alternative includes the null as a boundary case.
• Check the boundary estimate. An estimate of zero can legitimately yield \(D_R=0\). A tiny negative estimate is not an admissible variance; investigate parameterization and numerical tolerances rather than interpreting it as a negative variance.
• Check the reference distribution. Do not use an ordinary \(\chi^2_1\) tail for this boundary test: it doubles the standard asymptotic upper-tail p-value for \(D_R>0\). Conversely, do not use the half-mixture automatically when multiple or unidentified covariance parameters are involved.
• Check the data regime. The mixture is an asymptotic result. With few independent clusters or weak information about the variance, the approximation may be poor even when the model is otherwise correctly specified.

A Wald test of a variance component is not a substitute: near zero, the estimate is constrained and its sampling distribution is not well approximated by an unconstrained normal distribution. The restricted likelihood-ratio test uses the fitted likelihoods and accounts for the one-sided boundary through its reference law, but it still depends on model assumptions and adequate asymptotic information. It does not solve small-sample calibration or model misspecification by itself.