Fixed-Effect Tests With ML and REML

Why the Estimation Method Matters

A fixed-effect likelihood ratio test asks whether adding one or more population-level parameters improves the fit enough to justify the more complex model. In an NLME model, that question must be answered using maximum likelihood (ML) fits. Restricted maximum likelihood (REML) deliberately removes information associated with estimating fixed effects; its restricted likelihood therefore changes when the fixed-effect design changes. A difference between those restricted likelihoods is not the likelihood ratio for testing the added fixed effect.

This distinction is easy to miss because both methods may return parameter estimates and a quantity labelled “log likelihood.” The label alone does not make two values suitable for a likelihood ratio test. What matters is whether the two objective values are likelihoods for the same observed data under comparable models.

Definition: A fixed-effect likelihood ratio test compares nested models that differ in specified fixed-effect terms, while keeping the response data, likelihood formulation, and random-effects structure the same. The models are fitted by ML. If \(\ell_0\) and \(\ell_1\) are their maximized log likelihoods, with model 1 the larger model, the statistic is \(D=2(\ell_1-\ell_0)\). Under regularity conditions and the null hypothesis, \(D\) is approximately chi-square with degrees of freedom equal to the number of independent fixed-effect parameters added.

What ML and REML Optimize

Let \(y_i\) denote the observations from subject \(i\), \(b_i\) that subject’s random effects, \(\beta\) the fixed effects, and \(\theta\) the parameters governing random-effect and residual variability. In an NLME model, the likelihood for the observed data is generally a marginal likelihood: the subject-specific random effects are integrated out. In schematic form,

L(\beta,\theta;y)=\prod_{i=1}^{m}\int p(y_i\mid b_i,\beta,\theta)\,p(b_i\mid\theta)\,db_i.

ML chooses \(\beta\) and \(\theta\) to maximize this observed-data likelihood. In a fixed-effect comparison, each model gets its own ML estimates of all its parameters, including the shared variance parameters. Thus the smaller model is not evaluated using variance estimates fixed at the larger model’s values.

The classical REML construction is clearest for a linear Gaussian mixed model, \(y\sim N(X\beta,V)\), where \(X\) is the fixed-effect design matrix and \(V\) is the covariance matrix. REML forms likelihood information from error contrasts: linear combinations of the data that eliminate \(X\beta\). For example, if the columns of \(K\) span the contrast space, then \(K'X=0\), and the restricted data are \(K'y\). The restricted likelihood depends on the contrast space, and hence on the fixed-effect design \(X\).

\ell_R =-\frac{1}{2}\left[ (n-p)\log(2\pi)+\log|V|+\log|X'V^{-1}X|+y'Py \right], \qquad P=V^{-1}-V^{-1}X(X'V^{-1}X)^{-1}X'V^{-1}.

Here \(p\) is the number of fixed-effect columns. This expression shows that REML is not simply the ordinary likelihood with the same fixed effects omitted from the output: it contains terms that depend on \(X\). Comparing different fixed-effect designs therefore compares restricted likelihoods based on different error contrasts.

Why Fixed-Effect Tests Require ML

A likelihood ratio test is derived by comparing the maximized likelihoods of a null model and an alternative model for the same observed data. Under the null, the smaller model is a constrained version of the larger one. For a test of an additional fixed-effect coefficient, REML does not provide the required comparison: changing the fixed-effect design changes the restricted likelihood itself. The resulting difference is not a standard likelihood ratio statistic, so its usual chi-square reference distribution does not apply.

The ML comparison instead uses the same marginal data likelihood in both models. Provided the models are nested and the other test conditions hold, the statistic \(D=2(\ell_1-\ell_0)\) measures the improvement in maximized fit. The degrees of freedom are the number of independent parameters constrained by the null, not necessarily the total number of terms described informally as “the covariate effect.”

Conditions for the usual fixed-effect LRT: The models must use the same response observations and likelihood formulation; the null model must be nested within the alternative; the random-effects structure and other modeling choices must be held fixed; and both fits must be valid ML fits with successful optimization. The tested parameters must satisfy the regularity conditions for the chi-square approximation, including that the null value is not a boundary or a point where parameters become unidentified. If these conditions fail, the ordinary chi-square reference may be unreliable or inappropriate.

Worked Example: Testing an Added Covariate by ML

Suppose hypothetical concentration data from 48 subjects are fitted with the same structural model, residual model, and random-effects structure. The null model includes a typical clearance parameter but no body-weight effect on clearance. The alternative adds one fixed-effect coefficient for body weight. Assume the models are nested, use all the same observations, and the optimizer has converged to verified maxima. Their maximized ML log likelihoods are \(\ell_0=-120.0\) and \(\ell_1=-116.5\).

The alternative adds one fixed-effect parameter, so \(df=1\). The likelihood ratio statistic is

D=2(\ell_1-\ell_0) =2[-116.5-(-120.0)] =2(3.5)=7.00.

Under \(H_0\), the reference distribution is approximately \(\chi^2_1\). The upper-tail probability is \(P(\chi^2_1\ge 7.00)\approx 0.0082\). At \(\alpha=0.05\), this is evidence against the null model under the stated assumptions. The result supports including the weight effect as a fixed-effect term in this comparison; it does not by itself establish a causal effect or guarantee better predictions in a new population.

REML and Random-Effects Comparisons

A different question is whether the random-effects structure should include a particular variance or covariance parameter. In the classical linear mixed-model setting, REML can be used to compare covariance structures when the fixed-effect design is identical in both models. Because the same fixed-effect contrasts are used, the restricted likelihoods have a common basis for comparison. ML can also be used for such comparisons.

This is not a license to use REML for every test involving random effects. The models still need to be nested, fitted to the same data, and compared with a reference distribution justified for the null hypothesis. In particular, testing whether a variance component equals zero places the null on the boundary of the permissible parameter space; the ordinary chi-square result may then need a boundary-adjusted reference distribution. The earlier material on boundary tests for random-effect variances addresses that issue.

NLME models add a qualification: unlike the standard linear Gaussian case, there is not one universal exact REML construction for every nonlinear model. Some software offers REML-like estimation through a specified approximation or linearization. The analyst must know what likelihood that procedure computes. A REML comparison of random-effects structures is defensible only when the implementation supports that comparison and the fixed-effect design is held constant; the method label alone is not enough.

Worked Example: Comparing Random-Effects Covariance Structures

Consider hypothetical repeated concentration data fitted with the same fixed-effect design and the same nonlinear structural model. Model 0 gives the subject-level random effects for two parameters a diagonal covariance matrix; model 1 estimates their covariance as well. Suppose the marginal residual and random-effects models are otherwise identical, both fits are valid, and the relevant REML implementation defines comparable restricted likelihoods for this unchanged fixed-effect design. The two REML log likelihoods are \(-82.4\) and \(-80.9\).

The models differ by one covariance parameter. Since the two marginal variances are estimated as positive and the null covariance of zero is an interior value in this setup, the usual one-degree-of-freedom chi-square comparison is the stipulated approximation. The statistic is

$$ D=2[-80.9-(-82.4)]=2(1.5)=3.00. $$

For \(\chi^2_1\), the upper-tail probability is approximately \(0.0833\). At the 0.05 level, this comparison does not provide sufficient evidence to reject the diagonal covariance structure. This conclusion concerns a random-effects covariance term, not an added fixed effect. If the covariance parameterization or null were on a boundary, or the REML implementation did not provide comparable restricted likelihoods, this calculation would require a different justification.

A Practical Workflow and Diagnostics

Before interpreting any likelihood ratio statistic, identify what differs between the two fits. A short audit prevents a common method-selection error and can also uncover invalid comparisons.

Name the hypothesis.
Specify whether the null removes fixed-effect terms or changes the random-effects covariance structure. Do not treat these as interchangeable questions.

Check the models and data.
Confirm nesting, identical response observations, the same likelihood formulation, and successful convergence. For a fixed-effect test, also confirm that the random-effects structure is unchanged.

Choose the appropriate fit criterion.
Use ML for a comparison that changes fixed effects. For a covariance-structure comparison, REML may be suitable when fixed effects are identical and the software’s restricted likelihood supports the comparison.

Set the reference distribution.
Count the independent restrictions and check whether the null is regular or involves a boundary, nonidentifiability, or another complication that changes the usual chi-square approximation.

Inspect fit quality before reporting.
Check optimizer status, gradients or convergence diagnostics available in the software, and sensitivity to reasonable starting values. A difference between failed or inconsistent fits is not reliable evidence.

A further diagnostic is a negative value of \(D\). For properly maximized nested ML fits, the larger model’s maximized likelihood cannot be lower than that of the nested smaller model, so \(D\) should be nonnegative apart from negligible numerical tolerance. A material negative value suggests that optimization failed, the models are not actually nested as specified, the data or likelihoods differ, or the reported values are not comparable.

Common Mistake and Limits

Wrong: “Both outputs report log likelihoods, so I can subtract the REML values from two models that differ by a fixed covariate.” Correction: Different fixed-effect designs produce different restricted likelihoods. Refit both models by ML before conducting the fixed-effect LRT.
Wrong: “REML must never be used in a likelihood comparison.” Correction: A REML comparison can be appropriate for covariance structures when the fixed-effect design is held constant and the implementation and reference distribution justify it.
Wrong: “Any nested pair has a chi-square LRT with degrees of freedom equal to the parameter count.” Correction: The ordinary result relies on regularity conditions. Boundary parameters, unidentified parameters, approximate likelihoods, or failed optimization can invalidate the usual calibration.

The ML fixed-effect test answers a focused model-comparison question under its assumptions. It does not measure the clinical importance of an effect, ensure that the structural model is correct, or remove the need to examine parameter uncertainty and model diagnostics. If the likelihood is approximate or the sample provides weak information, the nominal chi-square p-value may not adequately describe the uncertainty.

Check Your Understanding

For each question, identify the relevant comparison and give the reasoning.

Two nested models differ by one fixed-effect covariate. Which estimation method should be used for the usual LRT, and why?
Two models have the same fixed-effect design but differ by one random-effect covariance parameter. When might a REML comparison be appropriate?
For ML fits with log likelihoods \(-64.2\) for the null model and \(-61.7\) for the alternative, where the alternative adds two regular fixed-effect parameters, calculate \(D\) and state its reference degrees of freedom.
A fixed-effect comparison gives a negative LRT statistic well beyond rounding error. Name two possible explanations.

Solutions. (1) Use ML. REML depends on the fixed-effect design, so the two restricted likelihoods do not form the ordinary likelihood ratio for this test. (2) REML may be appropriate if the fixed-effect design and data are identical, the nested covariance comparison is supported by the implementation, and the null’s reference distribution is justified. (3) \(D=2[-61.7-(-64.2)]=2(2.5)=5.00\), with \(df=2\); the usual reference is \(\chi^2_2\) if the stated regularity conditions hold. (4) Possible explanations include failed optimization of the larger model, non-nested models, different observations or likelihood formulations, or subtracting incomparable criteria such as REML values from models with different fixed effects.