Worked Fixed-Effect Model-Comparison Workshop

Why Compare Fixed Effects in a Clustered Model?

In a generalized linear mixed model (GLMM), observations within a cluster may be related because they share random effects. A fixed-effect comparison asks whether specified population-level regression terms improve the model, given its random-effects structure. For example, a researcher might ask whether treatment group and a treatment-by-time interaction are jointly associated with a binary outcome after accounting for clinic-level variation.

Two standard approaches are the Wald test, which assesses the estimated coefficients and their estimated covariance, and the likelihood-ratio test (LRT), which compares maximized likelihoods from nested models. They rely on related large-sample theory, but they use different summaries of the fitted models and can disagree in finite samples.

Model Setup and Conditions for Comparison

Let \(Y_{ij}\) denote the outcome for observation \(i\) in cluster \(j\), and let \(b_j\) denote the cluster’s random effects. A GLMM specifies a conditional response distribution from an exponential-family model and a conditional mean model such as

Here \(g\) is the link function, \(\beta\) is the vector of fixed-effect coefficients, \(x_{ij}\) is the fixed-effect design vector, and \(z_{ij}\) is the random-effect design vector. The model assumes the stated conditional response distribution and link are appropriate, clusters are independent, and the random-effects distribution and covariance structure are adequately specified. Conditional on the random effects, observations are assumed to follow the model’s specified response distribution with its corresponding conditional independence structure.

For a valid standard comparison, the fixed-effect models must be nested and fitted to the same observations. Keep the random-effects structure, response distribution, link, offset, and any weights unchanged. For a GLMM, fit fixed-effect comparisons by maximum likelihood; do not compare restricted maximum-likelihood criteria for models with different fixed-effect design matrices. The likelihood used in the LRT is the likelihood with the random effects integrated out, not a likelihood conditional on estimated random effects.

Under regularity conditions, the usual chi-square reference distributions are large-sample approximations. For a fixed-effect LRT, the null restrictions must be identifiable, the null must not create a boundary problem for nuisance parameters, and the model must be sufficiently regular for the likelihood-ratio asymptotics to apply. In particular, if variance parameters are themselves on a boundary or weakly identified, the ordinary fixed-effect chi-square calibration may not be reliable. The earlier material on parametric bootstrap for fixed-effect likelihood-ratio tests addresses calibration when the usual approximation is questionable.

The Two Tests

Suppose the larger model has fixed-effect vector \(\beta\), and the null hypothesis imposes \(q\) independent linear restrictions \(R\beta=r\). The reduced model is the larger model subject to those restrictions. Write \(\hat\beta\) for the maximum-likelihood estimate and \(\widehat{\mathrm{Var}}(\hat\beta)\) for its estimated covariance matrix.

If the restrictions are regular and the covariance estimate is valid, \(W\) is approximately distributed as \(\chi^2_q\) under \(H_0\). For one coefficient, this reduces to the squared Wald \(z\)-statistic: \(W=(\hat\beta_k/\mathrm{SE}(\hat\beta_k))^2\) when \(H_0:\beta_k=0\).

Because the larger model contains the reduced model, its maximized likelihood cannot be lower, apart from numerical fitting error. Under the same regularity conditions, \(D\) is approximately distributed as \(\chi^2_q\), where \(q\) is the number of independent restrictions (equivalently, the difference in the models’ fixed-effect parameter dimensions).

Why the Reference Distributions Arise

The Wald result follows from the large-sample approximation \(\hat\beta\approx N(\beta,\widehat{\mathrm{Var}}(\hat\beta))\). Under \(H_0:R\beta=r\), the standardized restriction vector \(R\hat\beta-r\) is approximately multivariate normal with mean zero and covariance \(R\widehat{\mathrm{Var}}(\hat\beta)R^T\). Whiten that vector by the inverse square root of its covariance matrix. The resulting \(q\)-vector is approximately standard normal, so the sum of its \(q\) squared components has a \(\chi^2_q\) distribution. That sum is exactly the Wald quadratic form above.

For the LRT, the log likelihood near its maximum is approximately quadratic in the parameters under regularity conditions. The reduction in fit from imposing \(q\) restrictions is then approximately the sum of \(q\) squared standard-normal components. Twice the log-likelihood difference therefore has an approximate \(\chi^2_q\) distribution under the null. This argument depends on regularity; boundary or weak-identification cases require care rather than automatic use of the ordinary chi-square reference.

Interpreting Agreement, Disagreement, and Diagnostics

Wald and likelihood-ratio tests often give similar conclusions when the sample is informative, the likelihood is well behaved, and the null is not close to a problematic boundary. But agreement is not guaranteed. The Wald test uses the fitted coefficient vector and a local covariance approximation at the full-model estimate. The LRT compares the optimized fit under the null with the optimized fit in the larger model. If the likelihood is asymmetric, the estimates are imprecise, the sample is limited, or the model is weakly identified, these summaries can differ appreciably.

Before interpreting either test, check that the models used the same observations and the same non-tested specifications. Confirm that the larger model truly contains the smaller model, that the log likelihoods are maximized on the same likelihood scale, and that the fixed-effect restriction count matches the degrees of freedom used. Convergence warnings, an ill-conditioned fixed-effect covariance matrix, very large standard errors, or estimates that change substantially under modest model refitting are reasons to investigate rather than to rely mechanically on a p-value. If the random-effects variance estimates indicate boundary behavior, ordinary chi-square calibration may need alternatives such as the parametric bootstrap.

A fixed-effect test does not decide whether the random-effects structure is appropriate; that is a different hypothesis and can have different reference-distribution issues. Nor does a nonsignificant result establish that the tested effects are exactly zero. The conclusion is limited to the evidence against the specified null under the fitted model and its assumptions. For a coefficient test, describe its scale and units; for example, a logistic coefficient is a change in conditional log odds per unit of its predictor, not a change in probability.

Common Pitfalls

• Comparing models that differ in more than the tested fixed effects. A change in random effects, observations, offsets, or response specification confounds the comparison. Keep these features fixed.
• Using the wrong degrees of freedom. Use the number of independent restrictions, not the number of observations or clusters. A two-coefficient joint null generally has \(q=2\).
• Calling the Wald test and LRT interchangeable. They are distinct procedures. Report the statistic and p-value for the method actually used, especially when results disagree.
• Using a REML comparison for different fixed-effect designs. Fit the fixed-effect comparison by maximum likelihood.
• Treating failure to reject as proof of no effect. A large p-value means the data do not provide sufficient evidence against the specified null at the chosen level; it does not prove the null true.