Why a Zero Variance Changes the Question
A random-effect correlation can look like an ordinary nuisance parameter in a generalized linear mixed model (GLMM). But if either random-effect variance is zero, the correlation is no longer defined. This is more than a technicality: when a likelihood-ratio test (LRT) sets a variance to zero, the null model may lose a parameter that was meaningful under the alternative. The usual reference distribution for a boundary test cannot be chosen without noticing that loss of identification.
Consider a random intercept and random slope for a covariate \(x_{ij}\), where \(i\) indexes clusters and \(j\) indexes observations within a cluster. Write the linear predictor as \(g\{E(Y_{ij}\mid b_i)\}=x_{ij}^{\mathsf T}\beta+b_{0i}+b_{1i}x_{ij}\), where \(g\) is the link function, \(b_{0i}\) is the random intercept, and \(b_{1i}\) is the random slope. Assume the random-effects vector is normally distributed with mean zero and covariance matrix \(\Sigma\). The fixed-effect structure, response distribution, link, and other model assumptions are held the same in the models being compared.
Here \(\tau_0^2\) and \(\tau_1^2\) are the random-intercept and random-slope variances, \(\tau_{01}\) is their covariance, and \(\rho\) is their correlation when both variances are positive. A covariance matrix must be positive semidefinite. For this two-dimensional matrix, that requires nonnegative diagonal entries and a nonnegative determinant.
The Covariance Constraint at the Boundary
The determinant condition gives \(\tau_0^2\tau_1^2-\tau_{01}^2\geq 0\), or equivalently \(\tau_{01}^2\leq\tau_0^2\tau_1^2\). If \(\tau_1^2=0\), this inequality becomes \(\tau_{01}^2\leq0\), so \(\tau_{01}=0\). The same conclusion holds if \(\tau_0^2=0\). In a parameterization using standard deviations and correlation, \(\tau_{01}=\rho\tau_0\tau_1\) is zero for every allowable \(\rho\) when either standard deviation is zero. The distribution under that boundary model therefore cannot distinguish one value of \(\rho\) from another.
Worked Example: A Correlation That Cannot Be Estimated
Suppose a fitted random-effects covariance matrix is at the random-slope boundary, with \(\tau_0^2=4\) and \(\tau_1^2=0\). What value must the covariance have, and what is the random-intercept/random-slope correlation?
The determinant constraint gives \(\tau_{01}^2\leq(4)(0)=0\). Since a square cannot be negative, \(\tau_{01}^2=0\), and therefore \(\tau_{01}=0\).
The correlation formula would be \(\rho=\tau_{01}/(\tau_0\tau_1)=0/(2\cdot0)\). This ratio is undefined; it is not equal to zero. In the equivalent parameterization \(\tau_{01}=\rho\tau_0\tau_1\), every value of \(\rho\) gives covariance zero at this boundary. The likelihood under the boundary model consequently contains no information for estimating \(\rho\).
The same reasoning applies when the random-intercept variance is zero, and when both variances are zero. A reported correlation near \(1\) or \(-1\) in a fit with a variance essentially zero should not be interpreted as a well-estimated association: near the boundary, tiny changes in the covariance can produce large changes in the correlation ratio.
Consequences for a Likelihood-Ratio Test
An LRT compares the maximized log likelihoods of a null model and an alternative model. For a test of the random-slope variance, the null hypothesis is \(H_0:\tau_1^2=0\); the alternative allows \(\tau_1^2>0\). If the random-intercept variance is positive and the alternative estimates an intercept-slope covariance, the alternative can have an estimable correlation. Under the null, however, the covariance is forced to zero and the correlation disappears. The null is on the boundary, and a nuisance parameter present under the alternative is unidentified under the null.
This breaks a key condition behind the ordinary likelihood-ratio theorem: parameters must be identifiable under the null, in addition to the other regularity conditions. The common boundary result for one nonnegative variance, with reference law \(\tfrac12\chi_0^2+\tfrac12\chi_1^2\), is appropriate only under conditions that justify that particular local likelihood geometry. It is not a universal correction for every test involving one variance. In a correlated random-intercept/random-slope model, the unidentified correlation means that the reference law needs to be justified for that model and test; it cannot be selected just by counting the variance being tested.
The relevant sampling asymptotics for clustered data are generally driven by the number of independent clusters, not merely the total number of observations. Even a theoretically appropriate asymptotic reference law may be inaccurate with few clusters or weak information about the random slope. The reference distribution may require a more specific derivation or a calibrated procedure. The important conclusion here is not a new universal mixture formula, but a check that must precede selecting one: determine which parameters are identified under the null.
Worked LRT Interpretation
Worked Example: An Unidentified Correlation Under the Null
In a hypothetical clustered-outcome analysis, the null model sets the random-slope variance to zero while retaining a random intercept. The alternative estimates both variances and their covariance. Suppose the maximized log likelihoods are \(\ell_0=-218.60\) and \(\ell_1=-216.55\). Explain what the LRT statistic says and what it does not establish.
The statistic is \(T=2(\ell_1-\ell_0)=2\{-216.55-(-218.60)\}=2(2.05)=4.10\). The alternative fits better in maximized log likelihood, as it must when it includes the null as a special case.
Under the null, the random-slope variance is zero. Positive semidefiniteness therefore forces the intercept-slope covariance to zero, and the correlation is undefined. Under the alternative, if both estimated variances are positive, a correlation may be estimated. Consequently, the null and alternative differ in more than the presence of a nonnegative variance: a correlation parameter is present under the alternative but not identified under the null.
For reference, a routine \(\chi_1^2\) calculation would give an upper-tail probability of about \(0.0428\) at \(4.10\); the simple half-\(\chi_1^2\) calculation would give about \(0.0214\). Neither number is automatically a valid p-value for this correlated model. A p-value requires a reference law justified for the model’s null geometry, or an appropriate calibration method. With only the log likelihoods and no such justification, the defensible report is the statistic \(T=4.10\), the model comparison, and the fact that the p-value remains to be calibrated. The statistic alone does not establish that the random slope is needed.
This example illustrates why a seemingly small modeling choice matters. If a scientific model specifies independent random intercepts and slopes, the covariance is fixed at zero by assumption, not estimated as a correlation. That is a different model from one that estimates an unrestricted covariance. The simpler boundary reference law may be available under the independent-effects formulation and its regularity conditions, but that does not make it valid for the correlated formulation.
When a Familiar Boundary Mixture Is Justified
A useful comparison is a test of one random-effect variance when its covariance with other random effects is fixed at zero by the model. Suppose the remaining nuisance parameters are identifiable and are not themselves on a boundary, and the design provides information about the tested variance. Under the standard one-component boundary conditions, the LRT has the half-at-zero, half-\(\chi_1^2\) reference law. Its point mass at zero accounts for the possibility that the constrained estimate remains at the boundary. This is the usual single-variance result, not a general rule for a correlated random-effects block.
Worked Example: A Test With Covariance Fixed at Zero
Consider a hypothetical model in which the random intercept and slope are assumed independent, so their covariance is fixed at zero. All other nuisance parameters are identifiable and interior, and the number of independent clusters is adequate for the asymptotic approximation. Suppose the LRT statistic for adding the random-slope variance is \(T=2.7055\).
Under the stated one-variance boundary conditions, the reference law is \(\tfrac12\chi_0^2+\tfrac12\chi_1^2\). Since \(T>0\), the point mass at zero contributes no upper-tail probability. The p-value is \(\tfrac12P(\chi_1^2\geq2.7055)\). The \(\chi_1^2\) upper tail is approximately \(0.1000\), so the mixture p-value is approximately \(\tfrac12(0.1000)=0.0500\).
At a \(0.05\) significance level, this is right at the conventional threshold, subject to rounding and the adequacy of the asymptotic conditions. The conclusion applies to this model with covariance fixed at zero. It cannot be transferred automatically to the correlated model, where the correlation is unidentified under the zero-slope null.
The comparison is not a choice between an always-correct “simple” model and an always-correct “complex” model. It is a distinction between assumptions. Fixing covariance at zero may be scientifically reasonable, but it rules out correlated cluster-specific intercept and slope deviations. If that correlation is important to the scientific model, treating it as fixed merely to obtain a convenient reference law changes the question being tested.
Diagnostics, Failure Modes, and Reporting
A fitted covariance matrix should be checked against its constraints. For two random effects, verify that estimated variances are nonnegative and that \(\hat{\tau}_{01}^2\leq\hat{\tau}_0^2\hat{\tau}_1^2\), allowing for numerical tolerance. A fit with a variance at or very near zero is a boundary or near-boundary fit. A covariance estimate that violates the determinant condition indicates an invalid covariance matrix or a numerical/reporting problem.
Other warning signs include optimizer convergence warnings, different solutions from different starting values, a correlation estimate that shifts sharply when a tiny variance change occurs, and a Hessian that is nearly singular. These signs do not by themselves prove that the model is wrong; they indicate that standard local approximations and numerical estimates deserve scrutiny. In particular, a software output that prints \(\hat{\rho}=0.99\) alongside a slope variance rounded to zero does not show strong evidence of a near-perfect population correlation.
| Model or test | Status of correlation under the null | Implication for the LRT |
|---|---|---|
| One variance tested; associated covariance fixed at zero by assumption | No correlation is estimated in the model | A single-variance boundary law may apply if its regularity conditions hold |
| One variance tested; covariance estimated under the alternative | Correlation is undefined when the variance is zero | Do not assume a simple mixture without a model-specific justification |
| Both variances positive and covariance estimated | Correlation is defined, subject to the covariance constraint | Ordinary interior inference may be relevant only away from boundaries and under other regularity conditions |
In reporting, state the null model, whether the covariance is fixed or estimated, which variance reaches the boundary, and how the LRT reference distribution was obtained. If calibration is unavailable, report the likelihood-ratio statistic without presenting an unjustified p-value. A procedure that accounts for the null model’s unidentified parameter structure is preferable to an automatic chi-square lookup.
Common Mistake
Correction: The covariance is forced to zero, but the correlation is undefined, not zero. If the correlation is estimated under the alternative, it is an unidentified nuisance parameter under the null. The half-\(\chi_1^2\) law requires its own assumptions and cannot be inferred merely from testing one variance.
Check Your Understanding
Answer each question, then compare with the solutions below.
- If \(\tau_0^2=9\) and \(\tau_1^2=0\), what must \(\tau_{01}\) be? Is \(\rho\) zero?
- Why is the correlation unidentified under \(H_0:\tau_1^2=0\) when the covariance is estimated under the alternative?
- A researcher reports \(T=3.00\) for this correlated-model test and uses the half-\(\chi_1^2\) reference law without further justification. What is the inferential problem?
- What is the half-\(\chi_1^2\) p-value for \(T=2.7055\) when the covariance is fixed at zero and the single-variance boundary assumptions hold?
Solutions. (1) The determinant condition gives \(\tau_{01}^2\leq9(0)=0\), hence \(\tau_{01}=0\). The correlation is undefined because its denominator includes \(\tau_1=0\); it is not zero. (2) Under the null, every value of \(\rho\) produces the same covariance, zero, so the null likelihood cannot distinguish correlation values. (3) The statistic is defined, but the p-value may not be: the unidentified null correlation invalidates automatic use of that reference law. A model-specific justification or calibration is needed. (4) The p-value is \(\tfrac12P(\chi_1^2\geq2.7055)\approx\tfrac12(0.1000)=0.0500\), under the assumptions stated in the question.