Testing Random-Effects Covariance Structures

Why Test a Random-Effects Covariance Structure?

An NLME model can allow subject-specific random effects to vary independently, or it can allow them to covary. For example, a model with random effects on clearance and volume may allow subjects with higher-than-typical clearance also to have systematically higher or lower-than-typical volume. The diagonal structure rules out that covariance; a correlated structure estimates it. The practical question is whether the data provide sufficient evidence for the additional flexibility.

A likelihood-ratio test (LRT) can compare these structures when one is nested in the other. Its familiar chi-square calibration, however, depends on more than simply counting the extra parameters. In particular, the null model must be identifiable and sufficiently far from a boundary. If a random-effect variance is zero, its correlation with another random effect is not defined, and the usual chi-square argument can fail.

Definition: Let the subject-level random-effects vector be \(\eta_i\sim N(0,\Omega)\), where \(\Omega\) is its covariance matrix. A diagonal covariance structure sets all off-diagonal elements of \(\Omega\) to zero. An unstructured covariance structure estimates all variances and covariances, subject to \(\Omega\) being positive semidefinite. A covariance parameter is an off-diagonal element; a correlation is the covariance divided by the product of the corresponding standard deviations.

For two random effects, the structures can be written as

\Omega_{\mathrm{diag}}= \begin{pmatrix} \omega_1^2 & 0\\ 0 & \omega_2^2 \end{pmatrix}, \qquad \Omega_{\mathrm{unstr}}= \begin{pmatrix} \omega_1^2 & \omega_{12}\\ \omega_{12} & \omega_2^2 \end{pmatrix}.

The corresponding correlation is \(\rho_{12}=\omega_{12}/(\omega_1\omega_2)\), when both standard deviations are positive. Thus, the diagonal model is the unstructured model with \(\omega_{12}=0\). If both variances are positive, zero covariance is an interior value for the covariance parameter: small positive and negative covariances are both possible while keeping \(\Omega\) positive definite.

When the Chi-Square Calibration Applies

Fit the null diagonal model and the alternative correlated model to the same data, using the same fixed-effects and residual-error specifications. Let \(\ell_0\) and \(\ell_1\) be their maximized log likelihoods. The LRT statistic is \(D=2(\ell_1-\ell_0)\), which is nonnegative when the alternative contains the null and both fits are successfully maximized.

Result: If the null covariance matrix is an interior, identifiable point of the alternative model, the added parameters are identifiable, and the usual large-sample likelihood regularity conditions hold, then under the null hypothesis \(D\) is approximately chi-square distributed with degrees of freedom equal to the number of independent covariance parameters added. For a diagonal versus unstructured \(p\)-dimensional random-effects matrix with positive variances, that number is \(q=p(p-1)/2\).

The conditions matter. In the \(p\)-dimensional case, a symmetric covariance matrix has \(p(p+1)/2\) distinct entries. The diagonal model estimates \(p\) variances, so the unstructured model adds \(p(p+1)/2-p=p(p-1)/2\) covariances. This count gives the usual degrees of freedom only if the null is regular: the extra parameters must be identifiable under the null and the null point must not be on a relevant boundary.

The basic justification is Wilks’ likelihood-ratio result. Near an identifiable interior null value, the log likelihood can be approximated by a quadratic function of the parameter deviations. After accounting for estimation of nuisance parameters, the \(q\) additional coordinates contribute a quadratic form in \(q\) asymptotically standard-normal score components. That quadratic form has a \(\chi^2_q\) distribution. This argument requires the \(q\) coordinates to be meaningful and locally estimable under the null; a nominal parameter count cannot replace those requirements.

D=2(\ell_1-\ell_0) \ \overset{\text{approximately under }H_0}{\sim}\ \chi^2_q, \qquad q=\frac{p(p-1)}{2}.

Worked Example: Testing One Covariance With Positive Variances

Suppose hypothetical NLME fits use the same observations, fixed effects, and residual model. The diagonal random-intercept/random-slope model has maximized log likelihood \(\ell_0=-412.80\). The correlated model has \(\ell_1=-410.29\). Both estimated random-effect variances are positive, and the fitted covariance matrix is positive definite. Test the diagonal structure at \(\alpha=0.05\).

There is one added covariance parameter, so \(q=1\). The positive variances mean that zero covariance is an interior value: a small covariance of either sign is possible. Assuming the likelihood fits are valid and the sample is large enough for the approximation, the regular \(\chi^2_1\) reference is appropriate.

Calculate the statistic. \[ D=2(\ell_1-\ell_0)=2[-410.29-(-412.80)]=2(2.51)=5.02. \]

The upper-tail probability is \(P(\chi^2_1\geq 5.02)\approx0.02506\), or \(p\approx0.0251\) to four decimal places. Since \(0.0251<0.05\), reject the diagonal-structure null at the 5% level. In this hypothetical setting, the data provide evidence that allowing the random intercept and slope to covary improves the likelihood. This result does not establish that the estimated covariance is biologically important or that the model is otherwise adequate.

When a Correlation Is Unidentified Under the Null

A central complication arises when a variance associated with a random effect is zero under the null. For two random effects, positive semidefiniteness requires \[ |\omega_{12}|\leq \omega_1\omega_2. \] If \(\omega_2=0\), this forces \(\omega_{12}=0\), whatever the value of a putative correlation \(\rho_{12}\). Equivalently, the covariance parameter cannot vary independently, and \(\rho_{12}=\omega_{12}/(\omega_1\omega_2)\) is undefined because its denominator is zero. The data under that null cannot identify the correlation.

This is not the regular test of one identifiable covariance set to zero. If the alternative also allows the variance to become positive, the comparison may involve testing a variance at its boundary as well as introducing a correlation that is unidentified under the null. The degrees of freedom are not automatically one, and the boundary mixture reference used for some single variance tests does not automatically solve this joint problem. The null distribution depends on the model and constraints.

Worked Example: A Random Slope Variance at Zero

Consider a hypothetical model with random intercept \(b_{0i}\) and random slope \(b_{1i}\). Suppose the null model fixes the slope variance at zero, while the more flexible fit allows a positive slope variance and an intercept-slope correlation. Explain why an ordinary one-degree-of-freedom covariance test is not justified.

Write the covariance as \(\omega_{01}=\rho_{01}\omega_0\omega_1\). Under the null, \(\omega_1=0\), so \[ \omega_{01}=\rho_{01}\omega_0(0)=0 \] for every possible \(\rho_{01}\). Also, \(\rho_{01}\) itself is undefined at \(\omega_1=0\). Therefore, the null does not identify the correlation parameter, and the comparison has changed more than one regular, identifiable covariance coordinate: it also allows the slope variance to move away from zero.

Consequently, counting “one correlation” and comparing the LRT to \(\chi^2_1\) would have no stated regularity justification. The null distribution should instead be calibrated by a procedure appropriate to this boundary and nonidentifiability problem, commonly a parametric bootstrap under the fitted null. The bootstrap must reproduce the null model, refit both structures to each simulated data set, and use the resulting LRT distribution.

Worked Comparison With Three Random Effects

With three random effects, a diagonal matrix has three variances and the unstructured matrix adds three covariances. The regular chi-square test is available when all three null variances are positive, the diagonal matrix is an interior point of the admissible unstructured matrices, the added covariances are identifiable, and the likelihood and sample-size conditions are adequate. If a variance is zero or nearly zero, those conditions require closer scrutiny.

Worked Example: Counting Covariances and Applying the Test

Suppose three hypothetical random effects represent subject-specific deviations in clearance, volume, and an absorption-rate parameter. The diagonal model has log likelihood \(-250.10\); the unstructured model has log likelihood \(-245.90\). Assume that all three estimated null variances are positive, the alternative covariance matrix is positive definite, and the two fits use the same data and other model components.

The unstructured model adds one covariance for each pair: clearance-volume, clearance-absorption, and volume-absorption. Thus \(q=3(3-1)/2=3\). The stated positivity and identifiability conditions support the regular \(\chi^2_3\) reference, subject to the usual large-sample approximation.

The statistic is \[ D=2[-245.90-(-250.10)]=2(4.20)=8.40. \] At \(\alpha=0.05\), the 95th percentile of \(\chi^2_3\) is approximately \(7.815\). Since \(8.40>7.815\), reject the diagonal-structure null at the 5% level. The test indicates evidence against the joint null that all three covariances are zero; it does not, by itself, identify which covariance drives the result.

Diagnostics, Alternatives, and Limitations

Before interpreting an LRT, verify the nesting and the fit. The alternative must contain the null as a special case, both models must use the same observations and comparable likelihood calculations, and optimization must have reached credible maxima. A negative \(D\) beyond small numerical error signals a fitting or comparison problem because the larger model should not have a lower maximized likelihood. Check convergence messages, parameter estimates, positive definiteness, and sensitivity to starting values. A covariance estimate near the positive-definite boundary, or a variance close to zero, warns that regular calibration may be unreliable.

The main approaches answer different questions or make different assumptions:

Approach	What it does	Important qualification
Regular likelihood-ratio test	Tests the joint null that the added, identifiable covariances are zero.	Uses \(\chi^2_q\) only under the regularity conditions described above.
Parametric bootstrap under the null	Simulates data under the fitted null, refits both models, and estimates the null distribution of \(D\).	Can address nonstandard calibration, but depends on a credible null simulation model and successful refitting.
Profile likelihood	Examines how the likelihood changes as covariance or correlation values are constrained.	Can reveal flatness or boundary behavior; it does not by itself make an invalid chi-square reference valid.
Information criteria	Compare models using a fit-and-complexity criterion rather than a null-hypothesis test.	A preference between models is not a p-value or evidence that a particular covariance is nonzero.

A parametric bootstrap is often a practical choice when a variance is on the boundary or a correlation is unidentified under the null. For each replicate, simulate under the fitted null, fit the null and alternative by the same procedure, and calculate \(D\). The proportion of bootstrap statistics at least as large as the observed statistic estimates the null tail probability. Poor convergence or too few successful fits can make that estimate unreliable, so report those failures rather than silently discarding them.

Even under regular conditions, the chi-square calibration is asymptotic. Sparse or uninformative data, weakly estimated variances, strong parameter dependence, or approximate likelihood calculations can reduce its accuracy. A statistically detectable covariance also does not establish that the covariance is important for exposure predictions, dosing decisions, or the scientific interpretation of the model.

Common Mistakes

Counting a correlation without checking identifiability. A correlation is defined only when both standard deviations are positive. If one variance is zero, the correlation is undefined under that null.
Using one degree of freedom for every diagonal-versus-correlated comparison. With \(p\) random effects and a regular diagonal null, the unstructured model adds \(p(p-1)/2\) covariances.
Applying a familiar boundary mixture automatically. A reference distribution for a single variance test does not automatically calibrate a test involving a variance boundary and an unidentified correlation.
Interpreting a significant joint test as identifying a specific covariance. The LRT tests the added covariance parameters jointly; it does not say which one is responsible.
Confusing better likelihood with a valid test result. A larger model can fit at least as well, but the p-value still requires a justified null distribution and reliable optimization.

Key takeaway: A diagonal-versus-unstructured LRT has the usual \(\chi^2_{p(p-1)/2}\) calibration only when the null variances are positive and the added covariances are identifiable at an interior null point. If a variance is zero, its associated correlation is unidentified; use a calibration suited to that nonregular setting rather than relying on parameter counting alone.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

For four random effects with positive null variances, how many covariances does an unstructured model add to a diagonal model?
A two-random-effect comparison has \(D=3.84\), with both null variances positive and the regularity conditions otherwise credible. Using the \(\chi^2_1\) reference, what is the approximate p-value, and what is the 5% decision?
Why is the correlation between random intercept and slope undefined when the slope variance is zero?
What should be checked if the more complex model reports a lower maximized log likelihood than the nested null model?

Solutions. (1) \(q=4(4-1)/2=6\) covariances. (2) For \(\chi^2_1\), \(P(\chi^2_1\geq3.84)\approx0.0500\); this is approximately the 5% threshold, so using the unrounded tail probability and a strict \(p<0.05\) rule, it is not below 0.05. (3) The correlation is covariance divided by the product of the standard deviations; with zero slope standard deviation the denominator is zero, and positive semidefiniteness forces the covariance to zero. (4) Check that the models are genuinely nested and use the same data and likelihood, then inspect convergence, optimization, and any numerical or fitting errors. A well-maximized nested alternative should have a likelihood at least as large as the null.