Cholesky Parameterization of Random-Effect Covariance

Why Parameterize the Covariance?

In a nonlinear mixed-effects model, a subject’s random-effect vector can contain several related quantities, such as deviations in clearance and volume. Their covariance matrix describes both the variability of each component and the dependence between components. During Bayesian sampling, however, a proposed matrix must be a valid covariance matrix: it must be symmetric and positive definite. Proposing its entries independently can violate that requirement, producing a matrix that cannot describe any multivariate normal random effect.

A Cholesky parameterization represents the covariance through a triangular factor. This changes the sampling problem: instead of proposing covariance entries that must jointly satisfy positive-definiteness constraints, one works with factor entries that construct a valid covariance matrix by design.

Here, \(I_q\) is the \(q\)-by-\(q\) identity matrix. The usual model assumptions for this representation are that, conditional on the population parameters, the subject effects have mean zero and covariance \(\Omega\), and that the components of \(z_i\) are independent standard normal variables. If subjects are modeled as independent, their \(z_i\) vectors are independent across subjects as well. The Cholesky transformation represents this multivariate normal distribution; it does not itself establish that the normal random-effects assumption is appropriate for the data.

How the Factor Guarantees a Valid Covariance

For a two-component random effect, write the lower-triangular factor as \(L\). Multiplication gives the covariance matrix:

The diagonal of \(\Omega\) gives the two variances, and the off-diagonal entries give their covariance. Notice that the entries of \(L\) are generally not the covariance entries: for example, the second variance is \(\ell_{21}^2+\ell_{22}^2\), not \(\ell_{22}\).

This is the key guarantee. The factor can be parameterized with unrestricted real values below its diagonal and positive diagonal values—for example, by sampling unconstrained values \(a_{jj}\) and setting \(\ell_{jj}=\exp(a_{jj})\). The exponential is positive for every finite \(a_{jj}\), so the resulting factor is nonsingular and its covariance is positive definite. A Cholesky factor with a zero diagonal would instead yield a singular, positive-semidefinite matrix, which lies on the boundary rather than in the positive-definite interior.

Generating Correlated Random Effects

The same factor gives a convenient way to generate random effects. Draw independent standard normal components in \(z_i\), then multiply by \(L\). The resulting components are generally correlated, even though the components of \(z_i\) are independent.

A scale-and-correlation representation is another useful way to interpret the same construction. If \(D=\operatorname{diag}(\sigma_1,\ldots,\sigma_q)\) contains random-effect standard deviations and \(R\) is a correlation matrix, then \(\Omega=DRD\). If \(R=L_RL_R^\mathsf{T}\), a compatible covariance factor is \(L=DL_R\), since \(LL^\mathsf{T}=DL_RL_R^\mathsf{T}D=DRD\). The entries of a correlation Cholesky factor must satisfy the constraints that make \(L_RL_R^\mathsf{T}\) a correlation matrix; they are not arbitrary triangular entries.

Sampling, Priors, and the Change of Variables

In Bayesian NLME sampling, the factor can be the parameter used to construct \(\Omega\) at every iteration. Positive diagonal elements ensure that every finite factor draw gives a valid positive-definite covariance. This avoids invalid covariance proposals, but it does not guarantee good mixing, adequate exploration, or a well-fitting model.

Care is needed when defining priors. A prior directly specified on \(L\) is not automatically the same prior as one specified on \(\Omega\). For example, if the intended prior density is \(p_\Omega(\Omega)\), but the sampled parameter is \(L\), the density on the factor scale must include the change-of-variables term:

The product shown is the absolute Jacobian determinant for the map from the free entries of a lower-triangular \(L\) to the unique entries of \(\Omega=LL^\mathsf{T}\). If a prior is instead defined directly on the factor entries, that is a different prior specification and this particular conversion is not made. The scale-and-correlation approach likewise requires priors to be interpreted on the parameters on which they were specified. The earlier material on LKJ priors concerns priors for correlations; a Cholesky representation is a computational form, not by itself a choice of prior.

Diagnostics, Alternatives, and Limitations

A valid factor can still reveal numerical or modeling problems. Very small diagonal elements imply a variance close to zero, while very large or highly uneven factor entries can produce poorly scaled covariance matrices. In a sampled chain, inspect the implied standard deviations, correlations, and covariance matrices, not only the raw factor entries. Check for extreme correlations, near-zero variances, implausibly large variation, poor exploration, and sensitivity to reasonable prior or initialization choices. These checks assess the fitted model and computation; positive definiteness alone is not a diagnostic of model adequacy.

The Cholesky approach is one alternative to sampling covariance entries directly. Direct-entry sampling is conceptually transparent, but must enforce symmetry and positive-definiteness constraints and can generate invalid proposals. A scale-and-correlation parameterization makes standard deviations and correlations easier to interpret and can support distinct prior choices for them, but its correlation factor has structural constraints of its own. Either approach can represent the same covariance model when used correctly; the preferred parameterization depends on interpretability, prior specification, and sampling behavior.

The Cholesky factor also depends on the ordering of the random-effect components. Reordering those components changes the entries of \(L\), even though the covariance matrix can be represented in the reordered coordinate system. Its off-diagonal elements therefore should not be interpreted as correlations. Finally, the factorization does not fix weak identification: if the data contain little information about a random-effect variance or correlation, the posterior may remain broad or difficult to explore. The transformation guarantees a valid covariance, not that every covariance is scientifically plausible or well estimated.