LKJ Priors for Random-Effect Correlations

Why Put a Prior on the Correlation Matrix?

In a nonlinear mixed-effects model, a vector of individual random effects can have several components: for example, individual deviations in clearance and volume. Their marginal standard deviations describe the amount of between-individual variation in each component. Their correlations describe whether individuals with above-average values of one component also tend to have above- or below-average values of another. A prior on the correlation matrix is therefore a substantive part of the model, not just a technical way to complete a covariance specification.

A valid correlation matrix must be symmetric, have ones on its diagonal, and be positive definite. These requirements link its off-diagonal entries: choosing each pairwise correlation separately can produce a matrix that is not a valid covariance structure. The LKJ distribution is a family of priors defined directly on valid correlation matrices.

Definition: Let $R$ be a $K \times K$ correlation matrix, where $K$ is the number of random-effect components. An LKJ prior with concentration parameter $\eta>0$ has density $$ p(R \mid \eta) = C_{K,\eta} \det(R)^{\eta-1} $$ over positive-definite correlation matrices. Here $\det(R)$ is the determinant, and $C_{K,\eta}$ is the normalizing constant for dimension $K$ and parameter $\eta$. The density is with respect to the free off-diagonal entries of $R$.

The determinant summarizes how far a correlation matrix is from the singular boundary: it is positive for a positive-definite matrix and approaches zero as the matrix approaches singularity. The identity matrix has determinant one, the largest possible determinant for a correlation matrix. Thus, when $\eta>1$, the density gives more relative weight to matrices with larger determinants. When $\eta=1$, the density is constant over the space of valid correlation matrices. When $0<\eta<1$, the density increases as the determinant approaches zero, favoring matrices near the boundary. The prior is proper for every $\eta>0$.

These descriptions concern whole matrices. They do not imply that the separate correlations are independent, or that their marginal distributions are the same for every matrix dimension. In particular, a constant density over valid matrices is not generally a collection of independent uniform distributions over the pairwise correlations.

The Two-Component Case

With two random-effect components, a correlation matrix is determined by one correlation, $\rho$:

R = \begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}, \qquad -1 < \rho < 1, \qquad \det(R)=1-\rho^2.

The LKJ density is then proportional to $(1-\rho^2)^{\eta-1}$. Its normalizing constant can be obtained by integrating over the allowed values of $\rho$:

\int_{-1}^{1}(1-\rho^2)^{\eta-1}\,d\rho = \int_0^1 u^{-1/2}(1-u)^{\eta-1}\,du = B\left(\frac12,\eta\right).

In the substitution, $u=\rho^2$; the final expression is the beta function. Consequently, the normalized density is $(1-\rho^2)^{\eta-1}/B(1/2,\eta)$. Equivalently, $(\rho+1)/2$ has a Beta distribution with both shape parameters equal to $\eta$. This makes the effect of $\eta$ concrete: the distribution is symmetric around zero, while its shape changes with the concentration parameter.

Worked Example: A Two-Effect LKJ Prior

Suppose a hypothetical model has two random effects and uses $\eta=2$. The density of their correlation is $$ f(\rho)=\frac{(1-\rho^2)}{B(1/2,2)} =\frac{3}{4}(1-\rho^2), \qquad -1<\rho<1, $$ because $B(1/2,2)=4/3$. Find the prior probability that the correlation lies between $-0.5$ and $0.5$.

Integrating the density over that interval gives $$ P(-0.5<\rho<0.5) = \frac{3}{4}\left[\rho-\frac{\rho^3}{3}\right]_{-0.5}^{0.5} = \frac{3}{4}\left(1-\frac{1}{12}\right) = 0.6875. $$

Under this hypothetical prior, the probability assigned to a correlation between $-0.5$ and $0.5$ is 0.6875. The density is symmetric, so positive and negative correlations have equal prior probability. The probability is not a claim about the correlation in any actual population; it follows from the specified prior.

Dimension Changes Pairwise Marginal Priors

For $K$ random-effect components, the marginal distribution of any one pairwise correlation under the LKJ($\eta$) prior is $$ \frac{\rho_{ij}+1}{2} \sim \operatorname{Beta}\left(\eta+\frac{K-2}{2},\, \eta+\frac{K-2}{2}\right). $$ This result applies to each pair $i\ne j$, but the pairwise correlations are generally dependent because they must jointly form a positive-definite matrix. The formula also shows why the same $\eta$ can imply different marginal prior distributions at different dimensions. As $K$ increases, the marginal Beta shape parameters increase, placing relatively more mass near zero.

Worked Example: Pairwise Correlation in Three Dimensions

Consider a hypothetical three-component random-effect vector with an LKJ(1) prior. For any one pairwise correlation, $K=3$ and $\eta=1$, so $$ \frac{\rho_{ij}+1}{2}\sim \operatorname{Beta}(1.5,1.5). $$

For a symmetric Beta($a,a$) variable transformed from $x$ to $\rho=2x-1$, the resulting correlation has mean zero and variance $1/(2a+1)$. Here $a=1.5$, so $$ E(\rho_{ij})=0, \qquad \operatorname{Var}(\rho_{ij})=\frac{1}{2(1.5)+1}=\frac14. $$

The probability that this pairwise correlation lies between $-0.5$ and $0.5$ can also be calculated. Its density is $f(\rho)=\frac{2}{\pi}\sqrt{1-\rho^2}$ on $(-1,1)$, giving $$ P(-0.5<\rho_{ij}<0.5) = \frac{2}{\pi}\left[\frac{\rho\sqrt{1-\rho^2}+\arcsin(\rho)}{2}\right]_{-0.5}^{0.5} = \frac{\sqrt{3}}{2\pi}+\frac13 \approx 0.6090. $$

Although $\eta=1$ gives a uniform density over the space of valid three-dimensional correlation matrices, this pairwise marginal is not uniform over $(-1,1)$. Nor does the marginal calculation mean the three pairwise correlations can be chosen independently.

Combining Correlation and Standard-Deviation Priors

Let $\tau_1,\ldots,\tau_K$ denote the positive standard deviations of the random-effect components, and let $D=\operatorname{diag}(\tau_1,\ldots,\tau_K)$. The random-effect covariance matrix is $$ \Omega = D R D. $$

A common prior specification assigns priors to the standard deviations and an LKJ prior to $R$, independently a priori. If the standard-deviation priors are $p(\tau_1,\ldots,\tau_K)$, this joint specification is $$ p(\tau_1,\ldots,\tau_K,R) = p(\tau_1,\ldots,\tau_K)\,p(R\mid\eta). $$ This is a modeling choice, not a mathematical requirement: dependence between standard deviations and correlations can be specified if there is a substantive reason and a suitable model. Under the independent specification, $\Omega$ is determined by the sampled standard deviations and correlation matrix. Its diagonal entries are variances, and its off-diagonal entries are covariances.

Worked Example: Building a Two-Effect Covariance Matrix

Suppose a hypothetical model has random effects for log clearance and log volume. At one draw from the prior, let their standard deviations be $\tau_{\mathrm{CL}}=0.30$ and $\tau_{\mathrm{V}}=0.50$, in log-scale units, and let their correlation be $\rho=-0.40$. Then $$ D = \begin{pmatrix} 0.30 & 0\\ 0 & 0.50 \end{pmatrix}, \qquad R = \begin{pmatrix} 1 & -0.40\\ -0.40 & 1 \end{pmatrix}. $$

Using $\Omega=DRD$, the diagonal entries are the squared standard deviations, and the off-diagonal entries are the product of the two standard deviations and their correlation: $$ \Omega = \begin{pmatrix} 0.30^2 & 0.30(-0.40)(0.50)\\ 0.30(-0.40)(0.50) & 0.50^2 \end{pmatrix} = \begin{pmatrix} 0.09 & -0.06\\ -0.06 & 0.25 \end{pmatrix}. $$

The variances are $0.09$ and $0.25$ in squared log-scale units; the covariance is $-0.06$ in the product of the two log-scale units. The correlation of $-0.40$ is dimensionless. A correlation draw must be compatible with a positive-definite $R$; in two dimensions this requires only $-1<\rho<1$.

Specification Checks, Alternatives, and Limitations

Before using an LKJ prior in a Bayesian NLME model, check the following features of the specification:

Dimension and support. Confirm that $K$ matches the number of random-effect components and that $R$ is a positive-definite correlation matrix.
Concentration and scale. State $\eta$ explicitly and examine the implied pairwise marginal distributions for the chosen dimension. A particular value of $\eta$ does not imply the same marginal spread for every $K$.
Prior dependence. If standard deviations and $R$ are assigned independent priors, recognize that this is an assumption about the prior. It does not say that the random-effect components are independent: their dependence is encoded by $R$.
Boundary behavior. With $0<\eta<1$, the density increases toward matrices with determinant near zero. Such matrices can represent correlations close to the boundary of positive definiteness, so this choice deserves particular scrutiny.

A common alternative is to fix $R$ to the identity matrix, which rules out correlation rather than estimating it. Another tempting approach is to assign independent uniform priors to every pairwise correlation. For $K=2$, a uniform prior on the single correlation is valid and agrees with LKJ(1). For $K\ge3$, independent pairwise choices can produce a matrix that is not positive definite; even when considering valid matrices, LKJ(1) is uniform over matrices, not independent uniform pairwise entries. The LKJ prior avoids the invalid-matrix problem by assigning density on the space of valid correlation matrices.

The LKJ family is flexible but does not determine whether a chosen prior is scientifically appropriate. It is symmetric around zero and has no prior preference for a positive rather than negative pairwise correlation. A single concentration parameter also shapes the joint matrix distribution through its determinant, so it is not a direct, dimension-invariant statement such as “each correlation is usually below a particular magnitude.” Review the implied distributions for the actual dimension and interpret them alongside the standard-deviation priors.

Common pitfall: “LKJ(1) means each pairwise correlation is independently uniform.” This is true only for the single correlation in a two-component model. For higher dimensions, LKJ(1) is uniform over valid correlation matrices; the pairwise correlations have dimension-dependent marginal distributions and are constrained by positive definiteness.

Check Your Understanding

Answer each question using the definitions and calculations above.

What values of $\eta$ give a constant density over valid correlation matrices, relatively more weight to larger determinants, or relatively more weight near singular matrices?
For $K=2$ and $\eta=1$, what is the prior distribution of the single pairwise correlation?
For $K=4$ and $\eta=1$, what Beta distribution describes the transformed pairwise correlation $(\rho_{ij}+1)/2$?
If two random effects have standard deviations $0.20$ and $0.40$ and correlation $0.50$, what is their covariance?
Why can independent uniform priors on all pairwise correlations fail to define a valid correlation matrix when $K\ge3$?

Solutions. (1) $\eta=1$ gives a constant density; $\eta>1$ relatively favors larger determinants; $0<\eta<1$ relatively favors matrices near the singular boundary. (2) The correlation is uniform on $(-1,1)$. (3) The Beta shape is $\eta+(K-2)/2=1+1=2$, so $(\rho_{ij}+1)/2\sim\operatorname{Beta}(2,2)$. (4) The covariance is $0.20(0.50)(0.40)=0.040$ in the product of the two random-effect units. (5) Pairwise values must jointly satisfy positive definiteness; independent draws do not enforce that constraint.