Correlation Priors with Few Groups

Why a Random-Effect Correlation Can Be Hard to Learn

A random-slope GLMM can describe whether groups with higher-than-average intercepts tend to have steeper or shallower slopes. That relationship is summarized by a correlation parameter, but estimating it is not the same as estimating a fixed-effect association from all individual observations. The correlation describes a distribution of group-level effects, so its information depends substantially on how many groups there are and how clearly each group’s effects can be learned.

A large number of observations within each group may make that group’s intercept and slope more precisely informed. It does not, by itself, provide the same information about the distribution of intercepts and slopes as observing many independent groups. When the number of groups is small, the data may be compatible with a wide range of correlations. A regularizing prior can stabilize inference, but it cannot create information that the data do not contain.

Definition: For group $j$, let $b_j=(b_{0j},b_{1j})^\mathsf{T}$ be a random intercept and random slope, with $b_j\sim N_2(0,D)$, where $$ D= \begin{pmatrix} \sigma_0^2 & \rho\sigma_0\sigma_1\\ \rho\sigma_0\sigma_1 & \sigma_1^2 \end{pmatrix}. $$ Here $\sigma_0,\sigma_1>0$ are group-level standard deviations and $-1<\rho<1$ is their correlation. The correlation is defined only when both standard deviations are positive.

For example, in a logistic random-intercept–random-slope model, observations $Y_{ij}$ in group $j$ can satisfy $Y_{ij}\mid b_j\sim\operatorname{Bernoulli}(p_{ij})$, with $\operatorname{logit}(p_{ij})=\beta_0+\beta_1x_{ij}+b_{0j}+b_{1j}x_{ij}$. Conditional on the group effects and covariates, outcomes are assumed independent; group effects are independent across groups and follow the specified bivariate normal distribution. These assumptions make $\rho$ a feature of the group-level distribution, not a direct correlation between the observed outcomes and covariates.

The likelihood for the GLMM integrates over the unobserved group effects. Thus the data do not generally reveal a separate, exact pair $(b_{0j},b_{1j})$ for every group. When each group has little information about its slope, or when there are few groups to characterize the distribution, this integration can leave the likelihood broad in $\rho$. The introductory material on random-effect scales and correlation priors provides the parameterization; here the emphasis is on diagnosing the information available for the correlation.

What the Correlation Likelihood Reveals

A simple bivariate-normal calculation isolates the group-count issue. Suppose, as a diagnostic approximation, that $G$ independent group-level pairs $(x_j,y_j)$ are observed directly from a bivariate normal distribution with zero means, known unit marginal standard deviations, and correlation $\rho$. This is not the full likelihood for a GLMM, whose group effects are latent; it is a tractable illustration of what information about a correlation looks like when group-level pairs are well measured.

L(\rho)\propto (1-\rho^2)^{-G/2} \exp\left\{-\frac{\sum_{j=1}^{G}(x_j^2-2\rho x_jy_j+y_j^2)}{2(1-\rho^2)}\right\}.

The $y_j^2$ term does not carry a factor of $\rho^2$. The expression follows from the bivariate-normal quadratic form $(x_j^2-2\rho x_jy_j+y_j^2)/(1-\rho^2)$, together with the determinant factor $(1-\rho^2)^{-1/2}$ for each pair.

Writing $A=\sum_jx_j^2$, $B=\sum_jy_j^2$, and $C=\sum_jx_jy_j$, the log likelihood, up to a constant, is $$ \ell(\rho)=-\frac{G}{2}\log(1-\rho^2)-\frac{A-2\rho C+B}{2(1-\rho^2)}. $$

At $\rho=0$, its score is $C$: differentiating the determinant term gives zero there, and differentiating the quadratic term gives the cross-product sum. Thus the observed cross-products are directly relevant to the likelihood’s local direction at zero. But with few pairs, those cross-products can vary greatly from one sample to another. In this simplified model the expected Fisher information at $\rho=0$ is $G$, so the information grows with the number of independent group-level pairs, not with the number of individual observations within each pair. In a GLMM, the information can be lower still if group-specific effects are imprecisely estimated.

Worked Example: A Nearly Uninformative Correlation Likelihood

Consider four hypothetical, directly observed group-level pairs with $A=4$, $B=4$, and $C=1$. These summaries are compatible with four centered bivariate observations. Compare the likelihood at $\rho=0.5$ with that at $\rho=0$.

At zero, the quadratic sum is $A+B=4+4=8$. At $0.5$, it is $$ A-2\rho C+B=4-2(0.5)(1)+4=7. $$ For $G=4$, the likelihood ratio is therefore $$ \frac{L(0.5)}{L(0)} =(1-0.5^2)^{-2} \exp\left\{-\frac{7}{2(1-0.5^2)}+\frac{8}{2}\right\} =(0.75)^{-2}\exp(-2/3) \approx 0.913. $$

The likelihood at $\rho=0.5$ is about 0.913 times its value at zero; equivalently, the likelihood at zero is about 1.10 times as high. This is weak discrimination between these two values, not a posterior probability statement and not evidence that either value is correct. The calculation also depends on the simplified assumptions of known unit marginal scales and directly observed pairs.

How an LKJ Prior Regularizes

For a two-dimensional random-effect correlation matrix, an LKJ prior with shape parameter $\eta>0$ has density proportional to $(1-\rho^2)^{\eta-1}$ on $-1<\rho<1$. Its normalized density is $$ p(\rho\mid\eta)=\frac{(1-\rho^2)^{\eta-1}}{\mathrm{B}(1/2,\eta)}, $$ where $\mathrm{B}$ is the beta function. With $\eta=1$, this is uniform on the correlation interval. With $\eta>1$, density is greater near zero than near the extremes; increasing $\eta$ strengthens that regularization. With $0<\eta<1$, density instead increases toward the extremes. These statements describe prior density, not a guarantee about the posterior.

Bayes’ rule gives $p(\rho\mid\text{data})\propto L(\rho)p(\rho)$. For two candidate values, the ratio of posterior densities is the likelihood ratio multiplied by the prior density ratio. This is a useful way to see the relative influence of a prior, but a density at a single point is not a probability; posterior probabilities require integrating over intervals.

Worked Example: Comparing Uniform and Regularizing Priors

Use the four-pair likelihood from the previous example. Compare the posterior density ratio at $\rho=0.5$ versus $\rho=0$ under an LKJ prior with $\eta=1$ and one with $\eta=2$.

For $\eta=1$, the prior is uniform, so the prior density ratio is 1. The posterior density ratio is therefore approximately $0.913\times1=0.913$.

For $\eta=2$, the prior density is proportional to $1-\rho^2$, so the prior density ratio is $(1-0.5^2)/(1-0^2)=0.75$. The posterior density ratio is $$ \frac{p(0.5\mid\text{data})}{p(0\mid\text{data})} =(0.75)^{-2}\exp(-2/3)(0.75)\approx 0.685. $$

Under this regularizing prior, the posterior density at $0.5$ relative to its density at zero is lower than under the uniform prior. This is a local density comparison, not a comparison of posterior probabilities at two exact values. The example shows how a prior matters when the likelihood is not strongly discriminating; it does not establish that $\eta=2$ is suitable for a particular application.

Regularization is most useful when it expresses a defensible preference for moderate correlations and prevents extreme values from being driven by very limited data. It is not a substitute for checking prior implications. Prior predictive checks can reveal whether the resulting group-level relationships are plausible, while sensitivity analysis can show whether substantive conclusions change under reasonable prior choices.

Scale, Coding, and Weak Identification

Correlation must be interpreted alongside the marginal scales. If $\sigma_1$ is near zero, the slope effects vary little across groups, and the data may have little practical information about how those nearly constant effects co-vary with intercepts. If $\sigma_1=0$ exactly, the covariance $\rho\sigma_0\sigma_1$ is zero for every $\rho$, and $\rho$ is not identified: it is not a meaningful parameter value to estimate in that boundary case. A posterior for $\rho$ can consequently be sensitive to its prior when the slope scale is weakly identified or concentrated near zero.

The intercept–slope correlation also depends on the predictor’s origin. Changing the origin changes which group-level value is called the intercept. This is not a defect in the model, but it means that a correlation should be reported with the predictor coding and centering stated. Centering at a scientifically meaningful value can make the intercept interpretable and may make the correlation easier to discuss.

Worked Example: Centering Changes the Random-Effect Correlation

Suppose a hypothetical random-intercept–random-slope model uses predictor $x$, with $\sigma_0=1$, $\sigma_1=0.5$, and $\rho=-0.4$. Re-express the predictor as $x_c=x-2$. The random intercept at $x_c=0$ is $b_{0c}=b_0+2b_1$, while the slope remains $b_1$.

The original covariance is $\operatorname{Cov}(b_0,b_1)=\rho\sigma_0\sigma_1=(-0.4)(1)(0.5)=-0.2$. Hence $$ \operatorname{Cov}(b_{0c},b_1)=-0.2+2(0.5^2)=0.3. $$ The new intercept variance is $$ \operatorname{Var}(b_{0c})=1^2+2(2)(-0.2)+2^2(0.5^2)=1.2. $$ Therefore, the centered intercept–slope correlation is $$ \operatorname{Corr}(b_{0c},b_1)=\frac{0.3}{\sqrt{1.2}(0.5)}\approx 0.548. $$

The correlation changes from $-0.4$ to about $0.548$ purely because the intercept now refers to the group effect at a different predictor value. The result cautions against interpreting the sign or magnitude without specifying the predictor origin; it does not indicate that centering has changed the underlying outcome data.

Diagnostics, Alternatives, and Limitations

Several signs suggest that the data do not identify the correlation well. The posterior for $\rho$ may remain broad, change substantially under reasonable LKJ shape choices, or track the prior even when the model fit for individual outcomes is stable. A strong dependence between posterior draws of $\rho$ and either random-effect scale can also signal that the data constrain covariance combinations more clearly than they constrain the correlation itself. These are prompts to investigate, not automatic proof of model failure.

A practical assessment compares posterior summaries under plausible correlation priors while holding the other modeling choices fixed, and checks prior predictive implications for group-level intercepts and slopes. Inspect whether the predictor varies sufficiently within groups to estimate slopes, whether the number of groups is adequate for a distributional correlation, and whether a small scale makes the correlation weakly meaningful. Also check sensitivity to the predictor’s centering. No one diagnostic can turn a small number of groups into a large one.

Approach	What it does	Main trade-off
Regularizing LKJ prior	Downweights extreme correlations when $\eta>1$.	Can stabilize weakly identified estimates, but results can depend on the chosen shape.
Uniform correlation prior	Assigns constant density to $\rho$ for two random effects.	Is a baseline choice, not an absence of assumptions; it may permit extremes with little data support.
Fix $\rho=0$	Assumes the random intercept and slope are uncorrelated.	Reduces the number of unknowns, but rules out any nonzero correlation rather than estimating it.
Covariance-matrix prior	Places a joint prior on covariance structure and scales.	May couple scales and correlations; its implications must be assessed on the model’s actual scale.

The main limitation is structural: inference about a distribution of group effects requires information across groups. More observations within existing groups can help estimate their effects, but cannot fully replace additional independent groups. Conversely, a large number of groups does not guarantee good correlation estimation if group slopes are poorly informed or the model is misspecified. Priors can stabilize a difficult inference and make assumptions explicit; posterior conclusions should be described as conditional on those assumptions.

Common mistake: “There are thousands of observations, so the random-intercept–slope correlation must be precisely estimated.” The correction is that the correlation describes variation across groups. Many observations can sharpen estimates within groups, but the number of independent groups and the information about each group’s effects remain central. A regularizing prior can influence an uncertain correlation; it cannot manufacture group-level evidence.

Key takeaway: With few groups, treat random-effect correlations as potentially weakly identified. Examine the likelihood and posterior sensitivity, interpret the prior explicitly, check the random-effect scales, and report the predictor coding that defines the intercept.

Check Your Understanding

Answer each question, then compare with the solutions below.

Why does having many observations per group not necessarily provide the same information about a random-effect correlation as having many groups?
In the four-pair example, what is the likelihood ratio $L(0.5)/L(0)$, and what does it say?
For a bivariate LKJ prior with $\eta=2$, what is the prior density ratio at $\rho=0.5$ versus $\rho=0$?
What happens to the meaning of $\rho$ when the random-slope standard deviation is exactly zero?
Why must predictor centering be reported when interpreting an intercept–slope correlation?

Solutions. (1) The correlation describes the distribution of effects across groups. More observations may improve the estimate of each group’s effects, but additional independent groups provide further information about their joint distribution. (2) The ratio is $(0.75)^{-2}\exp(-2/3)\approx0.913$. The likelihoods at these two values are similar, so this comparison alone indicates little discrimination; it is not a posterior probability. (3) The ratio is $1-0.5^2=0.75$, since the density is proportional to $1-\rho^2$. (4) The covariance is zero for all values of $\rho$, so the correlation is undefined and not identified. (5) Centering changes the random intercept to the group effect at a different predictor value, which can change its covariance and correlation with the slope.

Approach	What it does	Main trade-off
Regularizing LKJ prior	Downweights extreme correlations when \(\eta>1\).	Can stabilize weakly identified estimates, but results can depend on the chosen shape.
Uniform correlation prior	Assigns constant density to \(\rho\) for two random effects.	Is a baseline choice, not an absence of assumptions; it may permit extremes with little data support.
Fix \(\rho=0\)	Assumes the random intercept and slope are uncorrelated.	Reduces the number of unknowns, but rules out any nonzero correlation rather than estimating it.
Covariance-matrix prior	Places a joint prior on covariance structure and scales.	May couple scales and correlations; its implications must be assessed on the model’s actual scale.