Why Logistic GLMM Variance Components Lack Conjugate Updates

Why the Conditioning Matters

In a Gaussian hierarchical model, normal likelihoods and conjugate priors can give variance components familiar closed-form conditional updates. In a logistic GLMM, the binary-response likelihood is not Gaussian, and integrating over the random effects generally produces a variance posterior that is not in a standard conjugate family. There is an important qualification: conditioning on the random effects can still give a conjugate variance update. The difficulty arises when those effects are integrated out and the target is the variance posterior given the observed data alone.

This distinction matters because “the variance update is not conjugate” is incomplete unless the conditioning is specified. We will use a random-intercept model to identify what cancels in the conditional posterior and what remains in the marginal posterior. The focus is the form of these distributions, not a particular sampling algorithm.

Model and assumptions: For groups \(j=1,\ldots,J\), suppose \(Y_{ij}\mid b_j,\beta\) are conditionally independent Bernoulli variables with \(\operatorname{logit}\{P(Y_{ij}=1\mid b_j,\beta)\}=x_{ij}^{T}\beta+b_j\). Assume \(b_j\mid\tau^2\) are independent \(N(0,\tau^2)\), and assign \(\tau^2\) an inverse-gamma prior with fixed hyperparameters \(a>0\) and \(c>0\), using the density proportional to \((\tau^2)^{-a-1}\exp(-c/\tau^2)\). In the results below, \(\beta\) is treated as fixed or conditioned upon.

The Bernoulli likelihood for one group, conditional on its random intercept, can be written as \(L_j(b_j)=\prod_i p_{ij}^{y_{ij}}(1-p_{ij})^{1-y_{ij}}\), where \(p_{ij}=\operatorname{logit}^{-1}(x_{ij}^{T}\beta+b_j)\). The variance \(\tau^2\) does not appear in this likelihood once \(b_j\) is given. It appears in the normal density assigned to \(b_j\). That separation is the reason a conditional conjugate update is possible.

The Conditional Update

The inverse-gamma prior is conjugate to the normal random-effects density when the random effects are observed or conditioned upon. Multiplying the prior density by the \(J\) normal densities gives

p(\tau^2\mid b_1,\ldots,b_J,y,\beta) \propto (\tau^2)^{-a-1}\exp\left(-\frac{c}{\tau^2}\right) (\tau^2)^{-J/2}\exp\left(-\frac{\sum_{j=1}^J b_j^2}{2\tau^2}\right).

The logistic likelihood is absent from this expression because it is constant as a function of \(\tau^2\) when the \(b_j\)'s are held fixed. Collecting powers and exponential terms yields the conditional distribution

\tau^2\mid b_1,\ldots,b_J,y,\beta \sim \operatorname{IG}\left(a+\frac{J}{2},\ c+\frac{1}{2}\sum_{j=1}^J b_j^2\right).

Here \(\operatorname{IG}(A,C)\) denotes the inverse-gamma distribution with density proportional to \(t^{-A-1}\exp(-C/t)\) for \(t>0\). The observations still matter to the joint analysis: they inform the random effects. But once the random effects are conditioned upon, the observations add no further factor involving \(\tau^2\).

Worked Example: A Conjugate Update Given the Random Effects

Suppose there are four groups, the prior is \(\tau^2\sim\operatorname{IG}(2,1.5)\), and a hypothetical set of conditioned random-intercept values is \(0.2,-0.3,0.4,0.1\). Calculate the conditional posterior for \(\tau^2\).

First compute the sum of squares: \(0.2^2+(-0.3)^2+0.4^2+0.1^2=0.04+0.09+0.16+0.01=0.30\). The shape parameter becomes \(2+4/2=4\), and the scale parameter becomes \(1.5+0.30/2=1.65\). Thus

\tau^2\mid b_1,\ldots,b_4,y,\beta \sim \operatorname{IG}(4,1.65).

This update is valid under the stated normal random-effects model and inverse-gamma prior. It is conditional on those four random effects; it is not, by itself, the posterior distribution of \(\tau^2\) given only the binary observations.

What Changes When Random Effects Are Integrated Out

If the random effects are not conditioned upon, the likelihood for \(\tau^2\) must average the logistic likelihood over their normal distribution. For the random-intercept model, the marginal likelihood is

p(y\mid\beta,\tau^2) = \prod_{j=1}^J \int_{-\infty}^{\infty} L_j(b)\, \frac{1}{\sqrt{2\pi\tau^2}} \exp\left(-\frac{b^2}{2\tau^2}\right)\,db.

Each integral is an expectation of a product of logistic probabilities under a normal distribution: a logistic-normal integral. In general, it does not reduce to a factor of the form \((\tau^2)^{-r}\exp(-s/\tau^2)\) for fixed \(r\) and \(s\). Consequently, multiplying this marginal likelihood by an inverse-gamma prior does not generally produce another inverse-gamma distribution. This is the sense in which the logistic GLMM lacks a standard conjugate update for the variance when the random effects have been marginalized.

Key distinction: Under the stated prior, \(\tau^2\mid b,y,\beta\) has an inverse-gamma conditional distribution. In contrast, \(p(\tau^2\mid y,\beta)\), which integrates out the random effects, generally does not. Conditional conjugacy has not disappeared; it is not the same as conjugacy for the marginal posterior.

Worked Example: Two Successes Produce a Nonconjugate Marginal Factor

Consider one group with two binary observations, both equal to 1, and fixed intercept \(\beta=0\). The group intercept has distribution \(b\sim N(0,\tau^2)\). Conditional on \(b\), the probability of observing two successes is \(\operatorname{logit}^{-1}(b)^2\). Therefore the marginal likelihood factor is

q(\tau^2)=E\left[\operatorname{logit}^{-1}(B)^2\right], \qquad B\sim N(0,\tau^2).

At \(\tau^2=0\), the random intercept is \(B=0\), so \(q(0)=\operatorname{logit}^{-1}(0)^2=(1/2)^2=0.25\). As \(\tau^2\) increases without bound, \(B\) has increasingly large magnitude and is positive with probability \(1/2\). For positive diverging values, the squared logistic probability tends to 1; for negative diverging values, it tends to 0. Since the squared logistic probability is bounded between 0 and 1, its expectation tends to \(1/2\). Thus this likelihood factor varies from 0.25 near zero variance toward 0.5 at very large variance; it is not a constant.

With prior \(\operatorname{IG}(a,c)\), the marginal posterior kernel is proportional to \((\tau^2)^{-a-1}\exp(-c/\tau^2)q(\tau^2)\). If this were another inverse-gamma kernel, its ratio to the prior kernel would have the form \(K(\tau^2)^{-d}\exp(-e/\tau^2)\) for constants \(K,d,e\). A ratio with a finite, nonzero limit at zero and at infinity must have \(e=0\) and \(d=0\), making it constant. But \(q\) has different limits, 0.25 and 0.5. Hence this marginal posterior is not inverse-gamma. The conclusion concerns this collapsed variance posterior, not the conditional update given \(b\).

A Special Case and What It Teaches

Not every logistic-normal integral is difficult or informative. In a single-observation example, symmetry makes the marginal likelihood exactly constant in the variance. This special case helps separate the existence of a nonconjugate integral from the separate question of how much information the data contain about a variance component.

Worked Example: One Bernoulli Observation Gives No Variance Information

Suppose one group has one observation \(Y=1\), the fixed intercept is zero, and \(B\sim N(0,\tau^2)\). Its marginal success probability is

P(Y=1\mid\tau^2)=E\{\operatorname{logit}^{-1}(B)\}.

The normal distribution is symmetric around zero, and \(\operatorname{logit}^{-1}(-b)=1-\operatorname{logit}^{-1}(b)\). Therefore \(E\{\operatorname{logit}^{-1}(B)\}=E\{1-\operatorname{logit}^{-1}(B)\}\). Adding these equal quantities gives \(2E\{\operatorname{logit}^{-1}(B)\}=1\), so the marginal likelihood is \(1/2\) for every \(\tau^2\).

For any proper prior, Bayes' rule then gives \(p(\tau^2\mid Y=1)=p(\tau^2)\): the observation has not updated the variance prior. In particular, if the prior is \(\operatorname{IG}(2,1)\), the posterior remains \(\operatorname{IG}(2,1)\). This is a special symmetry result, not evidence that logistic GLMM variance posteriors are generally conjugate.

Why Numerical Evaluation Is Usually Needed

In typical data sets, the marginal posterior for \(\tau^2\) contains one logistic-normal integral per group, or a higher-dimensional integral for multiple correlated random effects. These integrals usually have no elementary closed form. The posterior can still be defined precisely: it is proportional to the prior density times the marginal likelihood. But evaluating that target, summarizing it, or generating draws from it generally requires numerical methods.

This issue is distinct from whether a conditional inverse-gamma draw is available. A joint approach can retain the random effects as unknown quantities and use their conditional distribution when updating \(\tau^2\); a collapsed approach integrates them out and must handle the logistic-normal marginal likelihood. Conditional methods can exploit the conjugate expression derived above, while marginal numerical methods work directly with the integrated target. Neither label guarantees computational ease: the joint target can have strong dependence between \(b_j\) and \(\tau^2\), while marginal integration can be computationally demanding, especially with many random effects.

Gaussian hierarchical models provide a useful comparison, but the exact source of conjugacy depends on the conditioning and likelihood being used. A normal random-effects density paired with an inverse-gamma prior supplies the conditional update in either a Gaussian or logistic outcome model, provided the random effects are conditioned upon. The logistic response makes the marginal likelihood integral nonstandard. One should therefore compare like with like: conditional updates with conditional updates, and integrated likelihoods with integrated likelihoods.

Diagnostics, Limitations, and Common Pitfalls

Before using a claimed variance update, write down the posterior kernel and check which quantities have been conditioned on. If the expression includes the random effects as fixed values, verify that the response likelihood has no remaining dependence on \(\tau^2\). If the random effects have been integrated out, inspect the resulting integral rather than assuming the conditional formula still applies.

Check the prior parameterization. In this tutorial, \(\operatorname{IG}(a,c)\) has kernel \(t^{-a-1}\exp(-c/t)\). Other sources may use a different scale or rate convention; substituting hyperparameters without checking can change the update.
Check the conditioning set. The update \( \operatorname{IG}(a+J/2,c+\sum b_j^2/2)\) is conditional on the \(J\) random effects. Applying it as the posterior given only \(y\) is incorrect.
Check whether the prior is conjugate. With a prior other than the stated inverse-gamma form, the conditional variance posterior need not be inverse-gamma, even though the normal random-effects likelihood is unchanged.
Check model dimensions. With vector random effects and a covariance matrix, the scalar inverse-gamma calculation does not automatically apply to each variance separately. Correlations and joint covariance structure change the relevant conditional distribution.
Check numerical integration where marginalization is used. Approximation error can affect tail behavior and summaries, particularly near zero variance or when the data weakly distinguish variance values.

A common mistake is to reason, “The outcome is Bernoulli, so there can be no conjugate variance update.” The correction is that the Bernoulli likelihood is independent of \(\tau^2\) given the random effects, so the stated conditional update is conjugate. The opposite mistake is to use that conditional formula after integrating the random effects. The correction is to retain the logistic-normal integral in the marginal posterior; in general, it prevents the standard inverse-gamma form.

The conclusions have limits. Conjugacy depends on the specified normal random-effects distribution and inverse-gamma prior, and the nonconjugacy claim is about the general marginal logistic model, not every possible data configuration. Symmetry can make an integral constant, as in the one-observation example. Special numerical approximations may also yield convenient expressions, but an approximation does not make the exact marginal posterior conjugate.

Key takeaway: In a logistic GLMM, the inverse-gamma variance update remains available when normal random effects are conditioned upon. Integrating those effects generally leaves a logistic-normal marginal likelihood, so the variance posterior is not in the standard conjugate family and usually requires numerical posterior evaluation.

Check Your Understanding

Answer each question, then compare with the solutions included below.

For \(J\) scalar normal random effects and an \(\operatorname{IG}(a,c)\) prior, state the conditional posterior parameters for \(\tau^2\) given the random effects.
Why does the logistic response likelihood cancel from \(p(\tau^2\mid b,y,\beta)\), but not from the marginal posterior \(p(\tau^2\mid y,\beta)\)?
One group has a single Bernoulli observation, fixed intercept zero, and a symmetric normal random intercept. What is the marginal probability of success, and what does this imply for the variance prior?
For two successes in one group, explain why the marginal likelihood factor is not constant in \(\tau^2\), using its limits at zero and at large variance.
A colleague uses the inverse-gamma conditional update as though it were \(p(\tau^2\mid y)\). Identify the error and the necessary correction.

Solutions. (1) The conditional posterior is \(\operatorname{IG}(a+J/2,\ c+\sum_{j=1}^J b_j^2/2)\). The \(J\) normal densities contribute \(J/2\) to the shape and half the sum of squared effects to the scale. (2) Given \(b\), the logistic likelihood depends on \(b\) and \(\beta\), but not on \(\tau^2\), so it is constant in the variance posterior kernel. After integrating over \(b\), the logistic likelihood is averaged against a normal density whose spread depends on \(\tau^2\), leaving a variance-dependent integral. (3) Symmetry and \(\operatorname{logit}^{-1}(-b)=1-\operatorname{logit}^{-1}(b)\) give marginal success probability \(1/2\) for every \(\tau^2\); thus the posterior for \(\tau^2\) equals its prior. (4) For two successes, the factor is \(E\{\operatorname{logit}^{-1}(B)^2\}\); it tends to \(0.25\) at zero variance and \(0.5\) as variance tends to infinity, so it is not constant. (5) The colleague has confused a conditional posterior with a marginal posterior. To obtain \(p(\tau^2\mid y)\), integrate the random effects out and retain the resulting logistic-normal likelihood factor; in general, numerical evaluation is then needed.