Skewed Conditional Distributions in Binary GLMMs

Why a Group Effect Can Have an Asymmetric Conditional Distribution

In a Gaussian random-intercept model, the conditional distribution of a group effect is Gaussian, so its mean and mode coincide. A binary GLMM does not generally have that property. Even with a normal prior for the random intercept, the binary likelihood is nonlinear on the log-odds scale. For a group with little data, that likelihood may tilt the prior unevenly, leaving a conditional distribution that is visibly skewed.

This matters because “the predicted group effect” can refer to different summaries of that distribution. The conditional mode is its highest-density point; the conditional mean is its center under squared-error loss. When the distribution is skewed, these summaries need not agree. Neither is automatically the right answer for every purpose.

Definition: Consider group \(j\), with \(y_j\) successes among \(m_j\) conditionally independent Bernoulli observations. Suppose the group has a logistic random intercept \(b_j\sim N(0,\tau^2)\), and the linear predictor for each observation in that group is \(\eta_j+b_j\), where \(\eta_j\) is the fixed-effects contribution. Conditional on fixed model parameters and the observed group outcome, the conditional distribution of \(b_j\) has density proportional to the group likelihood times its normal prior. Its conditional mean is the expectation under that density, and its conditional mode is the value of \(b_j\) at which the density is largest.

The discussion below treats the model parameters as fixed. In an empirical Bayes calculation, this usually means using fitted values for the fixed effects and \(\tau^2\). The resulting conditional distribution describes uncertainty about the group effect given those values; it does not include uncertainty in the estimated model parameters.

The Conditional Density and Its Shape

Write \(p_j(b)=\{1+\exp[-(\eta_j+b)]\}^{-1}\). The binomial likelihood for \(y_j\), given \(b\), is proportional to \(p_j(b)^{y_j}\{1-p_j(b)\}^{m_j-y_j}\). Multiplying by the \(N(0,\tau^2)\) prior gives the conditional density

f(b\mid y_j)\ \propto\ \left\{p_j(b)\right\}^{y_j} \left\{1-p_j(b)\right\}^{m_j-y_j} \exp\left(-\frac{b^2}{2\tau^2}\right).

The binomial coefficient and the normalizing constant of the prior do not depend on \(b\), so they can be omitted when finding the mode or forming a density proportional to \(f(b\mid y_j)\). The log of this unnormalized density, also called the log posterior up to an additive constant, is

\ell(b)=y_j(\eta_j+b)-m_j\log\{1+\exp(\eta_j+b)\} -\frac{b^2}{2\tau^2}+\text{constant}.

Differentiating gives a simple equation for the mode. With \(p=p_j(b)\), the first two derivatives are

\ell'(b)=y_j-m_jp-\frac{b}{\tau^2}, \qquad \ell''(b)=-m_jp(1-p)-\frac{1}{\tau^2}.

The second derivative is strictly negative for finite \(\tau^2>0\). Thus the log density is strictly concave and has at most one mode; because the normal-prior term drives the log density down in both tails, that mode exists and is unique. It solves

y_j-m_jp_j(\hat b_j)-\frac{\hat b_j}{\tau^2}=0.

This equation balances the evidence in the group against the pull of the prior toward zero. It identifies the mode, not the mean. The mean requires averaging \(b\) over the whole conditional density, including its tails.

Why Sparsity Can Produce Skewness

A sparse group contributes little likelihood information relative to the prior. But “little information” does not mean “no effect”: one observed success or failure can still tilt the prior, and the logistic likelihood does not generally tilt it symmetrically. For example, with one success, the likelihood increases with \(b\), but its increase levels off as the success probability approaches one. The normal prior penalizes large positive and negative values symmetrically. Combining a one-sided, flattening likelihood with that symmetric penalty can produce an asymmetric density.

The third derivative makes the changing curvature explicit:

\ell'''(b)=-m_jp(1-p)(1-2p).

Unlike a quadratic log density, whose curvature is constant and whose third derivative is zero, the logistic log likelihood changes curvature across the range of \(b\). That changing curvature is one reason a local symmetric approximation around the mode may not represent the full conditional distribution well. The prior remains normal, but the product of prior and likelihood is generally not normal.

Worked Example: One Success, One Group Member

Suppose a hypothetical group contains one Bernoulli observation, which is a success: \(m_j=1\), \(y_j=1\). Let the fixed-effects contribution be \(\eta_j=0\) and the prior standard deviation be \(\tau=1\). These values define the model and the observed data for this calculation.

The conditional density is proportional to \(p(b)\phi(b)\), where \(p(b)=1/(1+\exp(-b))\) and \(\phi\) is the standard normal density. The log-density derivative is \(1-p(b)-b\). Therefore the mode solves \(b=1-p(b)\). Substitution gives \(b=1-\{1+\exp(-b)\}^{-1}\). Numerically, at \(b=0.401\), the right side is about \(0.4011\); solving the equation gives the conditional mode \(\hat b_j\approx0.4011\).

For the mean, the normalizing constant is \(E\{p(B)\}=1/2\) when \(B\sim N(0,1)\), by the symmetry \(p(b)+p(-b)=1\). Also, integration by parts for a standard normal variable gives \(E\{B p(B)\}=E\{p'(B)\}\), and \(p'(b)=p(b)\{1-p(b)\}\). Hence

E(b\mid y_j=1) =\frac{E\{B p(B)\}}{E\{p(B)\}} =2E\left[\frac{1}{4\cosh^2(B/2)}\right] \approx 2(0.2066)=0.4132.

Thus the conditional mean is about \(0.4132\), compared with the mode \(0.4011\). Both are positive, as expected after observing a success when the fixed-effects contribution is zero, but they are not identical. The mean is slightly larger because it reflects the entire asymmetric density rather than only its peak.

Symmetry and a Contrast

Skewness is possible, not inevitable. A useful check is whether the conditional density is symmetric about a point. If it is symmetric and has a unique mode, its mean and mode agree at the symmetry point. The next two examples illustrate how an outcome reversal creates a mirror image, and how balanced outcomes can yield exact symmetry.

Worked Example: One Failure Instead of One Success

Keep the preceding hypothetical model, with \(m_j=1\), \(\eta_j=0\), and \(\tau=1\), but observe a failure: \(y_j=0\). The conditional density is now proportional to \(\{1-p(b)\}\phi(b)\). Since \(1-p(b)=p(-b)\) and the normal density is symmetric, this density is the reflection of the one-success density around zero.

Consequently its mode is the negative of the previous mode, approximately \(-0.4011\), and its mean is the negative of the previous mean, approximately \(-0.4132\). This reflection follows from the specified model and data: replacing a success with a failure while keeping \(\eta_j=0\) reverses the direction of the likelihood tilt. The mean and mode differ in magnitude for the same reason as before, but their signs reverse.

Worked Example: One Success and One Failure

Now suppose a hypothetical group has \(m_j=2\), \(y_j=1\), \(\eta_j=0\), and \(\tau=1\). Its likelihood is proportional to \(p(b)\{1-p(b)\}\). Replacing \(b\) by \(-b\) swaps \(p(b)\) and \(1-p(b)\), leaving their product unchanged. The prior is also symmetric, so the full conditional density is symmetric about zero.

The score is \(1-2p(b)-b\). At \(b=0\), \(p(0)=1/2\), so the score is \(1-2(1/2)-0=0\). Strict concavity makes this stationary point the unique mode. By symmetry, the conditional mean is also zero. The curvature at the mode is \(-2(1/2)(1/2)-1=-1.5\). The density is not Gaussian just because its mean and mode coincide; here they agree because of symmetry.

Mean, Mode, and Practical Computation

The conditional mean and mode answer different questions. Under squared-error loss, the conditional mean minimizes expected squared prediction error: among choices \(a\), it minimizes \(E\{(b-a)^2\mid y_j\}\). The mode is the point of greatest conditional density. It is often convenient to compute by solving the score equation, and it is a common prediction summary in GLMM software. If the conditional distribution is skewed, neither summary should be described as interchangeable with the other.

For the logistic random-intercept model, the mode can be found by a one-dimensional numerical root finder. The mean can be obtained by numerical integration: first evaluate the likelihood-times-prior density over a sufficiently wide range of \(b\), normalize it so its integral is one, and integrate \(b\) against that normalized density. Adaptive quadrature or other accurate integration methods can be useful when a local approximation is inadequate. A Laplace approximation uses the mode and local curvature to approximate the density as Gaussian; this can be efficient, but it may obscure skewness because it uses a symmetric local shape.

In practice, inspect the conditional density rather than relying only on one summary when the group is sparse or its outcome is strongly one-sided. Useful checks include evaluating the density on a grid, confirming that its numerical integral is close to one after normalization, and comparing the numerical mean with the mode. The model’s strict concavity predicts a single peak; a computed density with multiple peaks or a root finder that appears to find several modes signals a numerical or implementation problem.

The Gaussian random-intercept model is a useful contrast: with a Gaussian likelihood and Gaussian prior, the conditional distribution is Gaussian and mean equals mode. For a binary GLMM, using that Gaussian equality as an assumption is not justified. Conversely, a mean-mode difference is not by itself evidence that the model is wrong; it may simply reflect the conditional distribution implied by the logistic likelihood and observed data.

Common Mistakes and Limitations

Calling the mode the mean. The mode solves the score equation; it does not generally average over the conditional density. Compute or approximate the mean separately if it is the target.
Assuming a normal prior guarantees a normal conditional distribution. The conditional density is proportional to a nonlinear logistic likelihood times the prior. The product is not generally normal.
Reading a positive prediction as a probability. \(b_j\) is a random intercept on the log-odds scale. It is not a probability or a change in probability; it shifts the group’s linear predictor by \(b_j\).
Assuming every sparse group is skewed in the same direction. The direction and degree of asymmetry depend on the observed outcomes and the fixed-effects contribution. Symmetry can make mean and mode coincide, as in the balanced example.
Ignoring plug-in parameters. The conditional distribution described here conditions on the model parameters. It does not automatically include uncertainty in the fixed effects or variance component.

The main limitation of a single summary is that it suppresses shape. The mean and mode do not reveal the full amount of asymmetry or tail behavior. They are useful point summaries, but interpretation should retain the fact that they summarize a conditional distribution under the assumed model and parameter values.

Key takeaway: In a binary random-intercept GLMM, a normal prior does not make the conditional distribution normal. Sparse, one-sided data can yield skewness, so the conditional mean and mode may differ. The score equation identifies the mode; the mean requires averaging over the full conditional density.

Check Your Understanding

Answer each question using the model and results developed above. Short solutions follow the questions.

For \(m_j=1\), \(y_j=1\), \(\eta_j=0\), and \(\tau=1\), what equation defines the conditional mode?
Why does a normal prior not guarantee that the conditional mean equals the conditional mode in a binary GLMM?
For one failure under the same settings as Question 1, what are the conditional mean and mode approximately?
For \(m_j=2\), \(y_j=1\), \(\eta_j=0\), and \(\tau=1\), explain why the conditional mean and mode are both zero.
Which summary minimizes expected squared prediction error under the conditional distribution: the mean or the mode?

Solutions. (1) The mode solves \(1-p(b)-b=0\), with \(p(b)=\{1+\exp(-b)\}^{-1}\); its solution is about \(0.4011\). (2) The conditional density is the product of a nonlinear logistic likelihood and a normal prior, and is not generally Gaussian or symmetric. (3) Reflection around zero gives mean about \(-0.4132\) and mode about \(-0.4011\). (4) The likelihood \(p(b)\{1-p(b)\}\) and the normal prior are both symmetric about zero; strict concavity gives the unique mode at zero, and symmetry gives mean zero. (5) The conditional mean minimizes expected squared prediction error.