Conditional and Marginal Separation

Why Conditional and Marginal Likelihoods Can Tell Different Stories

In a binary GLMM, a cluster’s random effect shifts the linear predictor for its observations. If the random effects are treated as unknown cluster-specific quantities, they can make an outcome pattern look perfectly predictable. But the marginal likelihood does not select one best random effect for each cluster: it averages the conditional likelihood over the assumed random-effects distribution. Those are different operations, and a separation pattern in one objective does not automatically describe the other.

The distinction also depends on what is meant by “conditional separation.” It may refer to a fixed-effects likelihood evaluated at fixed random effects, or to a joint conditional likelihood maximized over both fixed effects and cluster effects. These are not interchangeable. The first has a separating direction in the fixed-effect design; the second can become large by sending cluster effects toward extreme values even when the fixed effects themselves are not separated.

Definition: For a binary GLMM with clusters $j=1,\ldots,J$, observations $i=1,\ldots,n_j$, and random effects $b_j$, the conditional likelihood is $$ L_c(\beta;b)=\prod_{j=1}^J\prod_{i=1}^{n_j} p_{ij}(b_j)^{y_{ij}}\{1-p_{ij}(b_j)\}^{1-y_{ij}}, \qquad \operatorname{logit}\{p_{ij}(b_j)\}=x_{ij}^{T}\beta+z_{ij}^{T}b_j. $$ Here $L_c(\beta;b)$ is a function of $\beta$ when $b$ is held fixed. The joint conditional likelihood treats both $\beta$ and $b$ as variables in $L_c$, without including a random-effects density. If $b_j$ has density $f(b_j\mid D)$, the marginal likelihood is $$ L_m(\beta,D)=\prod_{j=1}^J\int \left[\prod_{i=1}^{n_j}P(Y_{ij}=y_{ij}\mid b_j,\beta)\right] f(b_j\mid D)\,db_j. $$

The usual logistic-normal GLMM assumes conditionally independent Bernoulli outcomes within clusters given the random effects, a logit link, and a specified random-effects distribution, commonly $b_j\sim N(0,D)$, independently across clusters. Separation is a statement about how the likelihood behaves along a parameter direction; it is not simply a label for extreme fitted probabilities. In particular, a cluster-specific effect that predicts that cluster’s outcomes does not by itself show that a fixed-effect coefficient diverges.

Three Likelihood Questions

It helps to ask which parameters are being varied and which objective is being examined. A fixed-effects separation direction holds the random effects fixed and sends a direction in $\beta$ toward infinity, improving the conditional likelihood. A joint conditional separation direction may instead change both $\beta$ and $b$. Marginal likelihood evaluates neither objective: for each candidate $(\beta,D)$, it averages over possible random effects using their probability distribution.

Integration can therefore change the implication of a cluster pattern when the apparent fit requires unusual or extreme cluster effects. But integration is not a universal cure for separation. If a direction in the fixed effects improves the conditional likelihood for every possible random-effect value, integration preserves that monotone improvement. The next derivation states this distinction precisely.

Result: Suppose a fixed direction $d$ has the property that, for every fixed $b$, $L_c(\beta+td;b)$ is nondecreasing in $t$. Then, for a fixed random-effects distribution, $$ L_m(\beta+td,D) =\prod_j\int L_{c,j}(\beta+td;b_j)f(b_j\mid D)\,db_j $$ is also nondecreasing in $t$. This follows because each integrand is nondecreasing and each density is nonnegative. Thus a genuine fixed-effect separating direction cannot be dismissed merely because the likelihood is marginal.

This result is about a direction that improves the fixed-effects likelihood at every fixed $b$. It does not say that all joint conditional separation directions have this property. A joint direction can improve fit by changing a cluster effect while the same fixed-effect value remains finite and well supported by the integrated likelihood.

Worked Example: Cluster Effects Separate the Conditional Data, but Not the Fixed Intercept

Consider two clusters, each with one observation. In cluster A the outcome is $y_A=1$, and in cluster B it is $y_B=0$. Use a random-intercept model with linear predictor $\beta+b_j$, and first consider the conditional likelihood without a random-effects density.

At $\beta=0$, choosing $b_A=4$ and $b_B=-4$ gives $P(y_A=1\mid b_A)=\operatorname{logit}^{-1}(4)=0.9820$ and $P(y_B=0\mid b_B)=\operatorname{logit}^{-1}(4)=0.9820$. The conditional likelihood is therefore $0.9820^2=0.9643$. Sending $b_A$ to $+\infty$ and $b_B$ to $-\infty$ makes this likelihood approach 1. The joint conditional likelihood can thus fit these two observations arbitrarily well by separating them with the cluster effects.

Now hold a normal random-intercept variance $\tau^2$ fixed and integrate. Define $m(\beta)=E\{\operatorname{logit}^{-1}(\beta+b)\}$, where $b\sim N(0,\tau^2)$. The two clusters are independent marginally, so $L_m(\beta)=m(\beta)\{1-m(\beta)\}$. Symmetry of the normal distribution and the identity $\operatorname{logit}^{-1}(-u)=1-\operatorname{logit}^{-1}(u)$ give $m(0)=0.5$. Since $0\leq m(\beta)\leq1$, $$ L_m(\beta)=m(\beta)\{1-m(\beta)\} =\frac14-\{m(\beta)-\tfrac12\}^2\leq\frac14. $$ The maximum occurs at $\beta=0$, where $L_m=0.25$. The finite marginal estimate for the intercept contrasts with the joint conditional fit, which can approach 1 by taking opposite cluster effects to infinity. This example concerns joint conditional separation through $b_j$; the fixed-effects likelihood conditional on fixed $b_j$ is a different question.

When a Fixed-Effect Separating Direction Carries Through

For a logistic model, a fixed-effect direction $d$ changes the linear predictor by $t x_{ij}^{T}d$. If $x_{ij}^{T}d$ points in the outcome-correct direction for every observation—nonnegative for successes and nonpositive for failures—with at least one strict improvement, increasing $t$ improves the conditional likelihood at every fixed $b$. Contributions with $x_{ij}^{T}d=0$ do not change; contributions with a strict sign improve. The result above then shows that the marginal likelihood cannot decrease along that direction. Integration over random effects does not remove this fixed-design separation direction.

Worked Example: A Fixed-Effect Direction Remains Monotone After Integration

In one cluster, observe two outcomes: a failure at $x=0$, and a success at $x=1$. The model is $\operatorname{logit}\{P(Y_i=1\mid b)\}=\beta x_i+b$, with $b\sim N(0,1)$. The fixed-effect slope direction is separating: the failure has $x=0$, so its probability is unaffected by $\beta$; the success has $x=1$, so its success probability increases with $\beta$.

For fixed $b=0$, the conditional likelihood is $L_c(\beta;0)=0.5\operatorname{logit}^{-1}(\beta)$. At $\beta=0$, it is $0.5(0.5)=0.25$; at $\beta=2$, it is $0.5(0.8808)=0.4404$; and as $\beta$ tends to infinity it approaches $0.5$. The conditional likelihood has no finite maximizing slope.

The marginal likelihood is $L_m(\beta)=E[\operatorname{logit}^{-1}(-b)\operatorname{logit}^{-1}(\beta+b)]$. For each fixed $b$, its integrand increases with $\beta$, since $$ \frac{\partial}{\partial\beta} \left[\operatorname{logit}^{-1}(-b)\operatorname{logit}^{-1}(\beta+b)\right] = \operatorname{logit}^{-1}(-b)\operatorname{logit}^{-1}(\beta+b) \left[1-\operatorname{logit}^{-1}(\beta+b)\right]>0. $$ The normal density is positive, so integration preserves strict increase. Hence the marginal likelihood also has no finite maximizing slope in this example. This is a fixed-effect separating direction, not merely a pattern that can be fitted by choosing extreme cluster effects.

How to Diagnose Which Claim Is Supported

A report that says “the model is separated” is incomplete unless it identifies the likelihood and the parameters involved. A practical diagnosis should distinguish a fixed-effect direction from a pattern explained by cluster-specific effects, then examine the likelihood that is actually used for inference. Numerical trouble alone is not proof of separation: convergence failures, sparse factor levels, and boundary estimates can also produce extreme estimates or large uncertainty.

Specify the objective.
State whether random effects are held fixed, jointly varied with fixed effects, or integrated out. Do not use “conditional likelihood” without saying which meaning is intended.

Identify the direction.
Check whether the alleged separating direction changes fixed-effect predictors in the outcome-correct direction for the observations, or instead relies on sending particular cluster effects to extreme values.

Check the fitted marginal objective.
For marginal-likelihood estimation, inspect estimates and uncertainty alongside convergence diagnostics, and check whether the marginal likelihood has a finite maximum in the relevant fixed-effect direction.

A useful comparison is between conditional and marginal fitted probabilities, but they answer different questions. Conditional probabilities describe outcomes given specified random effects. Marginal probabilities average over the random-effects distribution and need not equal the inverse-logit of the fixed-effects linear predictor. A conditional fitted probability near zero or one is therefore not, by itself, evidence that the integrated likelihood has the same shape.

Worked Example: Reading a Cluster Pattern Without Overstating It

Suppose a hypothetical binary-outcome data set has four clusters. In two clusters every recorded outcome is 1, and in the other two every outcome is 0. Each cluster contains observations at both $x=0$ and $x=1$. A random-intercept model is being considered.

Within each cluster, the predictor $x$ does not separate the outcomes: the same cluster has both predictor values but only one outcome value. However, a joint conditional fit can assign large positive intercepts to the all-success clusters and large negative intercepts to the all-failure clusters. That is evidence of a cluster-effect fitting pattern, not evidence that the fixed slope for $x$ has a separating direction.

For a fixed-effects separation check, compare outcomes across the $x$ values rather than assigning each cluster a freely chosen effect. Since the description gives both outcome values at both $x$ values across the data, there is no strict slope direction that raises the predictors for every success while lowering them for every failure. The cluster pattern may still make estimation difficult, but it does not establish fixed-slope separation. A marginal fit must be assessed using its integrated likelihood and the assumed random-effects distribution; the conditional pattern alone does not determine its fixed-effect estimate.

Common Mistakes and Limits

Equating a high conditional likelihood with marginal separation. A conditional fit may achieve high likelihood by choosing extreme cluster effects. The marginal likelihood averages over effects rather than optimizing each one freely; the two likelihood values and their behavior need not match.
Assuming random effects automatically repair fixed-effect separation. If a fixed-effect direction improves the conditional likelihood at every fixed random-effect value, the marginal likelihood is also nondecreasing along it. Integration is not a general cure.
Calling every extreme fitted probability separation. Extreme probabilities can reflect sparse data, weak identification, or optimization problems. The defining issue is likelihood behavior along a parameter direction, not a probability cutoff.

The comparison also depends on model specification. The integrated likelihood uses the chosen random-effects distribution and covariance structure. Changing that distribution changes the averaging operation, and estimating its parameters can complicate the likelihood further. The examples above isolate the distinction by treating the random-effects distribution as specified; they do not claim that every finite marginal estimate is stable or well identified in a more complex model.

Key takeaway: Ask whether separation is in the fixed-effects likelihood with random effects held fixed, in the joint conditional likelihood through cluster effects, or in the integrated marginal likelihood. Joint cluster-effect separation may not imply a divergent marginal fixed-effect estimate. A genuine fixed-effect separating direction that improves the conditional likelihood for every fixed random effect does carry through integration.

Check Your Understanding

Answer each question using the likelihood named in the question.

In the two-cluster example with one success and one failure, why can the joint conditional likelihood approach 1 while the marginal likelihood is maximized at $\beta=0$?
For a fixed random-effects distribution, suppose $L_c(\beta+td;b)$ is nondecreasing in $t$ for every $b$. What follows for the marginal likelihood along the same direction?
In the one-cluster example with a failure at $x=0$ and a success at $x=1$, calculate the conditional likelihood at $b=0,\ \beta=1$.
Does observing all successes in some clusters and all failures in others, by itself, prove that a fixed slope is separated? Explain.
A fitted model reports very large conditional probabilities and failed convergence. What additional distinction must be checked before concluding that the marginal fixed-effect likelihood is separated?

Solutions. (1) The joint conditional likelihood can send the success cluster’s effect to $+\infty$ and the failure cluster’s effect to $-\infty$, fitting both observations. In the marginal model, $L_m(\beta)=m(\beta)\{1-m(\beta)\}$, which is at most $1/4$ and reaches that value at $\beta=0$. (2) The integrand is nondecreasing for every random-effect value, and integration against a nonnegative density preserves that ordering, so the marginal likelihood is nondecreasing. (3) $L_c(1;0)=0.5\operatorname{logit}^{-1}(1)=0.5(0.7311)=0.3655$. (4) No. The pattern may be fitted through cluster effects; fixed-slope separation requires a fixed-effect direction that predicts outcomes across the design. (5) Check whether the pattern is joint separation through extreme cluster effects or a fixed-effect separating direction at fixed random effects, and assess the integrated likelihood and optimization diagnostics separately.