Why Separation Needs a GLMM-Specific Distinction
In a binary generalized linear mixed model, a fixed-effect predictor may perfectly order observed zeros and ones even though the model also includes cluster-specific random effects. That pattern is called separation. It matters because the conditional Bernoulli likelihood can keep increasing as a fixed-effect coefficient moves farther in one direction, rather than attaining its maximum at a finite coefficient value.
The key distinction is between the Bernoulli likelihood conditional on the random effects and the marginal likelihood, which integrates over their distribution. Separation is defined here through the conditional Bernoulli contributions and directions in the fixed-effect design. Random effects enter the linear predictor, but they do not change the fixed-effect design vectors used to define those directions.
Model and Definition
Let \(Y_i\in\{0,1\}\) be the binary response for observation \(i\), with fixed-effect design vector \(x_i\in\mathbb{R}^p\) and random-effect design vector \(z_i\). A standard binary GLMM with a logit link is
Here \(\beta\) is the vector of fixed effects, and \(b\) denotes the latent random effects, commonly assigned a distribution such as \(N(0,\Sigma)\). Conditional independence means the responses are independent given \(b\) and the model parameters. For defining separation, hold \(b\) at a finite value and consider changing \(\beta\). The term \(z_i^\mathsf{T}b\) is then an offset in each linear predictor.
Define \(s_i=2Y_i-1\), so \(s_i=1\) for a success and \(s_i=-1\) for a failure. A fixed-effect direction is a vector \(d\in\mathbb{R}^p\). Its signed margin for observation \(i\) is \(s_i x_i^\mathsf{T}d\): a positive margin moves the predictor toward the observed outcome as \(\beta\) moves along \(d\), while a zero margin leaves that observation's predictor unchanged.
The direction is allowed to involve any linear combination of the fixed effects, including the intercept if the model has one. The definition is about the observed design and outcomes. A random intercept or random slope changes \(z_i^\mathsf{T}b\), but does not change the signed margins \(s_i x_i^\mathsf{T}d\).
Why the Conditional Likelihood Rises
Given \(b\), the Bernoulli log likelihood as a function of \(\beta\) is
Consider the path \(\beta(t)=\beta+td\), for \(t\geq 0\). Its signed predictor for observation \(i\) is \(s_i\eta_i(t)=s_i\eta_i(0)+t\,s_i x_i^\mathsf{T}d\). If all signed margins are nonnegative, none of these signed predictors decreases. Since \(-\log\{1+\exp(-u)\}\) is an increasing function of \(u\), no observation's log-likelihood contribution decreases along the path. A positive margin makes that contribution strictly increase at every finite \(t\).
This gives the basic likelihood consequence: under complete separation, every contribution improves and the conditional log likelihood approaches its upper bound of zero as \(t\) tends to infinity. Under quasi-complete separation, some contributions improve toward zero while at least one zero-margin contribution remains unchanged. In either case, the conditional likelihood does not attain its supremum at a finite point along this path.
This is a statement about the conditional Bernoulli likelihood. The GLMM likelihood for the observed data is marginal: it integrates the conditional likelihood against the random-effects distribution. It is not obtained by simply treating the latent \(b\)'s as observed data, and the two likelihoods should not be confused. For a fixed random-effects distribution independent of \(\beta\), pointwise monotonicity along a separating direction also gives monotonicity of the integrated likelihood; however, a full GLMM fit may estimate covariance parameters as well. The conditional definition identifies a fixed-effect separation pattern, not every feature of that joint optimization problem.
Worked Example: Complete Separation With Clustered Observations
Suppose a hypothetical study observes two people in each of two clinics. Let \(x_i=(1,x_i)^\mathsf{T}\), where \(x_i\) is a binary predictor, and include a clinic random intercept. The observed rows are: clinic A has \((x,Y)=(0,0)\) and \((1,1)\); clinic B has the same two pairs. Check whether the fixed effects separate the outcomes and show the likelihood behavior with the random intercepts held at zero.
| Clinic | \(x_i\) | \(Y_i\) | \(s_i\) |
|---|---|---|---|
| A | 0 | 0 | -1 |
| A | 1 | 1 | 1 |
| B | 0 | 0 | -1 |
| B | 1 | 1 | 1 |
Choose \(d=(-1,2)^\mathsf{T}\). For an \(x=0\) row, \(x_i^\mathsf{T}d=-1\), so the signed margin for \(Y=0\) is \((-1)(-1)=1\). For an \(x=1\) row, \(x_i^\mathsf{T}d=-1+2=1\), so the signed margin for \(Y=1\) is \(1(1)=1\). All four margins are positive; this is complete separation.
Set the initial fixed effects to zero and hold both clinic random intercepts at zero. Along \(\beta(t)=t(-1,2)^\mathsf{T}\), the \(x=0\) predictors equal \(-t\), and the \(x=1\) predictors equal \(t\). Each of the four observed outcomes then has probability \(\operatorname{logit}^{-1}(t)\). Therefore
At \(t=0\), \(L_c(0)=0.5^4=0.06250\). At \(t=1\), it is approximately \(0.7311^4=0.2856\); at \(t=2\), it is approximately \(0.8808^4=0.6019\); and at \(t=4\), it is approximately \(0.9820^4=0.9300\). It approaches 1 as \(t\) increases, but no finite \(t\) gives likelihood 1. The random intercepts are part of the model, but their presence does not alter the signed fixed-effect margins in this check.
Worked Example: Quasi-Complete Separation
In a second hypothetical data set, a random-intercept model has two observations with \(x=0\), one outcome of each kind, and two observations with \(x=1\), both outcomes equal to 1. Use an intercept and slope as fixed effects. Consider \(d=(0,1)^\mathsf{T}\), the direction that increases only the slope.
For both \(x=0\) observations, \(x_i^\mathsf{T}d=0\), so their signed margins are zero, regardless of their outcomes. For each \(x=1,Y=1\) observation, the signed margin is \(1(1)=1\). The margins are all nonnegative, with both zero and positive values; this is quasi-complete separation.
For arithmetic, hold the random effects at zero and set the intercept to zero. At slope \(t\), the two \(x=0\) observations each have success probability \(0.5\), so their joint likelihood contribution is \(0.5(1-0.5)=0.25\). The two \(x=1\) successes each have probability \(\operatorname{logit}^{-1}(t)\). Thus
At \(t=0\), this is \(0.25(0.5)^2=0.06250\). At \(t=2\), \(\operatorname{logit}^{-1}(2)\approx0.8808\), giving \(0.25(0.8808)^2\approx0.1940\). At \(t=4\), the value is approximately \(0.25(0.9820)^2=0.2411\). As \(t\) tends to infinity, the likelihood approaches \(0.25\), the unchanged contribution from the two \(x=0\) observations. That limit is not attained at finite \(t\).
Worked Example: A Cluster Pattern That Is Not Fixed-Effect Separation
Suppose a hypothetical random-intercept model has one observation in each of two clinics. Both observations have the same fixed-effect design \(x_i=(1)\), containing only an intercept. The outcome is 1 in clinic A and 0 in clinic B. Take finite random intercepts \(b_A=1.5\) and \(b_B=-1.5\), and set \(\beta_0=0\).
The conditional success probabilities are \(\operatorname{logit}^{-1}(1.5)\approx0.8176\) in clinic A and \(\operatorname{logit}^{-1}(-1.5)\approx0.1824\) in clinic B. The conditional likelihood for the two observed outcomes is
For a fixed-effect direction \(d\), the intercept design gives the same \(x_i^\mathsf{T}d=d\) for both observations. The success in clinic A requires \(d\geq0\) for a nonnegative signed margin; the failure in clinic B requires \(-d\geq0\), or \(d\leq0\). Both can hold only at \(d=0\), which has no strict margins. Hence there is no fixed-effect separating direction. The clinic-specific outcomes are accommodated by the stated random-intercept values, but that cluster pattern is not itself separation by the fixed intercept.
The check concerns directions in the fixed-effect design. It does not declare the random-effects distribution appropriate or establish that a fitted GLMM will be stable.
Diagnostics, Comparisons, and Limits
A useful separation check asks whether the observed outcomes can all be moved in the correct direction by changing fixed effects alone. The definition provides a direct conceptual diagnostic: look for a direction whose signed margins are all nonnegative, and determine whether they are all positive or include zeros. In practice, the design matrix and outcome pattern can be examined computationally; numerical instability or a very large coefficient is a warning sign, not by itself a proof of separation.
Keep separation distinct from other fitting problems. A flat likelihood direction can have causes other than a binary separating pattern. A singular random-effects fit concerns the estimated random-effects covariance structure; it does not establish fixed-effect separation. Likewise, an optimizer warning can arise from scaling, finite-precision arithmetic, or difficult curvature even when there is no separating direction. The earlier material on convergence diagnostics is useful for those distinct questions.
| Question | What it concerns | What it does not establish |
|---|---|---|
| Is there a fixed-effect separating direction? | Signed margins \(s_i x_i^\mathsf{T}d\) in the conditional Bernoulli model | That every optimization warning is caused by separation |
| Is the random-effects covariance singular? | The estimated covariance structure for \(b\) | That fixed effects separate the outcomes |
| Does the marginal GLMM fit behave well? | The likelihood after integrating over random effects, often with covariance parameters estimated | That the conditional and marginal likelihoods are interchangeable |
The fixed-effect definition is independent of the numerical values of finite random effects because changing \(\beta\) along \(d\) changes \(\eta_i\) by \(x_i^\mathsf{T}d\). Nevertheless, conditional probabilities and likelihood values do depend on \(b\). In a GLMM, marginalization and covariance estimation are additional parts of the analysis; the conditional definition should not be mistaken for a complete description of all their behavior.
The geometry is the same conditional Bernoulli geometry used for binary logistic regression in a GLM. The GLMM-specific caution is to keep the fixed-effect design \(X\), random-effect design \(Z\), and integration over \(b\) conceptually separate. Adding random effects does not redefine the fixed-effect margins, and the fact that a model has random effects is not a separation diagnostic.
A common mistake is to see a cluster with only successes and conclude that the fixed effects are separated, or to assume random intercepts automatically prevent separation. Both conclusions skip the relevant check. Separation requires a direction in the fixed-effect design satisfying the signed-margin conditions across all observations. A one-outcome cluster may contribute to a separating pattern, but it does not determine that pattern on its own.
Check Your Understanding
For each item, decide what the signed-margin condition implies. Solutions follow the questions.
- For two observations with \(x_i=(1)\), the outcomes are 1 and 0. Can any intercept direction have nonnegative signed margins for both?
- For \(x=0,Y=0\) and \(x=1,Y=1\), use an intercept and slope. Does \(d=(-1,2)^\mathsf{T}\) give complete or quasi-complete separation?
- For \(x=0\), observe one success and one failure; for \(x=1\), observe two successes. What type of separation is produced by \(d=(0,1)^\mathsf{T}\)?
- Why does a clinic with only successes not, by itself, prove fixed-effect separation in a random-intercept GLMM?
- In the quasi-complete example above, what is the limiting conditional likelihood as the slope increases without bound, and why is it not attained at a finite slope?
Solutions. (1) No. The success requires \(d\geq0\), while the failure requires \(d\leq0\); only \(d=0\) satisfies both, and it has no strict margin. (2) Complete separation: the \(x=0,Y=0\) margin is \((-1)(-1)=1\), and the \(x=1,Y=1\) margin is \(1(-1+2)=1\). (3) Quasi-complete separation: the \(x=0\) observations have zero margins, while each \(x=1,Y=1\) observation has margin 1. (4) Separation requires a single fixed-effect direction whose signed margins satisfy the condition across all observations; the other clusters and their outcomes also matter. (5) The limit is \(0.25\), the constant contribution from the two \(x=0\) observations. At any finite slope, the success probabilities at \(x=1\) are below 1, so the likelihood remains below that limit.