A Grouped Binary Example
Complete separation is easiest to recognize when the data are grouped. The groups make it possible to see directly whether every observed outcome falls on the same side of a predictor threshold. A random intercept does not necessarily prevent separation: a fixed predictor can separate outcomes across every cluster, even when each cluster has its own baseline propensity.
Consider hypothetical data from four clinics. Each clinic contributes five binary outcomes at each of two values of a binary predictor \(x\). The response is coded as success \(1\) or failure \(0\).
| Clinic | Predictor \(x\) | Trials | Successes |
|---|---|---|---|
| A | 0 | 5 | 0 |
| A | 1 | 5 | 5 |
| B | 0 | 5 | 0 |
| B | 1 | 5 | 5 |
| C | 0 | 5 | 0 |
| C | 1 | 5 | 5 |
| D | 0 | 5 | 0 |
| D | 1 | 5 | 5 |
Use a logistic random-intercept model. Let \(Y_{jx}\) be the number of successes among the five trials in clinic \(j\) at predictor value \(x\), and let \(b_j\) be that clinic’s random intercept.
Assume trials are conditionally independent given the random intercepts, the random intercepts are independent across clinics, and the logit link is correctly specified. For the argument about the unpenalized likelihood, take \(\tau>0\) as fixed. This isolates the fixed-effect separation issue; estimating a random-intercept variance does not make the displayed data pattern disappear.
Worked Example: Finding the Separating Predictor
For each clinic, \(x=0\) has five failures and no successes, while \(x=1\) has five successes and no failures. Thus the fixed-effect direction that increases \(\beta_1\) while placing the two predictor levels on opposite sides of the logit scale moves the fitted success probability down at \(x=0\) and up at \(x=1\). The predictor \(x\) completely separates the observed outcomes.
The pattern is not merely that \(x=1\) has a higher success proportion. It is that the observed proportions are exactly 0 and 1 at the two predictor levels, with no contrary outcomes. The same separation appears in all four clinics, so allowing a clinic-specific intercept does not supply an observed exception to the separating pattern.
The grouped rows represent \(4\times5=20\) trials at each value of \(x\): 20 failures at \(x=0\) and 20 successes at \(x=1\). These totals will be used in the likelihood calculations below.
Why the Unpenalized Estimate Has No Finite Slope
For a generalized linear mixed model, the likelihood for the observed counts is marginal: it integrates over the random intercepts. Conditional on the intercepts, the probability of the observed pattern in clinic \(j\) is \(\{1-\pi_{j0}\}^{5}\pi_{j1}^{5}\). Therefore the marginal likelihood is the product of these conditional probabilities integrated against the random-intercept distributions.
Here \(\operatorname{logit}^{-1}(u)=1/(1+e^{-u})\), and \(\phi(b;0,\tau^2)\) is the \(N(0,\tau^2)\) density. Consider the parameter sequence \(\beta_0=-t/2,\ \beta_1=t\), with \(t\) increasing. For any finite random intercept \(b\), the linear predictor at \(x=0\) is \(-t/2+b\), which tends to negative infinity; at \(x=1\) it is \(t/2+b\), which tends to positive infinity. Consequently, the conditional probability of every observed failure and success tends to one.
Each conditional likelihood contribution is bounded between zero and one. The dominated convergence theorem therefore permits taking this limit through the random-intercept integral: each clinic’s marginal likelihood contribution tends to one, and the full marginal likelihood tends to one. At any finite parameter values, the fitted probabilities are strictly between zero and one for finite \(b\); thus the observed pattern has probability strictly below one. The likelihood has supremum one but does not attain it at finite coefficients. There is no finite maximum-likelihood estimate along this separating direction.
Worked Example: Tracing the Conditional Likelihood
The marginal likelihood involves integration, but its conditional likelihood at \(b_j=0\) for all clinics provides a transparent numerical illustration. Along \(\beta_0=-t/2,\ \beta_1=t\), each of the 20 observed failures and 20 observed successes has conditional probability \(\operatorname{logit}^{-1}(t/2)\). Hence this conditional likelihood slice is
At \(t=6\), \(\operatorname{logit}^{-1}(3)=0.9526\), so \(L_{\mathrm{slice}}(6)=0.9526^{40}\approx0.1432\), using the unrounded probability for the power. At \(t=10\), \(\operatorname{logit}^{-1}(5)=0.9933\), giving \(L_{\mathrm{slice}}(10)\approx0.7645\). At \(t=20\), \(\operatorname{logit}^{-1}(10)=0.99995\), giving \(L_{\mathrm{slice}}(20)\approx0.9982\).
The likelihood rises as the slope increases and approaches one, rather than reaching its supremum at a finite slope. This slice is a conditional illustration, not the marginal GLMM likelihood; the integration argument above establishes the same limiting behavior for the marginal likelihood.
A Finite Regularized Alternative
A Gaussian prior on a separated fixed effect adds a penalty that opposes unlimited coefficient growth. The following calculation uses a joint posterior mode for the fixed effects and random intercepts, rather than claiming that it is the marginal maximum-likelihood estimate. Leave \(\beta_0\) unpenalized, assign \(\beta_1\sim N(0,1)\), and retain \(b_j\sim N(0,\tau^2)\). The prior standard deviation of one is on the log-odds scale.
Ignoring constants that do not depend on the parameters, the joint log posterior is the conditional grouped log likelihood plus the Gaussian log-prior penalties:
A symmetry argument is valid here only because it applies to this entire objective. Under the transformation \((\beta_0,\beta_1,b_j)\mapsto(-\beta_0-\beta_1,\beta_1,-b_j)\), the predictor at \(x=0\) becomes the negative of the former predictor at \(x=1\), and the predictor at \(x=1\) becomes the negative of the former predictor at \(x=0\). The failure and success contributions exchange because \(1-\operatorname{logit}^{-1}(u)=\operatorname{logit}^{-1}(-u)\). Both Gaussian penalties are unchanged. The objective is strictly concave for this dataset: the likelihood supplies curvature for the intercept and each clinic’s predictor, while the penalties supply curvature in the slope and random-intercept directions. Thus its unique mode must be fixed by this transformation, giving \(\beta_0=-\beta_1/2\) and \(b_j=0\) for every clinic.
With 20 trials in each predictor group, the objective on that set is \(40\log\{\operatorname{logit}^{-1}(t/2)\}-t^2/2\). Differentiating and setting the derivative to zero gives
Worked Example: Calculating the Regularized Mode
Solve \(t=20/(1+e^{t/2})\). At \(t=3.267\), the right-hand side is approximately \(20/(1+e^{1.6335})\approx20/6.122\approx3.267\). The solution is \(t\approx3.267\) (more precisely, about 3.2670). The derivative is positive at \(t=0\) and becomes negative for sufficiently large \(t\); strict concavity makes this root the unique maximum.
The resulting joint mode is approximately \(\hat\beta_0=-1.634\), \(\hat\beta_1=3.267\), and \(\hat b_j=0\) for all four clinics. At the mode, the conditional success probabilities are approximately \(\operatorname{logit}^{-1}(-1.6335)=0.1634\) at \(x=0\) and \(\operatorname{logit}^{-1}(1.6335)=0.8366\) at \(x=1\). The conditional odds ratio for a one-unit increase in \(x\) is \(\exp(3.267)\approx26.2\). It describes the fitted conditional odds contrast at the joint mode; it is not a marginal odds ratio averaged over clinics.
The finite slope is supplied by the regularization, not by an unpenalized maximum-likelihood estimate suddenly becoming available. Its numerical value depends on the prior scale: a wider Gaussian prior penalizes large slopes less and generally permits a larger fitted slope in these data.
Diagnostics, Alternatives, and Limits
A rising absolute slope accompanied by fitted probabilities approaching zero and one is a direct warning sign. Other signs include iterations that keep increasing a coefficient, standard errors that become very large, or warnings about nonconvergence and a nearly singular information matrix. In this example, however, a warning is not by itself proof of separation: the decisive evidence is the data pattern, together with the likelihood behavior along the separating direction. The earlier material on score and information diagnostics and on distinguishing separation from convergence failure provides complementary checks.
The main approaches answer different questions. Unpenalized maximum likelihood describes the likelihood supported by the observed data, but under complete separation it has no finite maximizer for the separating coefficient. A proper prior or ridge penalty yields finite estimates by adding information that is not contained in the likelihood alone. Firth correction changes the likelihood-based objective to reduce small-sample bias; it is not the same calculation as the Gaussian-prior joint mode above. Collecting more informative data can also help, especially if outcomes occur at both predictor levels, but changing the study design is not a post hoc numerical fix.
Regularization does not erase the underlying data limitation. The finite coefficient depends on the penalty or prior, so its sensitivity should be examined and reported. The joint mode calculated here also conditions on a particular random-effects representation and is not a substitute for a marginal likelihood estimate or a general prediction procedure. Finally, collapsing predictor levels simply to remove the pure cells can obscure the scientific contrast and does not create information about outcomes that were never observed.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the grouped data, how many trials are observed at \(x=0\), and how many successes are observed there?
- Why does the path \(\beta_0=-t/2,\ \beta_1=t\) make the observed outcomes increasingly likely for every finite random intercept?
- What is the conditional likelihood slice at \(t=4\), and approximately what value does it have?
- What role does the Gaussian prior on \(\beta_1\) play in the regularized mode?
- Does the regularized odds ratio of about 26.2 establish a causal effect?
Solutions. (1) There are \(4\times5=20\) trials at \(x=0\), with zero successes. (2) The linear predictors are \(-t/2+b_j\) at \(x=0\) and \(t/2+b_j\) at \(x=1\); for any finite \(b_j\), they tend respectively to negative and positive infinity, matching the failures and successes. (3) \(L_{\mathrm{slice}}(4)=\{\operatorname{logit}^{-1}(2)\}^{40}=0.8808^{40}\approx0.00623\). (4) Its penalty \(-\beta_1^2/2\) offsets the likelihood’s incentive to increase the separated slope without bound, producing a finite mode. (5) No. It is a conditional odds contrast from a regularized fit; neither the coefficient nor the separation pattern establishes causation.