Why Separated Coefficients Diverge

Why an Increasing Coefficient Can Have a Finite Likelihood Limit

In a binary model, a coefficient can grow without bound while the likelihood keeps improving by smaller and smaller amounts. The likelihood does not diverge to infinity: it approaches a finite ceiling. The important consequence is that the ceiling may not be attained at any finite coefficient, so ordinary maximum likelihood has no finite estimate in that direction.

The argument begins with the binary conditional likelihood used inside a GLMM. It then extends to the marginal likelihood by integrating over the random effects. The fixed-effect direction is the source of separation; random effects change the integration, but do not necessarily remove that direction.

Definition: For binary observations $y_i\in\{0,1\}$, let $x_i$ be the fixed-effect design vector, $z_i$ the random-effect design vector, and $b$ the vector of random effects. Under the logit link, $$ P(Y_i=1\mid b)=\operatorname{logit}^{-1}(x_i^\mathsf{T}\beta+z_i^\mathsf{T}b), \qquad \operatorname{logit}^{-1}(u)=\frac{1}{1+\exp(-u)}. $$ Define $s_i=2y_i-1$, so $s_i=1$ for a success and $s_i=-1$ for a failure. A vector $d$ is a separating fixed-effect direction if $s_i x_i^\mathsf{T}d\geq 0$ for every observation, with strict inequality for at least one. It is a complete separating direction if the inequality is strict for every observation. If some margins are zero and some are positive, it is a quasi-complete separating direction.

The margin $s_i x_i^\mathsf{T}d$ tells whether movement along $d$ moves observation $i$'s linear predictor in the direction of its observed outcome. The definition concerns the fixed-effect design: the random-effect contribution $z_i^\mathsf{T}b$ is held in the linear predictor, not incorporated into the separation condition.

The Likelihood Along a Separating Direction

Hold the random effects $b$ and all other parameters fixed, and move the fixed effects along the path $\beta(t)=\beta_0+td$, where $t\geq 0$. Write $\eta_i(0)=x_i^\mathsf{T}\beta_0+z_i^\mathsf{T}b$. The observed binary outcome has conditional probability $\operatorname{logit}^{-1}(s_i\eta_i)$. Thus its log-likelihood contribution along the path is

\ell_i(t\mid b) = -\log\left\{1+\exp\left[-s_i\eta_i(0)-t\,s_i x_i^\mathsf{T}d\right]\right\}.

This expression gives both the monotonicity and the limit. If the margin $m_i=s_i x_i^\mathsf{T}d$ is positive, increasing $t$ makes the exponent's argument $-s_i\eta_i(0)-tm_i$ smaller, so the contribution increases toward zero. If $m_i=0$, the contribution stays constant. No contribution exceeds zero because it is the logarithm of a probability.

Separate the observations by margin.
For $m_i>0$, the fitted probability of the observed outcome tends to one. For $m_i=0$, the linear predictor for that observation does not change along the path.

Take the conditional likelihood limit.
Strict-margin contributions tend to zero on the log scale; zero-margin contributions retain their original finite log-likelihood values.

Integrate over random effects.
The conditional likelihood is bounded between zero and one. With a fixed random-effect distribution, dominated convergence permits taking its pointwise limit inside the integral.

For the whole conditional log-likelihood, the result is

\ell(t\mid b) = -\sum_i\log\{1+\exp[-s_i\eta_i(0)-tm_i]\} \ \longrightarrow\ -\sum_{i:m_i=0}\log\{1+\exp[-s_i\eta_i(0)]\}.

Under complete separation the sum on the right is empty, so the conditional log-likelihood tends to zero. Under quasi-complete separation it tends to the finite log-likelihood contributed by the observations with zero margin. In either case, if at least one margin is positive, the log-likelihood strictly increases for finite $t$ and does not attain its limit at any finite $t$.

From Conditional to Marginal GLMM Likelihood

A GLMM's marginal likelihood for the observed outcome vector integrates the conditional likelihood over the random-effect distribution. For example, with random effects having density $f(b;\theta)$, and with the random-effect parameters $\theta$ held fixed, it is

L(t)=\int \left\{\prod_i P(Y_i=y_i\mid b,\beta_0+td)\right\}f(b;\theta)\,db.

Assume the binary outcomes are conditionally independent given $b$, the link is logistic, the random effects are finite with probability one, and the separating margins satisfy the definition above. The integrand is between zero and one. For each finite $b$, it converges to the product of the zero-margin probabilities: observations with positive margins contribute one in the limit. The dominated convergence theorem therefore gives

L(t)\ \longrightarrow\ \int \left\{\prod_{i:m_i=0}P(Y_i=y_i\mid b,\beta_0)\right\}f(b;\theta)\,db.

This limit is finite. It is also positive for finite linear predictors under the logistic model: each retained Bernoulli probability is positive, and the random-effect density integrates to one. For complete separation, the product over zero-margin observations is empty and the limit is one. Since a likelihood for a particular outcome vector cannot exceed one, this is the conditional-path ceiling for the marginal likelihood as well.

At every finite $t$, each positive-margin observation's conditional probability of its observed outcome strictly increases for every finite $b$. The conditional likelihood product, and hence its integral, strictly increases. Consequently, the limiting value is not attained at a finite $t$. Since this argument can be applied starting at any finite $\beta_0$, a separating direction rules out a finite global maximum of the ordinary marginal likelihood when the other parameters are held at admissible fixed values. It does not say that every GLMM parameter must diverge; the nonexistence concerns a finite maximizer in the separated fixed-effect direction.

Worked Example: Complete Separation in a Logistic Likelihood

Suppose four hypothetical observations have $x=(-2,-1,1,2)$ and $y=(0,0,1,1)$. Consider a logistic model with intercept fixed at zero and slope $\beta_1=t$. The direction $d=1$ separates all four observations: for the failures, $x_i<0$, and for the successes, $x_i>0$.

For an observation with $x_i$ in this pattern, the probability of the recorded outcome is $\operatorname{logit}^{-1}(t|x_i|)$. Thus $$ \ell(t)=-2\log(1+\exp(-t))-2\log(1+\exp(-2t)). $$ At $t=1$, this is $\ell(1)=-2\log(1.3679)-2\log(1.1353)\approx-0.8804$. At $t=2$, it is $\ell(2)=-2\log(1.1353)-2\log(1.0183)\approx-0.2901$.

The log-likelihood has increased, but neither value is zero. As $t$ grows, both terms tend to zero, so $\ell(t)\to 0$. The likelihood tends to $\exp(0)=1$, but no finite $t$ gives probability one to all four observed outcomes. The coefficient therefore has no finite maximum-likelihood estimate in this direction.

Worked Example: Quasi-Complete Separation Leaves a Finite Remainder

Consider three hypothetical observations with $x=(-1,0,1)$, $y=(0,0,1)$, and intercept fixed at zero. Increasing the slope $t$ moves the first and third observations toward their recorded outcomes. The middle observation has zero margin: its linear predictor remains zero.

The log-likelihood is $$ \ell(t)=-2\log(1+\exp(-t))-\log(2). $$ At $t=1$, $\ell(1)=-2\log(1.3679)-0.6931\approx-1.3197$. At $t=2$, $\ell(2)=-2\log(1.1353)-0.6931\approx-0.9470$. As $t\to\infty$, the first two contributions tend to zero while the middle contribution remains $-\log(2)\approx-0.6931$.

The limiting likelihood is $1/2$, not one, because the middle observation continues to have probability $1/2$ of its recorded outcome. The slope still has no finite maximum: every finite increase in $t$ improves the two strict-margin contributions, while leaving the zero-margin contribution unchanged.

Worked Example: A Random Intercept Does Not Remove the Likelihood Limit

Suppose there are two hypothetical clusters, each with one binary observation. Let $y_1=1,x_1=1$ and $y_2=0,x_2=-1$. Use a random-intercept model with $b_j\sim N(0,0.25)$, independently across clusters, and fix the intercept at zero. The slope $t$ separates both outcomes.

Conditional on $b_1,b_2$, the probability of both recorded outcomes is $$ P(Y_1=1,Y_2=0\mid b_1,b_2) =\operatorname{logit}^{-1}(t+b_1)\operatorname{logit}^{-1}(t-b_2). $$ At $b_1=b_2=0$, this conditional probability is $\operatorname{logit}^{-1}(t)^2$. At $t=1$, it is $0.7311^2\approx0.5345$; at $t=2$, it is $0.8808^2\approx0.7758$. These are conditional values at zero random effects, not marginal likelihoods.

The marginal likelihood is the expectation of the displayed product over the two normal random intercepts. For every finite pair $(b_1,b_2)$, the product tends to one as $t\to\infty$, and it is bounded by one. Dominated convergence therefore gives a marginal likelihood limit of one. At any finite $t$, the conditional product is less than one for finite $b_1,b_2$, so its expectation is less than one as well. The random intercept changes the finite-$t$ likelihood, but does not create a finite maximizing slope.

What the Limit Does—and Does Not—Establish

The distinction between a finite supremum and a finite estimate is central. The log-likelihood is bounded above, and along the separating path it converges to a finite value. But a maximum-likelihood estimate is a parameter value attaining the largest likelihood. A sequence of increasingly large coefficients whose likelihood approaches a ceiling is not such an estimate.

This argument also clarifies why separation is different from a routine numerical convergence problem. An optimizer may stop after taking large steps, report a large coefficient, or encounter a numerical warning. Those outcomes can be clues, but the mathematical issue is that the likelihood keeps improving along an unbounded path. Conversely, an optimizer warning alone does not prove separation; other fitting problems can also prevent convergence.

For comparison, with overlap there is no fixed-effect direction that moves every strict-margin observation toward its recorded outcome while moving none against it. This removes the specific monotone path derived here, though it does not guarantee a well-behaved fit for every GLMM. Conditional and marginal likelihoods should also not be conflated: the conditional argument fixes $b$, whereas the marginal argument integrates over its distribution. Under the assumptions stated above, both exhibit the same nonattainment along a separating fixed-effect direction.

Assumptions, Diagnostics, and Limits

The derivation relies on a logistic binary response model, conditional independence given the random effects, a fixed random-effect distribution along the coefficient path, and a separating direction in the fixed-effect design. For the marginal argument, the random effects must be finite almost surely and the conditional likelihood must be bounded by an integrable function; the bound of one supplies this condition. If the direction does not satisfy the margin inequalities, some observations can move against their outcomes, and the monotonicity argument no longer applies.

In practice, large fixed-effect estimates together with steadily improving likelihoods as their magnitudes grow are consistent with separation. To establish the explanation, however, inspect whether a direction actually satisfies the margin condition; a large coefficient by itself is not proof. Numerical likelihood values can also be affected by approximation, stopping rules, or other optimization difficulties, so the exact likelihood argument should not be inferred from a warning alone.

The result is not a remedy for separation and does not guarantee that every other parameter is estimable. It concerns ordinary likelihood with the specified fixed-effect path and fixed nuisance parameters. Penalized likelihood or Bayesian estimation changes the objective or adds information, so its finite estimates are not ordinary maximum-likelihood estimates obtained by attaining this likelihood limit.

Common mistake: “The log-likelihood approaches a finite number, so the coefficient must have a finite maximum-likelihood estimate.” A finite limit is not necessarily attained. Under separation, the likelihood can improve at every finite coefficient and approach its ceiling only as the coefficient tends to infinity.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

In a complete-separation path, what is the limiting conditional log-likelihood contribution from each observation?
Why is the quasi-complete-separation limit not necessarily zero on the log scale?
What allows the pointwise limit of the conditional likelihood to be integrated over random effects?
In a logistic GLMM, a coefficient is very large and the optimizer reports a warning. Does this alone establish separation?

Solutions. (1) Each observed-outcome probability tends to one, so each log contribution tends to $\log(1)=0$. (2) Zero-margin observations do not change along the path and retain their finite log-likelihood contributions. (3) The conditional likelihood is bounded between zero and one, and the random-effect distribution is a probability distribution, so dominated convergence applies. (4) No. The warning and large estimate may motivate investigation, but separation requires a direction satisfying the margin inequalities; numerical difficulties can have other causes.