Why Boundary Observations Matter
In a binary-outcome model, a predictor pattern may sort the observed outcomes almost perfectly while leaving some observations exactly on the dividing boundary. This is not the same as complete separation: the boundary observations cannot be assigned strictly to the correct side. Yet the distinction matters, because the likelihood may still favor increasingly large fixed-effect coefficients rather than a finite estimate.
The geometric definition below is stated for the fixed-effects design. In a logistic GLMM, the conditional linear predictor for observation \(i\) in cluster \(j\) is \(x_{ij}'\beta+z_{ij}'b_j\), where \(x_{ij}\) is its fixed-effects design vector, \(\beta\) is the fixed-effect coefficient vector, \(z_{ij}\) is its random-effects design vector, and \(b_j\) is the cluster-specific random effect. A pattern in the fixed-effects design is an important warning about estimation, but the fitted model and its likelihood also involve the random-effects distribution.
The sign convention makes the criterion easy to read. For an outcome \(y_i=1\), \(s_i=1\), so the weak-separation condition requires \(x_i'c\geq0\). For \(y_i=0\), \(s_i=-1\), so it requires \(x_i'c\leq0\). Equality means the observation's linear score in direction \(c\) is zero. If both outcomes occur at the same design point, they cannot both be strictly on the correct side; they may both lie on the boundary.
A Worked Boundary Pattern
Worked Example: A One-Predictor Quasi-Complete Separation
Consider four hypothetical observations, modelled with a logistic regression containing an intercept and predictor \(x\).
| Observation | \(x\) | \(y\) |
|---|---|---|
| 1 | -1 | 0 |
| 2 | 0 | 0 |
| 3 | 0 | 1 |
| 4 | 1 | 1 |
Take \(c=(0,1)'\), so \(x_i'c=x_i\). At \(x=-1,y=0\), \(s_i x_i'c=(-1)(-1)=1\); at \(x=1,y=1\), it is \(1(1)=1\). Both observations at \(x=0\) have score zero. The inequalities are therefore weak for all four rows, strict for two, and equal for two. There is no strict separator because the same \(x=0\) design point has both outcomes. This is quasi-complete separation, not complete separation.
The likelihood shows why a finite slope can be problematic. Let \(\sigma(u)=1/(1+\exp(-u))\), and set the intercept to zero and slope to \(t>0\). The two boundary observations each have probability \(1/2\) for their observed outcome. The two outer observations each have probability \(\sigma(t)\). Hence
At \(t=2\), \(\sigma(2)\approx0.880797\), so \(L(0,2)=0.25(0.880797)^2\approx0.19395\). As \(t\) increases, \(\sigma(t)\) increases toward 1, and the likelihood approaches \(0.25\). No finite \(t\) attains that limit along this path. Moreover, for any intercept \(a\) and slope \(t\), the likelihood is
Here \(\sigma(a)\{1-\sigma(a)\}\leq1/4\), while each remaining factor is less than 1 for finite arguments. The path \(a=0,\ t\to\infty\) approaches the upper bound, so the likelihood has supremum \(1/4\) but no finite parameter pair attaining it. This is the fixed-effects logistic illustration of why quasi-complete separation can prevent a finite maximum-likelihood estimate.
Checking the Pattern in a Categorical Predictor
Worked Example: Mixed Outcomes at One Factor Level
Suppose a hypothetical binary GLMM includes a three-level categorical fixed effect, coded with an intercept and two indicator variables. At level A there are two failures, at level B there is one failure and one success, and at level C there are two successes. Consider the fitted-logit direction that assigns scores \(-t,0,t\) to levels A, B, and C, respectively.
At level A, the outcomes are all 0 and the score is negative; at level C, the outcomes are all 1 and the score is positive. Both outcomes at level B have score zero. Thus every row satisfies the weak sign condition, some rows satisfy it strictly, and the B observations are on the boundary. No strict separator can exist because the same level B has both outcomes. The pattern is quasi-complete separation.
For the conditional logistic likelihood with random effects held fixed at zero, this direction gives probability \(\sigma(t)\) to each of the two observed failures at A and each of the two observed successes at C. The two B observations each have probability \(1/2\). At \(t=1\), the likelihood contribution is
As \(t\to\infty\), this contribution approaches \(1/4\), without attaining it at a finite \(t\). The calculation illustrates the boundary pattern; in a GLMM the full likelihood integrates over the random-effects distribution, so this conditional calculation alone is not a full mixed-model likelihood evaluation.
Why the Likelihood Can Keep Improving
For a Bernoulli observation with logistic probability \(p_i=\sigma(\eta_i)\), its likelihood contribution is \(p_i\) if \(y_i=1\), and \(1-p_i\) if \(y_i=0\). Along a coefficient path \(\beta(t)=\beta_0+tc\), the linear predictor changes by \(t x_i'c\). If \(s_i x_i'c>0\), increasing \(t\) moves the predicted probability toward the observed outcome: toward 1 for a success and toward 0 for a failure. The observation's likelihood contribution therefore increases. If \(s_i x_i'c=0\), its linear predictor and contribution do not change along this path.
With quasi-complete separation, at least one contribution improves and none worsens along the separating direction. In an ordinary logistic model this gives a nondecreasing likelihood along the path, with improvement when a strict row is present. In a logistic GLMM, the same conditional sign argument applies for fixed random effects and fixed random-effects parameters. Integrating the conditional likelihood over a random-effects distribution preserves the nondecreasing comparison along that fixed-effect direction. This explains why a finite fixed-effect maximum may fail to exist in such a pattern; it does not imply that every unstable GLMM fit has separation, or replace examination of the actual fitted model.
Distinguishing Quasi-Complete Separation from Its Alternatives
Worked Example: When the Boundary Outcome Changes
Return to a predictor with three hypothetical observations: \(x=-1,y=0\), \(x=0,y=1\), and \(x=1,y=1\). Unlike the previous examples, there is no failure at \(x=0\). Use an intercept and slope direction \(c=(0.5,1)'\), which gives scores \(-0.5,0.5,1.5\).
For the failure at \(x=-1\), \(s_i x_i'c=(-1)(-0.5)=0.5>0\). For both successes, the scores are positive, so \(s_i x_i'c\) is also positive. Every observation is strictly on the correct side. This is complete separation. The change from a mixed-outcome boundary to a single success at \(x=0\) changes the classification because the intercept can move the boundary away from that observation.
By contrast, overlap means no weak separating direction of this form can place the observations on their correct sides. It is not enough that the dataset contains some correctly ordered observations: the sign inequalities must hold simultaneously for all rows. In practice, inspect the joint fixed-effects patterns rather than judging separation from one predictor column in isolation.
A useful comparison is:
| Pattern | Sign condition for a direction | Implication for the boundary |
|---|---|---|
| Complete separation | \(s_i x_i'c>0\) for every row | No observation lies on the separating boundary |
| Quasi-complete separation | \(s_i x_i'c\geq0\) for every row, with both strict and equal cases | At least one observation lies on the boundary |
| Overlap | No direction satisfies the weak sign inequalities for every row with at least one strict inequality | The observed patterns cannot be separated by such a boundary |
Diagnostics, Limitations, and Common Pitfalls
Start with the observed outcome patterns across the fixed-effect design: combinations of factor levels, indicator variables, and relevant continuous-predictor ranges. A level containing only successes or only failures is a warning, but it is not by itself proof of quasi-complete separation. The rest of the design must also admit a common weak separating direction. Conversely, mixed outcomes at the same design point often expose boundary observations and rule out strict separation in the corresponding direction.
In a GLMM, use this check alongside the estimation evidence: very large fixed-effect estimates or standard errors, fitted probabilities close to zero or one, a flat likelihood in a coefficient direction, and sensitivity to small data changes may all be warning signs. These symptoms are not unique to separation; poor scaling, sparse factor levels, an over-complex random-effects structure, or optimization failure can also cause instability. Examine the design and the optimization diagnostics before deciding which explanation is plausible.
- Pitfall: treating any pure factor level as proof of quasi-separation. A level with only successes may appear separable, but other predictor combinations can prevent a single direction from satisfying the required inequalities. Check the joint design.
- Pitfall: calling boundary cases complete separation. If even one row has \(x_i'c=0\), that direction is not a strict separator. Check whether another direction is strict for every row before classifying the pattern.
- Pitfall: treating a large estimate as a separation test. Large estimates may arise for other reasons. Verify the outcome-design geometry and distinguish it from an optimizer that simply failed to converge.
- Pitfall: assuming random effects automatically fix the problem. Random effects change the model likelihood, but they do not erase a separating pattern in the observed fixed-effects design. Assess the full GLMM rather than relying on that assumption.
The sign check is a conceptual and practical diagnostic, not a complete analysis of every GLMM estimation issue. Random-effects structure, dependence within clusters, and the marginal likelihood matter for inference. When a quasi-separated pattern is suspected, report the pattern and estimation instability clearly; do not interpret an extreme finite coefficient as a stable, precisely estimated effect merely because software returned a number.
Check Your Understanding
Answer each question, then compare with the short solutions.
- For a success with score \(x_i'c=0\), does the row satisfy strict or weak separation?
- A predictor level contains both successes and failures, and all other levels lie on the appropriate sides of a candidate boundary. What does the mixed level imply?
- In the one-predictor example, calculate \(L(1)\) along the path with intercept zero. Use \(\sigma(1)\approx0.731059\).
- Why does a coefficient estimate that is merely very large not prove quasi-complete separation?
Solutions. (1) It satisfies weak separation because \(s_i x_i'c=0\), but not strict separation. (2) Both outcome types at that level share the same design point, so they cannot both be strictly on the correct side; if the other rows satisfy the weak inequalities and at least one is strict, the pattern is quasi-complete separation. (3) \(L(1)=0.25(0.731059)^2\approx0.13361\). (4) Large estimates can also result from sparse data, model complexity, scaling, or failed optimization; separation requires a direction satisfying the sign inequalities for the observed design.