Geometric Separation in the Design Matrix

From Outcomes to Geometry

For a binary GLMM, the fixed-effects part of the linear predictor is \(x_i^\mathsf{T}\beta\), where \(x_i^\mathsf{T}\) is row \(i\) of the fixed-effects design matrix \(X\), and \(\beta\) is the fixed-effects coefficient vector. With a logit link, a common conditional model is \(\operatorname{logit}\{P(Y_i=1\mid b)\}=x_i^\mathsf{T}\beta+z_i^\mathsf{T}b\). Here \(b\) denotes the random effects, \(z_i\) is the corresponding random-effects design row, and observations are conditionally independent given \(b\). The geometric conditions below concern \(X\) and the observed binary outcomes; they do not require the random-effects design to have the same columns.

The key idea is to ask whether some coefficient direction moves the linear predictors in the correct direction for every observed outcome. Encode that direction using a signed design row:

where \(y_i\in\{0,1\}\) and \(d\) is a proposed direction in fixed-effects coefficient space. If \(y_i=1\), then \(m_i>0\) means that \(d\) increases the fixed linear predictor for that observation. If \(y_i=0\), then \(m_i>0\) means that \(d\) decreases it. Either change moves the fitted probability toward the observed outcome.

Geometrically, the rows \(x_i\) are points in a \(p\)-dimensional design space. The equation \(x^\mathsf{T}d=0\) describes a hyperplane through the origin in that space. For a model with an intercept, its intercept coordinate allows this condition to represent an affine boundary in the predictor coordinates. Complete separation places every observed row strictly on the outcome-appropriate side. Quasi-complete separation permits some rows to lie on the boundary, while every row off the boundary is on the appropriate side.

Why Zero Projections Matter

A zero projection is not a minor exception to the definition. Along the coefficient path \(\beta(t)=\beta_0+td\), an observation with \(x_i^\mathsf{T}d=0\) has a linear predictor that does not change with \(t\). If the observation is on the boundary, its probability is not driven toward zero or one by this direction. This blocks complete separation, which requires every observed outcome to be driven in the correct direction. It does not block quasi-complete separation if the other observations have positive signed projections.

In particular, if the same design row occurs with both outcomes, strict separation is impossible: its projection cannot be strictly positive for both signs of \(2y_i-1\). But both observations can have zero projection, so quasi-complete separation may still hold. The earlier material on Overlap and Finite Fixed-Effect Estimates provides a useful conceptual contrast, with this distinction made explicit: overlap rules out strict separation at that row, not necessarily quasi-complete separation.

Likelihood Justification

The geometry explains why separation is associated with fixed-effect estimates that can move without reaching a finite maximum. For fixed \(b\), write the conditional success probability as \(p_i(t)=\operatorname{logit}^{-1}\{x_i^\mathsf{T}(\beta_0+td)+z_i^\mathsf{T}b\}\). The probability assigned to the observed outcome is \(q_i(t)=p_i(t)\) if \(y_i=1\), and \(q_i(t)=1-p_i(t)\) if \(y_i=0\). Its signed margin \(m_i\) is constant along the path.

To verify the expression, when \(y_i=1\), the derivative is \(x_i^\mathsf{T}d\{1-p_i(t)\}=m_i\{1-q_i(t)\}\). When \(y_i=0\), it is \(-x_i^\mathsf{T}d\,p_i(t)=m_i\{1-q_i(t)\}\), because then \(m_i=-x_i^\mathsf{T}d\) and \(1-q_i(t)=p_i(t)\). Therefore a positive margin makes the observed-outcome probability increase; a zero margin leaves it unchanged.

For the conditionally independent Bernoulli observations, the conditional likelihood is the product of the \(q_i(t)\). Its log derivative is the sum of the nonnegative terms above, and is strictly positive whenever at least one margin is positive. Under complete separation, every \(q_i(t)\) tends to one. Under quasi-complete separation, positive-margin contributions tend to one, while zero-margin contributions remain unchanged along this direction.

The same monotonicity applies to the marginal GLMM likelihood when the random-effects distribution is held fixed: for each \(b\), the conditional likelihood increases along the direction; integrating over \(b\) preserves that ordering. The conditional likelihood is bounded by one, so its marginal integral is also bounded by one. This gives a likelihood justification for the geometric diagnostic without asserting that every GLMM’s marginal likelihood has the same numerical limit as the fixed-effects illustration above.

Checking the Geometry in Practice

The definitions provide an exact criterion for a fixed design matrix and observed outcomes. In practice, first construct or inspect the design matrix as it was actually fitted: include its intercept, factor-contrast columns, interactions, and any transformations. Then evaluate whether a direction can satisfy the signed inequalities. A plot of one predictor can reveal an obvious pattern, but it cannot generally settle the question when several columns jointly define the separating direction.

A few checks help prevent misleading conclusions:

• Check the sign convention. For \(y_i=0\), the desired direction is a decreasing predictor, which is why its signed row is \(-x_i\).
• Check boundary rows separately. A zero margin is allowed for quasi-complete separation, but not for complete separation. It is not evidence of a strictly separated observation.
• Check that at least one margin is positive. The zero vector always gives zero margins and must not be mistaken for quasi-complete separation.
• Use the fitted design coding. Releveling a factor or changing contrasts changes the coordinates of \(X\), although a nonsingular reparameterization preserves whether a separating direction exists.

A convergence warning alone does not establish separation: numerical scaling, weak identification, or other optimization difficulties can also cause trouble. Conversely, a successful optimizer termination does not disprove a geometric separating direction. The condition is about the observed design and labels; optimization diagnostics describe the behavior of a particular fitting procedure.

Interpretation and Limitations

The geometric criterion is a statement about fixed-effects directions in the specified binary model. It is useful because it makes the distinction between strict and boundary separation precise and can be checked independently of an optimizer’s path. Its conclusion is still conditional on the model specification and observed data. A different set of columns, a changed outcome coding, or a different sample can change the geometry.

The criterion should not be confused with a claim that every coefficient is individually separated, or that every coefficient estimate must diverge at the same rate. It identifies a direction along which the likelihood improves; it does not by itself describe all parameter directions or quantify uncertainty. Nor should the simple fixed-effects likelihood numbers in the boundary example be transferred to a GLMM with random effects: marginal likelihoods depend on the random-effects distribution and design.

A common mistake is to reason, “Both outcomes occur at \(x=0\), so there is overlap; therefore there can be no separation of any kind.” The correction is that identical rows with opposite outcomes rule out a direction strictly positive for every observation. They can both have zero projection, leaving the remaining rows to satisfy the quasi-complete inequalities. Another mistake is to count a direction with all zero projections as quasi-complete separation; the definition requires at least one strict margin.