Overlap and Finite Fixed-Effect Estimates

Why overlap matters

A binary-response model estimates how outcome probabilities vary with predictors. If predictor patterns for the two outcome classes are separated, the likelihood can keep improving as one or more coefficients grow without bound; a finite maximum-likelihood estimate may then fail to exist. Overlap changes that picture: seeing both outcomes at the same predictor pattern prevents a model from assigning that pattern probability arbitrarily close to one for either class without also worsening the fit to the other.

The important qualification is that overlap is evidence, not an automatic guarantee. A shared predictor pattern forces a proposed separating linear predictor to tie at that pattern, but other patterns might still be strictly ordered in a way that permits quasi-complete separation. The full arrangement of outcome patterns matters.

Definition: In a binary logistic model with fixed-effect design vector \(x_i\), the linear predictor is \(\eta_i=x_i^\mathsf{T}\beta\), and \(P(Y_i=1\mid x_i)=p_i=\operatorname{logit}^{-1}(\eta_i)\). A predictor pattern is a particular value of \(x_i\). Overlap at a pattern means that at least one observation with that pattern has \(Y_i=1\) and at least one has \(Y_i=0\). A finite fixed-effect maximum is a finite vector \(\hat{\beta}\) at which the likelihood attains its maximum.

This tutorial first establishes the overlap argument for the ordinary fixed-effect logistic likelihood. In a GLMM, the conditional model also has a linear predictor, typically \(x_{ij}^\mathsf{T}\beta+z_{ij}^\mathsf{T}b_j\), and the marginal likelihood integrates over the random effects. Consequently, overlap in the fixed-effect predictors is a useful structural check, but by itself does not prove that the integrated GLMM likelihood has a finite maximum. That distinction matters when interpreting an apparently reassuring pattern in clustered data.

What a shared pattern does to the likelihood

For a Bernoulli observation, the likelihood contribution is \(p_i^{y_i}(1-p_i)^{1-y_i}\). Suppose two observations have the same design vector \(x\), with one outcome equal to one and the other equal to zero. Their joint contribution is

\operatorname{logit}^{-1}(x^\mathsf{T}\beta) \left[1-\operatorname{logit}^{-1}(x^\mathsf{T}\beta)\right].

Writing \(p=\operatorname{logit}^{-1}(x^\mathsf{T}\beta)\), this is \(p(1-p)\), which is at most \(1/4\), with equality at \(p=1/2\). If \(x^\mathsf{T}\beta\) tends to \(+\infty\) or \(-\infty\), the contribution tends to zero. Thus, a direction that drives the shared pattern toward a deterministic prediction damages the fit to at least one of its observed outcomes.

This argument rules out a direction that strictly classifies every observation at that shared pattern. It does not rule out quasi-complete separation: a separating direction may assign the shared pattern a tied prediction while strictly classifying other patterns. Finite estimation requires that the likelihood not be able to increase toward a limiting boundary in any coefficient direction, as well as that the fixed-effect parameters be identifiable.

Key result: For a full-rank ordinary logistic regression with a finite data set, a finite maximum-likelihood estimate exists when there is no complete or quasi-complete separation. Overlap at predictor patterns can help prevent separation, but overlap alone is not a general substitute for checking the entire set of patterns and the model's identifiability.

Worked examples

Worked Example: Balanced Outcomes at Every Pattern

Consider a hypothetical binary dataset with an intercept and one predictor \(x\). At each of \(x=-1,0,1\), there are two observations with \(Y=0\) and two with \(Y=1\). Use the logistic model \(\eta=\beta_0+\beta_1x\). Show that \(\beta_0=\beta_1=0\) is a finite maximum.

At \(\beta_0=\beta_1=0\), each fitted probability is \(p_i=1/2\). For independent Bernoulli observations, the log-likelihood score is

U(\beta)=\sum_i x_i(y_i-p_i),

where \(x_i\) in this score denotes the full design vector \((1,x_i)^\mathsf{T}\). The intercept score is the sum of \(y_i-p_i\). There are six zeros and six ones, so it is \(6(-0.5)+6(0.5)=0\).

For the slope score, include the zero and one outcomes at both nonzero predictor values. The full contribution is

$$ 2(-1)(-0.5)+2(-1)(0.5)+2(1)(-0.5)+2(1)(0.5)=1-1-1+1=0. $$

Observations at \(x=0\) contribute zero to the slope score because their slope multiplier is zero. Both score components therefore vanish. The log-likelihood is concave. Its negative Hessian at this point is \(X^\mathsf{T}WX\), where \(W\) has diagonal entries \(p_i(1-p_i)=1/4\). The design columns \(1\) and \(x\) are linearly independent because the dataset contains distinct \(x\) values, so \(X^\mathsf{T}WX\) is positive definite. The log-likelihood is strictly concave and the stationary point is its unique maximum. The finite estimate is \((\hat{\beta}_0,\hat{\beta}_1)=(0,0)\).

Worked Example: A Nonzero but Finite Slope

Now suppose a hypothetical dataset has the following counts: at \(x=-1\), one outcome is 1 and three are 0; at \(x=0\), two are 1 and two are 0; at \(x=1\), three are 1 and one is 0. Both outcome classes occur at every predictor pattern. Find and verify the logistic maximum.

The proposed fitted probabilities are \(0.25\), \(0.50\), and \(0.75\), respectively. These are generated by \(\beta_0=0\) and \(\beta_1=\log(3)\), since

\operatorname{logit}(0.25)=-\log(3),\qquad \operatorname{logit}(0.50)=0,\qquad \operatorname{logit}(0.75)=\log(3).

Thus the fitted linear predictors at \(x=-1,0,1\) are \(-\log(3),0,\log(3)\), as required. Verify the score using each outcome class. The intercept-score contributions, grouped by predictor pattern, are

\begin{aligned} x=-1 &: 1(1-0.25)+3(0-0.25)=0.75-0.75=0,\\ x=0 &: 2(1-0.50)+2(0-0.50)=1-1=0,\\ x=1 &: 3(1-0.75)+1(0-0.75)=0.75-0.75=0. \end{aligned}

For the slope score, multiply each observation's residual by its \(x\) value. At \(x=-1\) the contribution is \((-1)[1(0.75)+3(-0.25)]=0\); at \(x=0\) it is zero; and at \(x=1\) it is \(1[3(0.25)+1(-0.75)]=0\). Both score components are zero.

The design has full rank. Each fitted probability is strictly between zero and one, so every Bernoulli variance weight \(p_i(1-p_i)\) is positive. The negative Hessian \(X^\mathsf{T}WX\) is therefore positive definite, making the log-likelihood strictly concave. The score-zero point is the unique finite maximum: \(\hat{\beta}_0=0\), \(\hat{\beta}_1=\log(3)\approx1.099\). A one-unit increase in \(x\) multiplies the fitted odds by \(e^{\hat{\beta}_1}=3\); this is an odds comparison under the specified model, not a causal effect.

Worked Example: Overlap That Still Allows Quasi-Complete Separation

Consider a hypothetical dataset with two observations at each \(x=-1,0,1\). At \(x=-1\), both outcomes are 0; at \(x=0\), one outcome is 0 and one is 1; at \(x=1\), both outcomes are 1. The pattern \(x=0\) overlaps, but the other patterns are perfectly ordered.

Set \(\beta_0=0\) and let \(\beta_1=t>0\). As \(t\) increases, the fitted probability at \(x=-1\) tends to zero, which fits both zeros there, and the fitted probability at \(x=1\) tends to one, which fits both ones there. At \(x=0\), the fitted probability remains \(1/2\), so the mixed outcomes there retain a nonzero likelihood contribution of \(1/4\). The endpoint observations' contributions approach one each, and the likelihood increases toward a boundary as \(t\) grows. No finite slope attains that limiting fit. This is quasi-complete separation: the mixed pattern is tied while the other patterns are strictly ordered. It demonstrates why a check for overlap must not stop after finding a single shared pattern.

Justifying a finite maximum

The examples use two standard likelihood facts. For independent Bernoulli observations with a logistic mean, the log-likelihood is

\ell(\beta)=\sum_i\left[y_i x_i^\mathsf{T}\beta-\log\{1+\exp(x_i^\mathsf{T}\beta)\}\right].

Differentiating gives \(U(\beta)=\sum_i x_i(y_i-p_i)\). Differentiating again gives the Hessian

H(\beta)=-\sum_i p_i(1-p_i)x_ix_i^\mathsf{T}=-X^\mathsf{T}WX.

Because \(p_i(1-p_i)>0\) at every finite linear predictor, \(X^\mathsf{T}WX\) is positive definite whenever the design matrix \(X\) has full column rank. The log-likelihood is then strictly concave: it has at most one stationary point, and any stationary point is its unique global maximum. In the first two examples, the displayed fitted probabilities produce a zero score, and the rank and positive-weight conditions establish that the maximum is finite and unique.

This reasoning also clarifies the two checks involved. Identifiability asks whether different coefficient vectors can produce distinct linear predictors in the specified design; full column rank is the usual condition in the examples. Existence asks whether the likelihood attains its maximum at finite coefficients rather than improving toward a boundary. A full-rank design does not by itself guarantee existence: the third example has an identifiable intercept-and-slope design but no finite maximum because of quasi-complete separation.

Using overlap in GLMM work

In a GLMM, examine the fixed-effect predictor patterns in the context of the actual model matrix, including its contrasts and interactions. Ask whether both outcome classes occur at comparable fixed-effect patterns, and whether any coefficient direction could still order the remaining patterns while tying the shared ones. This is a useful first screen for separation-related instability, not a replacement for model-specific diagnostics.

A practical workflow is to tabulate outcomes against important categorical predictors and inspect the observed ranges for continuous predictors. Then check the fitted model's convergence information and whether coefficient estimates, standard errors, or profile likelihood behavior suggest instability. These signals are not interchangeable: a failed optimizer may reflect numerical difficulty, while separation concerns whether a finite maximum exists. Conversely, a convergence message alone does not establish that the fitted fixed-effect estimates are well supported.

Main alternatives depend on the problem. If an apparent lack of overlap is due to an unnecessarily elaborate fixed-effect specification, a scientifically justified simpler model may be appropriate. Penalized likelihood or proper priors can yield finite regularized estimates when ordinary maximum likelihood is problematic, but those estimates target a different procedure and should be reported as such. Collecting additional observations in the poorly represented outcome-pattern combinations may improve support when feasible. None of these choices should be made solely to produce finite-looking coefficients.

Common pitfalls and limitations

“I found one shared pattern, so separation is impossible.” The third example refutes this: the shared pattern can be tied while other patterns are strictly separated.
“Both outcomes occur somewhere in the dataset, so there is overlap.” Overlap is about predictor patterns, not just the overall outcome totals. If the two classes occupy distinct patterns, their totals do not prevent separation.
“A finite estimate in a fixed-effect logistic calculation proves a finite GLMM estimate.” A GLMM's integrated likelihood includes the random-effects distribution. Fixed-design overlap is informative, but the calculation for an ordinary logistic likelihood is not, on its own, a proof about every marginal GLMM likelihood.
“A finite estimate must be precise.” Even when the maximum exists, sparse patterns can yield weak information and large standard errors. Existence, numerical convergence, and precision are different questions.

The examples also rely on independent Bernoulli observations conditional on the specified predictors, a correctly specified logit mean, and a full-rank fixed-effect design. Grouping observations into a GLMM changes the likelihood calculation; omitted interactions or alternative contrasts can change the predictor patterns being assessed. The overlap argument is therefore a useful way to reason about fixed-effect support, not a universal theorem that bypasses the chosen model.

Check Your Understanding

Answer each question, then compare with the solutions.

In a logistic model, two observations have the same design vector, one with outcome 0 and one with outcome 1. What happens to their combined likelihood contribution as their shared linear predictor tends to positive infinity?
At \(x=-1,0,1\), suppose every pattern has two zeros and two ones. Why is the zero-coefficient estimate a finite maximum in the intercept-and-slope model?
Why does an outcome mixture at \(x=0\) fail to guarantee a finite estimate when all outcomes at \(x=-1\) are zero and all at \(x=1\) are one?
Does full column rank alone guarantee a finite logistic maximum? Explain.
What does fixed-effect pattern overlap establish, and what does it not establish, for a GLMM?

Solutions. (1) The contribution is \(p(1-p)\); as the linear predictor tends to positive infinity, \(p\to1\), so \(p(1-p)\to0\). (2) At zero coefficients all fitted probabilities are \(1/2\); the intercept and slope scores both sum to zero, and the full-rank design with positive logistic weights gives a strictly concave log-likelihood. The stationary point is the unique finite maximum. (3) Taking a positive slope with intercept zero sends the fitted probabilities at \(-1\) and \(1\) toward zero and one, respectively, while leaving the mixed pattern at probability \(1/2\). The likelihood approaches a boundary, so quasi-complete separation remains possible. (4) No. Full rank supports identifiability and strict concavity at finite coefficient values, but the likelihood may still increase toward infinity under separation. (5) Overlap constrains a fixed-effect separating direction at the shared patterns; it does not rule out quasi-complete separation elsewhere or, by itself, prove a finite maximum for the integrated GLMM likelihood.