Separation in Grouped Binomial Regression

Why Separation Matters

Grouped logistic regression estimates how a success probability changes with covariates. A complication arises when the observed success-failure pattern can be sorted perfectly, or almost perfectly, by a direction in the model's covariate space. In that case, increasing a coefficient in that direction can keep improving the likelihood, even as the coefficient grows without bound. A numerical fitting algorithm may report very large estimates or fail to converge; these symptoms can reflect the data pattern rather than a merely inconvenient choice of starting values.

The diagnosis is made from the observed groups and the model's design vectors. It is not enough to notice a group with no successes or no failures: several such groups may or may not be separable, depending on their covariates and on the model. The key question is whether one coefficient direction moves every observed success toward probability one and every observed failure toward probability zero.

Grouped Logistic Setup and Definitions

For group \(i\), let \(s_i\) be the number of successes, \(f_i\) the number of failures, and \(m_i=s_i+f_i\) the trial total. Let \(\boldsymbol{x}_i\) be its fixed \(p\)-dimensional design vector, including an intercept if the model has one. The logistic model is

\operatorname{logit}(p_i)=\boldsymbol{x}_i^{\mathsf T}\boldsymbol{\beta}, \qquad p_i=\frac{\exp(\boldsymbol{x}_i^{\mathsf T}\boldsymbol{\beta})} {1+\exp(\boldsymbol{x}_i^{\mathsf T}\boldsymbol{\beta})}.

Groups with \(m_i=0\) contribute no observations and are ignored. For the remaining groups, the log-likelihood, apart from parameter-independent binomial coefficients, is

\ell(\boldsymbol{\beta}) =\sum_i \left\{ s_i\log(p_i)+f_i\log(1-p_i) \right\}.

A covariate direction is a fixed vector \(\boldsymbol{d}\) in the model's coefficient space. Along the path \(\boldsymbol{\beta}(t)=\boldsymbol{\beta}+t\boldsymbol{d}\), the linear predictor for group \(i\) changes by \(t a_i\), where \(a_i=\boldsymbol{x}_i^{\mathsf T}\boldsymbol{d}\). Thus the sign of \(a_i\) says whether this path raises or lowers that group's log odds.

Definition: A grouped pattern is completely separated if there is a direction \(\boldsymbol{d}\) such that \(a_i>0\) for every group with \(s_i>0\), and \(a_i<0\) for every group with \(f_i>0\). It is quasi-completely separated if there is a direction for which \(a_i\geq 0\) whenever \(s_i>0\), and \(a_i\leq 0\) whenever \(f_i>0\), with at least one of these inequalities strict for a group with an observed outcome. If the strict conditions hold for all observed outcomes, call the pattern complete; if only the weak conditions hold, call it quasi-complete.

These definitions apply to grouped counts as though each success and failure in a group were an individual Bernoulli outcome with the same covariate vector. In particular, a mixed group, with both \(s_i>0\) and \(f_i>0\), requires \(a_i=0\) under the weak conditions: the same direction must not move its linear predictor toward both incompatible outcomes. A group containing successes only requires a nonnegative direction, while a failures-only group requires a nonpositive one.

Worked Examples of Separation

Worked Example: Complete Separation With Two Groups

Suppose a logistic model has an intercept and one covariate \(x\), with two groups: 8 failures at \(x=-1\) and 12 successes at \(x=1\). Take the coefficient direction \(\boldsymbol{d}=(0,1)^{\mathsf T}\), so \(a_i=x_i\). The failure group has \(a_i=-1<0\), and the success group has \(a_i=1>0\). Both observed outcomes satisfy the strict conditions, so the groups are completely separated.

Along \(\boldsymbol{\beta}(t)=(0,t)^{\mathsf T}\), the two linear predictors are \(-t\) and \(t\). The failure probability at \(x=-1\) is \(1-p(-1)=1/(1+\exp(-t))\), and the success probability at \(x=1\) is \(p(1)=1/(1+\exp(-t))\). Therefore the parameter-dependent log-likelihood is

\ell(t)=8\log\!\left(\frac{1}{1+\exp(-t)}\right) +12\log\!\left(\frac{1}{1+\exp(-t)}\right) =-20\log(1+\exp(-t)).

At \(t=0\), this is \(-20\log 2\approx-13.863\). At \(t=2\), it is \(-20\log(1+\exp(-2))\approx-2.539\); at \(t=4\), it is approximately \(-0.363\). As \(t\) increases, both observed outcome probabilities approach one and \(\ell(t)\) approaches zero, but no finite \(t\) gives probability one. The coefficient \(t\) can improve the fit without reaching its limiting likelihood at a finite value.

Worked Example: Quasi-Complete Separation With a Mixed Group

Consider an intercept-and-slope model with a mixed group at \(x=0\) containing 3 successes and 2 failures, a success-only group of 15 trials at \(x=0.5\), and a failure-only group of 15 trials at \(x=-0.5\). Use the direction \(\boldsymbol{d}=(0,1)^{\mathsf T}\). The direction's changes in linear predictor are \(0\) for the mixed group, \(0.5\) for the successes, and \(-0.5\) for the failures. The weak inequalities hold, with strict inequalities for both pure groups; the mixed group has to remain unchanged. This is quasi-complete separation.

On the path \(\boldsymbol{\beta}(t)=(0,t)^{\mathsf T}\), the mixed group has \(p=1/2\) throughout and contributes \(5\log(1/2)=-5\log 2\approx-3.466\). Each pure group's contribution is \(-15\log(1+\exp(-t/2))\). Hence

\ell(t)=-5\log 2-30\log(1+\exp(-t/2)).

At \(t=2\), each pure group contributes \(-15\log(1+\exp(-1))\approx-4.699\), so the total is approximately \(-3.466-9.398=-12.864\). At \(t=4\), each contributes \(-15\log(1+\exp(-2))\approx-1.904\), giving a total of approximately \(-7.274\). In the limit, the pure groups contribute zero, while the mixed group's contribution stays at \(-3.466\). The fitted probabilities tend to one for the success-only group and zero for the failure-only group; the mixed group's fitted probability remains one half.

Worked Example: A Mixed Group Can Prevent Separation

In an intercept-only model, suppose one group has 4 successes and 6 failures. The design vector is \(x_1=1\). Because both outcomes occur, any direction satisfying the weak separation conditions must have \(a_1=d=0\). There is no strict inequality for an observed outcome, so this pattern is not separated.

Here \(p=\exp(\beta)/(1+\exp(\beta))\), and the log-likelihood is \(4\log(p)+6\log(1-p)\). Its derivative is \(4-10p\), which is zero at \(p=0.4\). The corresponding finite coefficient is \(\hat{\beta}=\log(0.4/0.6)=\log(2/3)\approx-0.405\). Its second derivative with respect to \(\beta\) is \(-10p(1-p)<0\) for finite \(\beta\), so this stationary point is the unique maximum. The mixed group supplies evidence for both outcomes at the same covariate value, rather than a direction that can push its fitted probability toward an outcome-only boundary.

Why a Separating Direction Rules Out a Finite Maximum

The likelihood behavior follows directly by differentiating each group's contribution along a candidate direction. This argument does not require a special starting value for the coefficients. Assume a finite number of groups, fixed finite design vectors, and finite counts \(s_i,f_i\). Work with the logistic model above, and suppose a complete or quasi-complete separating direction exists.

Theorem: If a direction satisfies the complete or quasi-complete separation conditions, with at least one strict inequality for an observed outcome, the log-likelihood strictly increases along that direction from every finite coefficient vector. Consequently, no finite coefficient vector maximizes the grouped logistic likelihood.

Differentiate the group contribution.
At a finite linear predictor \(\eta_i=\boldsymbol{x}_i^{\mathsf T}\boldsymbol{\beta}\), \(p_i=\exp(\eta_i)/(1+\exp(\eta_i))\) lies strictly between zero and one. If \(\eta_i\) changes at rate \(a_i\), the derivative of \(s_i\log(p_i)+f_i\log(1-p_i)\) is \(\{s_i(1-p_i)-f_i p_i\}a_i\).

Check its sign for each outcome.
Equivalently, write the group contribution as \(s_i\log(p_i)+f_i\log(1-p_i)\). A success term has derivative \((1-p_i)a_i\), nonnegative when \(a_i\geq0\); a failure term has derivative \(-p_i a_i\), nonnegative when \(a_i\leq0\).

Sum the contributions.
Every observed success and failure term has a nonnegative derivative under the separation conditions. At least one observed outcome has a strict inequality, and its derivative is strictly positive because \(0<p_i<1\). Thus the derivative of the full log-likelihood along the path is strictly positive at every finite point.

For completeness, the derivative identity in the first step follows from \(\log(p_i)=\eta_i-\log(1+\exp(\eta_i))\) and \(\log(1-p_i)=-\log(1+\exp(\eta_i))\). Their derivatives with respect to \(\eta_i\) are \(1-p_i\) and \(-p_i\), respectively. Multiplying by the rate \(a_i\) and by the observed counts gives the stated derivative. The strict increase means that from any finite \(\boldsymbol{\beta}\), moving a sufficiently small positive distance along \(\boldsymbol{d}\) increases \(\ell\). Therefore that \(\boldsymbol{\beta}\) cannot be a maximizer. Since the argument applies to every finite \(\boldsymbol{\beta}\), no finite maximum exists. \(\square\)

The likelihood is nevertheless bounded above: each probability term is at most one, so each log-probability term is at most zero. Along a separating path, the log-likelihood can approach a finite supremum while the coefficient vector diverges. For a strict success inequality \(a_i>0\), the group's fitted probability tends to one as \(t\to\infty\); for a strict failure inequality \(a_i<0\), it tends to zero. Groups with \(a_i=0\) retain their fitted probabilities along that path. These facts explain why diverging estimates do not mean that the likelihood itself becomes infinite.

Diagnosing the Pattern in Practice

For a grouped data set, a useful first check is to classify each observed group as success-only, failure-only, or mixed. Then ask whether a direction in the actual model's design space can satisfy the associated sign restrictions. This last qualification matters: changing or omitting covariates changes which directions are possible, so a pattern may be separated in one model and not in another.

For a success-only group, check whether the proposed direction gives \(a_i>0\) for complete separation, or \(a_i\geq0\) for quasi-complete separation.
For a failure-only group, check whether it gives \(a_i<0\) or \(a_i\leq0\), respectively.
For a mixed group, require \(a_i=0\) under the weak conditions. Any nonzero change moves its fitted probability toward one outcome while moving it away from the other.
Confirm that at least one observed outcome has a strict inequality; otherwise the proposed vector does not establish separation.

A single all-success or all-failure group is therefore a warning sign, not a complete diagnosis. Its covariate position and the other groups determine whether the necessary inequalities can hold simultaneously. Also check that the model includes the intended intercept and covariates before declaring a pattern separated.

Common Pitfall

Wrong reasoning: “There is an all-success group, so the data are completely separated and the logistic coefficient must diverge.” This conclusion skips the other groups and the design vectors. For example, a mixed group at the same covariate value forces the direction's change for that group to be zero; in a model whose only available direction is an intercept, that can also rule out any strict separation. Conversely, groups at different covariate values may permit a separating direction even when the data are grouped.

Correction: Test the sign restrictions for one common coefficient direction across all observed groups. Separation is a property of the outcome pattern together with the chosen model matrix, not a label attached to one group in isolation. When the restrictions do hold, the likelihood argument above explains why ordinary maximum-likelihood estimates cannot be finite; when they do not, an all-success group alone does not prove that estimates diverge.

Key takeaway: Complete separation requires strict movement toward every observed outcome; quasi-complete separation permits some groups to remain unchanged but requires at least one strict movement. In either case, a valid direction makes the likelihood increase along an unbounded coefficient path, ruling out a finite maximum-likelihood estimate.

Check Your Understanding

Use the group counts and the model's design vectors together in each answer.

What sign restriction must a separating direction satisfy for a group containing both successes and failures?
In a logistic model, a direction gives \(a_i=0\) for every observed group. Why does that direction not establish quasi-complete separation?
Explain why the likelihood can have a finite upper bound even when the coefficient estimates diverge.
A data set has a success-only group and a failure-only group. What additional information is needed before deciding whether they are separated?
For a group with \(s_i>0\), \(f_i=0\), and \(a_i>0\), what happens to its fitted success probability as the path parameter tends to infinity?