Complete Separation in Binary Logistic Regression

Separation as a Geometric Property

In binary logistic regression, the response records one of two outcomes, conventionally $y_i=1$ for a success and $y_i=0$ for a failure. The predictors place each observation at a point in covariate space. Complete separation occurs when a single linear boundary puts every observed success strictly on one side and every observed failure strictly on the other. It is a property of the observed responses and covariates, not a claim that the same boundary will classify future observations correctly.

Write the predictor vector for observation $i$ as $x_i$, including an intercept coordinate if the model has an intercept. For example, with one predictor $z_i$, use $x_i=(1,z_i)^\mathsf{T}$. A vector $b$ defines the linear score $x_i^\mathsf{T}b$, and the equation $x^\mathsf{T}b=0$ describes a line or hyperplane in the predictor coordinates. The vector $b$ here describes a geometric separator; it need not be a fitted logistic-regression coefficient.

Definition: A binary logistic-regression dataset is completely separated if there exists a vector $b$ such that $$ (2y_i-1)x_i^\mathsf{T}b>0 \qquad \text{for every observation } i. $$ Equivalently, $x_i^\mathsf{T}b>0$ for every success and $x_i^\mathsf{T}b<0$ for every failure. Thus no observed point lies on the separating boundary.

The factor $2y_i-1$ is a compact way to encode the two required signs: it is $1$ for a success and $-1$ for a failure. The strict inequality matters. A boundary that merely puts all observations on the correct side or on the boundary does not satisfy this definition of complete separation.

When the model includes an intercept, the boundary need not pass through the origin of the predictor coordinates. For a single predictor, if $b=(b_0,b_1)^\mathsf{T}$ with $b_1\ne 0$, the boundary is $z=-b_0/b_1$. With two predictors, the boundary is a line; with more predictors, it is a hyperplane. In every case, complete separation asks whether all observed points of the two response classes fall strictly into opposite half-spaces.

Reading the Definition in a Dataset

To check a proposed separator, calculate its score $x_i^\mathsf{T}b$ for every observed covariate vector and inspect the signs against the responses. A positive score is required for each success and a negative score for each failure. One conflicting observation is enough to disqualify that particular vector $b$; to establish separation, the same vector must work for the whole dataset.

The definition concerns the observed covariate points, including any repeated patterns. If two observations have exactly the same covariates but different responses, their scores under every $b$ are identical. They therefore cannot be assigned the two strict signs required for complete separation. More generally, it is useful to list distinct covariate patterns together with the responses observed at each pattern before trying to imagine a boundary.

Worked Example: A Threshold Separates One-Predictor Data

Suppose the observed data are $z=(0,1,2)$, with responses $y=(0,0,1)$. Include an intercept, so $x_i=(1,z_i)^\mathsf{T}$. Check whether $b=(-1.5,1)^\mathsf{T}$ separates the classes.

For $z=0$, the score is $x^\mathsf{T}b=-1.5+0=-1.5$. This observation is a failure, so its score has the required negative sign.

For $z=1$, the score is $-1.5+1=-0.5$. This is also a failure and again has the required negative sign.

For $z=2$, the score is $-1.5+2=0.5$. This observation is a success, so its score has the required positive sign. All three signed inequalities are strict, and the boundary is $z=1.5$. The dataset is completely separated.

This verification establishes separation of these observed points. It does not establish that all future successes will have $z>1.5$, or that the boundary has any particular predictive performance outside this dataset.

Worked Example: A Line Separates Two-Predictor Data

Let the predictors be $z_1,z_2$, and include an intercept. The failures are $(0,0),(1,0),(0,1)$; the successes are $(2,0),(0,2),(2,2)$. Use the score $h(z_1,z_2)=-1.5+z_1+z_2$, corresponding to $b=(-1.5,1,1)^\mathsf{T}$.

For the failures, the scores are \[ h(0,0)=-1.5,\qquad h(1,0)=-0.5,\qquad h(0,1)=-0.5. \] All are strictly negative. For the successes, the scores are \[ h(2,0)=0.5,\qquad h(0,2)=0.5,\qquad h(2,2)=2.5. \] All are strictly positive. Therefore this dataset is completely separated by the line $z_1+z_2=1.5$.

The check uses the sum of the coordinates, not either coordinate alone: some successes have a zero value in one predictor, and some failures have a value of one in one predictor. The relevant fact is that every observed point has the correct sign for the same linear score.

Useful Consequences of the Strict-Inequality Definition

For a finite dataset, strict separation can also be expressed with a positive margin. This does not change which datasets are separated; it shows that a separating vector can be rescaled so that every correctly signed score is at least one.

Theorem: For a finite binary dataset, complete separation holds if and only if there exists a vector $b$ satisfying $$ (2y_i-1)x_i^\mathsf{T}b\geq 1 \qquad \text{for every } i. $$

Proof. If the displayed inequalities hold, then every signed score is at least one and hence strictly positive. The defining condition for complete separation follows.

Conversely, suppose the dataset is completely separated by some vector $b$. Define $m_i=(2y_i-1)x_i^\mathsf{T}b$. Each $m_i$ is strictly positive. Since there are finitely many observations, their minimum $\delta=\min_i m_i$ exists and satisfies $\delta>0$. Set $b^\star=b/\delta$. Then, for every $i$, \[ (2y_i-1)x_i^\mathsf{T}b^\star=m_i/\delta\geq 1. \] Thus the margin-one inequalities have a solution. $\square$

This result is a convenient way to reason about strictness: for a finite dataset, one does not need to accept observations lying arbitrarily close to the boundary as a special case. If a strict separator exists, scaling it gives a positive common margin. The result does not prescribe a fitting method; it is simply an equivalent geometric description.

The geometric property also does not depend on the particular units or origin used to record predictors, provided the recoding is invertible and affine and the intercept is retained. For instance, changing a predictor from metres to centimetres or centering it changes the coordinates of the boundary but cannot create or remove complete separation.

Theorem: If the predictors are recoded by an invertible affine transformation and the model includes an intercept, complete separation is preserved.

Proof. Write the original predictor vector as $z$, and recode it as $z^\star=Az+c$, where $A$ is invertible and $c$ is a fixed vector. Suppose the original data are separated by the affine score $b_0+b^\mathsf{T}z$. Since $z=A^{-1}(z^\star-c)$, the same score, expressed in the new coordinates, is \[ b_0+b^\mathsf{T}A^{-1}(z^\star-c) =(b_0-b^\mathsf{T}A^{-1}c)+(A^{-\mathsf{T}}b)^\mathsf{T}z^\star. \] This is an affine score in $z^\star$ with exactly the same value at each observation, so it preserves every strict sign. The inverse recoding is also an invertible affine transformation; applying the same argument in reverse shows that separation in the new coordinates implies separation in the original coordinates. $\square$

A One-Predictor Criterion

With one numeric predictor and an intercept, the geometry has a particularly simple description. Assume both response classes are present. Complete separation occurs exactly when the observed values for one response class are all strictly larger than those for the other class. There is no need for the values to be evenly spaced or for the classes to form contiguous groups in the order in which the data were collected; only their positions on the predictor axis matter.

Result: For one predictor $z$, complete separation holds if and only if either the smallest success value is greater than the largest failure value, or the largest success value is less than the smallest failure value.

Proof. A separating score has the form $h(z)=a+bz$. Because both classes are present, $b$ cannot be zero: a constant score cannot be strictly positive for every success and strictly negative for every failure. If $b>0$, then $h(z)>0$ exactly when $z>-a/b$, and $h(z)<0$ exactly when $z<-a/b$. Thus every success must lie above every failure. If $b<0$, the sign order reverses, so every success must lie below every failure. This proves necessity.

For sufficiency, suppose every success value is greater than every failure value. Let $t$ be the midpoint between the largest failure and the smallest success. Then every failure has $z<t$ and every success has $z>t$. The score $h(z)=z-t$ strictly separates them. If instead every success value is less than every failure value, take the midpoint $t$ between the largest success and the smallest failure and use $h(z)=t-z$. This gives positive scores for all successes and negative scores for all failures. Therefore either strict ordering implies complete separation. $\square$

Worked Example: A Repeated Covariate Pattern Prevents Separation

Consider one predictor $z$ and the observations $(z,y)=(0,0),(0,1),(1,1)$. The pattern $z=0$ occurs once with a failure and once with a success. Let any candidate affine score be $h(z)=a+bz$.

For the failure at $z=0$, strict separation requires $h(0)=a<0$. For the success at the same value, it requires $h(0)=a>0$. These conditions contradict each other. No choice of $a$ and $b$ can separate the data.

The one-predictor criterion gives the same conclusion: the smallest success value is $0$, while the largest failure value is $0$, so neither strict ordering holds. The third observation at $z=1$ does not resolve the conflict between the two responses at the repeated pattern.

Why Recognizing Complete Separation Matters

Complete separation is worth identifying because it describes a special geometry in the data that can be obscured by looking only at marginal summaries or by inspecting fitted probabilities. A table of response counts at distinct covariate patterns can reveal a contradiction immediately, while a plot can make a separating line or threshold visible. In larger predictor spaces, the same question is whether one affine score gives every success a positive value and every failure a negative one.

A common mistake is to treat perfect classification of the observed labels as sufficient evidence of complete separation. A rule that correctly classifies observations but assigns some of them score zero has not met the strict definition: those observations lie on the boundary. Another mistake is to check predictors one at a time and conclude that no separator exists because no single predictor divides the classes. As the two-predictor example shows, a linear combination of predictors can separate the points even when either coordinate alone does not.

Conversely, a strong association is not automatically complete separation. A single repeated covariate pattern with both response values can block strict separation, even if most observations lie on opposite sides of an apparent boundary. The reliable check is always the same: identify a candidate affine score and verify every signed inequality, or show that the observed pattern makes those inequalities impossible.

Key takeaway: Complete separation means there is one affine score that is strictly positive for every observed success and strictly negative for every observed failure. Verify all observations, including repeated covariate patterns; visual impression or high classification accuracy alone is not enough.

Check Your Understanding

Use the strict signed-score definition for each question.

For one-predictor data, successes occur at $z=3,4$ and failures at $z=0,2$. Is the dataset completely separated? Give a boundary and verify the signs.
Why can a covariate pattern observed with both response values never satisfy the strict complete-separation condition?
For a two-predictor dataset, what geometric object is described by an affine score equal to zero?
Explain why an invertible change of predictor units, with an intercept in the model, does not alter whether the data are completely separated.
Can a candidate boundary that passes through one observed success establish complete separation? Explain using the strict inequality in the definition.