Worked Example of Cluster-Specific Separation

Why the Location of Separation Matters

In clustered binary data, the same response pattern can look very different depending on whether observations are pooled across clusters or examined within them. A predictor may perfectly divide successes from failures in the entire data set; it may do so only within some clusters; or clusters may show perfectly separating slopes in opposite directions. These patterns are not interchangeable. They have different implications for a common fixed effect, for cluster-specific effects, and for the information available to estimate variation in slopes.

Consider a logistic mixed model with a random intercept and, when specified, a random slope:

Y_{ij}\mid b_j \sim \operatorname{Bernoulli}(p_{ij}), \qquad \operatorname{logit}(p_{ij})=\beta_0+\beta_1x_{ij}+b_{0j}+b_{1j}x_{ij}.

Here \(i\) indexes observations within cluster \(j\), \(\beta_0\) and \(\beta_1\) are population-level fixed effects, and \(b_{0j}\) and \(b_{1j}\) are cluster-specific deviations. A random-intercept-only model sets \(b_{1j}=0\). In the random-slope model, the random-effects vector \(b_j=(b_{0j},b_{1j})^\mathsf{T}\) is typically assumed to follow a mean-zero multivariate normal distribution with covariance matrix \(D\), independently across clusters and independently of the observed predictors under the model.

Definition: Pooled separation means a fixed-effect direction separates the observed outcomes across all clusters when their observations are considered together. Cluster-specific separation means a separating direction exists within one or more clusters, possibly with different directions in different clusters. Random-slope separation describes a pattern in which within-cluster predictor contrasts perfectly separate outcomes, so cluster-specific slope directions are separating even if the pooled data do not have a separating direction. These are descriptions of the data pattern; they are not, by themselves, guarantees about every parameter of an integrated GLMM likelihood.

A Pooled Separating Direction

For a logistic model with fixed-effect design vector \(x_{ij}\), a strict pooled separating direction is a vector \(a\) for which \((2Y_{ij}-1)x_{ij}^{\mathsf T}a>0\) for every observation. Moving the fixed-effect vector farther along \(a\) then pushes every fitted probability toward its observed outcome. In a mixed model, this fixed-effect separation remains important even though the likelihood integrates over random effects.

Worked Example: Separation in the Same Direction Across Clusters

Suppose two clinics each contribute four patients at \(x=0\) and four at \(x=1\). At both clinics, all four patients with \(x=0\) have \(Y=0\), and all four with \(x=1\) have \(Y=1\). The data are hypothetical.

Clinic	Successes at \(x=0\)	Successes at \(x=1\)	Pattern
A	0 of 4	4 of 4	Increasing predictor, increasing outcome
B	0 of 4	4 of 4	Increasing predictor, increasing outcome

The pooled data have 0 successes among 8 observations at \(x=0\), and 8 successes among 8 at \(x=1\). Thus \(x\) strictly separates the outcomes in the same direction in both clinics. In a model with a random intercept, take \(\beta_1=t\) and \(\beta_0=-t/2\). For any finite clinic intercept \(b_{0j}\), the linear predictor tends to \(-\infty\) at \(x=0\) and \(+\infty\) at \(x=1\) as \(t\) increases. Each observed response probability therefore tends to 1.

The resulting likelihood approaches its upper bound, 1, as the fixed slope increases without bound. The finite-slope model cannot assign probability 1 to every observed binary outcome, so the likelihood supremum is not attained at a finite slope. This is the key fixed-effect consequence of strict pooled separation: ordinary maximum likelihood does not yield a finite estimate in that direction. Adding a random intercept does not undo a predictor pattern that separates every observation in the same direction.

Separation Within Clusters Can Cancel When Pooled

A different pattern occurs when each cluster is perfectly predictable from its own \(x\)-contrast, but the directions disagree. Pooling can then hide the within-cluster structure. A random intercept shifts a cluster's baseline log-odds but cannot reverse its predictor slope; a random slope allows cluster-specific slope deviations, but those deviations are drawn from a shared distribution rather than estimated as unrelated unrestricted parameters.

Worked Example: Opposite Separating Directions

Suppose clinic A has four failures at \(x=0\) and four successes at \(x=1\). Clinic B has four successes at \(x=0\) and four failures at \(x=1\). The data are hypothetical.

In clinic A, increasing \(x\) perfectly predicts success, so a clinic-specific logistic slope can increase without bound. In clinic B, increasing \(x\) perfectly predicts failure, so its clinic-specific slope can decrease without bound. But pooling gives four successes out of eight at each predictor value:

\widehat{\operatorname{OR}}_{\text{pooled}} = \frac{(4/4)}{(4/4)} =1, \qquad \hat{\beta}_{1,\text{pooled}}=\log(1)=0.

The pooled two-group comparison has no separating direction in \(x\): both outcomes occur at both predictor values. Its finite slope estimate of zero describes the pooled comparison, not the two sharply opposed within-clinic patterns. A random-intercept model cannot represent those opposite slopes simply by shifting clinic baselines. A random-slope model can represent slope heterogeneity, but the two cluster patterns do not mean that the population fixed slope must be infinite. The mixed model integrates over a distribution of random slopes; it does not treat each cluster slope as a separate unconstrained fixed parameter.

This distinction is especially important when there are few clusters. A small number of perfectly separated within-cluster contrasts can provide weak information about the distribution of slopes. A fitted random-slope variance may be imprecise or unstable, and the fixed slope may depend strongly on the model and the observed balance of clusters. The cluster patterns are a warning to inspect the model, not a mechanical rule that dictates the fixed-effect estimate.

What the Integrated Likelihood Shows

The likelihood of a GLMM is obtained by averaging the conditional response likelihood over the random-effects distribution. For observed data \(y\), it has the form

L(\beta,D) = \prod_{j=1}^{J} \int \left\{ \prod_{i=1}^{n_j} P(Y_{ij}=y_{ij}\mid b_j;\beta) \right\} f(b_j;D)\,db_j.

This expression explains why separation diagnostics based on cluster-specific fits need careful interpretation. A conditional fit that lets every cluster have an unrelated slope may send those slopes to infinity. The GLMM instead assigns a probability distribution to the cluster effects and integrates over it. A separating slope in one cluster is not an independent fixed coefficient in that integrated model.

There is, however, a direct justification for the pooled strict-separation warning. Suppose a fixed-effect direction makes every observed outcome probability tend to 1 for every finite value of \(b_j\). The conditional likelihood for each cluster then tends to 1. It is bounded between 0 and 1, so the dominated convergence theorem allows the limit to pass through the integral:

\lim_{t\to\infty} \int L_j(\beta+ta,b_j)f(b_j;D)\,db_j = \int \lim_{t\to\infty}L_j(\beta+ta,b_j)f(b_j;D)\,db_j = \int f(b_j;D)\,db_j =1.

Thus the integrated likelihood approaches its response-probability upper bound along that fixed-effect direction. This argument relies on strict separation: every observed response must be driven toward its observed outcome for each finite random-effect value. Quasi-complete separation has observations on the boundary, and cluster-specific patterns with opposing directions do not satisfy this same pooled condition. They require examining the actual model and likelihood rather than applying the strict pooled argument automatically.

Random-Slope Separation With a Finite Pooled Association

Worked Example: A Finite Overall Slope and Separated Cluster Slopes

Consider three hypothetical clinics, each with four observations at each value of \(x\). Clinics A and C have no successes at \(x=0\) and four successes at \(x=1\). Clinic B has four successes at \(x=0\) and none at \(x=1\).

Clinic	Successes at \(x=0\), out of 4	Successes at \(x=1\), out of 4
A	0	4
B	4	0
C	0	4

Every clinic has a separating within-clinic slope, but B's direction is opposite to A's and C's. Pooled across clinics, there are 4 successes among 12 observations at \(x=0\), and 8 among 12 at \(x=1\). The pooled odds are \(4/8=0.5\) at \(x=0\) and \(8/4=2\) at \(x=1\), so the pooled odds ratio is

\widehat{\operatorname{OR}}_{\text{pooled}} = \frac{8/4}{4/8} =4, \qquad \hat{\beta}_{1,\text{pooled}}=\log(4)\approx 1.386.

This finite calculation is the unadjusted pooled logistic comparison, not a fitted random-slope GLMM estimate. It shows why the aggregate data are not strictly separated: there are both successes and failures at each \(x\). At the same time, each clinic has a perfectly separating contrast. A random-slope model may use the repeated within-clinic contrasts to represent heterogeneity, while its fixed slope describes the mean slope in the specified model. Its estimate need not equal \(1.386\), and the data shown alone do not guarantee a precise estimate of either the fixed slope or the random-slope variance.

This example also shows why “there is no pooled separation” is not the same as “there is no separation issue.” Separation confined to clusters can make conditional estimates extreme, expose a mismatch in a random-intercept-only model, or leave too little information to estimate slope variation reliably. Conversely, cluster-specific separation does not establish that the integrated model's fixed slope has no finite estimate.

Diagnostics, Alternatives, and Limits

A useful diagnostic sequence keeps the three levels separate: inspect the pooled predictor-outcome pattern, tabulate the pattern within each cluster, and then assess whether the assumed random-effects structure can represent the observed differences. For a categorical predictor, cross-tabulations by cluster often make the pattern visible. For a continuous predictor, plots of outcomes against the predictor within clusters can reveal whether outcomes fall on one side of a cluster-specific threshold.

Signs of pooled separation: one or more predictor values or combinations consistently correspond to only one outcome across the full data set. Check whether a fixed-effect direction separates all observations, rather than relying only on large estimates or warnings.
Signs of cluster-specific separation: within a cluster, one outcome occurs only on one side of a predictor contrast. Record the direction for each cluster; directions may agree or conflict.
Signs the random-effects structure may be inadequate: clusters show sharply different slopes, while the fitted model allows only a common slope and random intercepts. Check whether a random-slope model is scientifically justified and whether there are enough clusters and within-cluster predictor variation to estimate its additional parameters.
Signs of unreliable estimation: very large fixed-effect estimates or standard errors, strong sensitivity to individual clusters, unstable random-slope variance estimates, or fitting warnings. These are prompts for investigation, not proof of separation on their own.

The main alternatives answer different questions. A pooled logistic regression describes an overall fixed association without modeling cluster dependence. A random-intercept model allows cluster baselines to vary while retaining a common slope. A random-slope model allows slope variation according to a specified distribution. A model with cluster-specific fixed slopes estimates a separate slope for each cluster and can make within-cluster separation explicit, but it is not the same model as a random-slope GLMM. None of these choices should be selected solely to make an estimate finite: the target of inference, design, and assumptions matter.

The limitations are consequential. Random-slope models need meaningful within-cluster predictor variation and a sufficient number of clusters to estimate slope variation credibly. A normal random-effects distribution is a modeling assumption, not a guarantee that a few extreme cluster patterns are well represented. Also, pooled summaries can conceal opposing cluster associations, while cluster-specific summaries can exaggerate how much the data say about a population distribution. Interpret fixed effects with the model's conditional structure in view; they are not automatically marginal effects for the population averaged over random effects.

Common Mistake

Wrong reasoning: “Every clinic has a perfectly separating slope, so the GLMM fixed slope must be infinite.” This treats cluster slopes as unrelated parameters and ignores the random-effects distribution and likelihood integration.

Correction: First ask whether one fixed-effect direction strictly separates all observations pooled across clusters. If so, the integrated likelihood can approach its supremum at an infinite fixed-effect value, as the limit argument shows. If the cluster-specific directions differ, pooled separation may be absent. Then investigate the random-slope structure and the information supporting it; do not infer an infinite fixed slope merely from separated cluster-specific fits.

Key takeaway: State where the separating direction occurs and whether its direction agrees across clusters. Strict pooled separation directly threatens finite maximum-likelihood fixed-effect estimation; cluster-specific separation with conflicting directions is a different pattern whose consequences depend on the integrated model and its random-effects structure.

Check Your Understanding

Answer each question, then compare with the brief solutions.

Two clusters both show failures at \(x=0\) and successes at \(x=1\). Is the pattern pooled separation, cluster-specific separation, or both?
Two clusters have opposite perfectly separating slopes, with equal numbers of successes and failures pooled at each predictor value. Does this establish an infinite pooled fixed slope?
In the three-clinic example, calculate the pooled odds ratio from the success counts.
Why does a random-intercept model not represent clusters whose predictor slopes have opposite directions?
What should be concluded if a cluster-specific logistic fit has an infinite slope but the pooled data do not have a separating direction?

Solutions. (1) Both: the common direction separates the pooled observations and each cluster. (2) No. The pooled predictor comparison has both outcomes at each value, so those opposite cluster patterns do not themselves give a strict pooled separating direction. (3) The pooled odds are \(4/8=0.5\) at \(x=0\) and \(8/4=2\) at \(x=1\), giving an odds ratio \(2/0.5=4\). (4) A random intercept changes a cluster's baseline log-odds but leaves the common slope direction unchanged. (5) Treat it as evidence of within-cluster separation, then examine the pooled pattern, the random-effects specification, and estimation stability; it does not by itself prove that the integrated GLMM fixed slope is infinite.