Why Compare Remedies on the Same Separated Data?
When a binary predictor perfectly distinguishes the observed outcomes, an ordinary logistic likelihood can keep increasing as its coefficient grows. A clustered model does not automatically avoid this problem: a random intercept describes dependence within clusters, but it need not supply information that the separated fixed effect lacks. Different remedies can produce finite estimates, yet they do so by adding different information or assumptions. Comparing them on the same hypothetical dataset helps clarify what changes and what remains comparable.
We use a binary outcome \(Y_{ij}\), where \(i\) indexes observations and \(j=1,\ldots,4\) indexes clusters. The binary predictor \(x_{ij}\) is 0 for control and 1 for treatment. Conditional on a cluster random intercept \(b_j\), suppose \(Y_{ij}\mid b_j\sim\operatorname{Bernoulli}(p_{ij})\), with \(\operatorname{logit}(p_{ij})=\alpha+\beta x_{ij}+b_j\) and \(b_j\sim N(0,\tau^2)\) independently. Here \(\operatorname{logit}(p)=\log\{p/(1-p)\}\). A positive \(\beta\) means higher conditional odds of \(Y=1\) for treatment than control, holding \(b_j\) fixed.
For a transparent numerical comparison, we use the joint conditional-mode objective: the Bernoulli log likelihood conditional on the \(b_j\)'s plus their Gaussian log densities, and, when specified, a fixed-effect penalty or prior. Set \(\tau=0.7\), known for this illustration. This is a legitimate conditional-mode estimation target, but it is not the exact marginal likelihood obtained by integrating out the random effects. That distinction matters when comparing numerical estimates across implementations.
The Separated Dataset and Its Score
In each of the four clusters, suppose three control observations all have \(Y=0\), and three treatment observations all have \(Y=1\). Thus there are 12 observations in each predictor group. Treatment status perfectly separates the outcomes: every treatment outcome is 1 and every control outcome is 0. We assume independent Bernoulli outcomes conditional on the random intercepts, the stated Gaussian random-effects distribution, and the specified predictor coding.
The data are symmetric around the log-odds midpoint. At \(b_j=0\) and \(\alpha=-\beta/2\), the control probability is \(q=\operatorname{logit}^{-1}(-\beta/2)\) and the treatment probability is \(p=\operatorname{logit}^{-1}(\beta/2)=1-q\). Each cluster has equal numbers of control failures and treatment successes, so the conditional-mode score for \(b_j\) is zero at \(b_j=0\). The intercept score is also zero at \(\alpha=-\beta/2\). The random-effect density is maximized at zero, so this symmetry gives the random-intercept modes used in the calculations below.
Substituting these values into the conditional log likelihood gives \(12\log(1-q)+12\log(p)=24\log(p)\). Differentiating with respect to \(\beta\), using \(dp/d\beta=p(1-p)/2\), yields the fixed-effect likelihood score \(12(1-p)=12/\{1+\exp(\beta/2)\}\). It is positive for every finite \(\beta\) and approaches zero as \(\beta\) grows. Consequently, the unpenalized likelihood has no finite maximizing treatment coefficient in this example.
Worked Example: A Proper Prior on the Treatment Effect
Place a proper prior \(\beta\sim N(0,6.25)\) on the treatment coefficient. Keep the intercept unpenalized and retain the Gaussian random-effect terms. The prior contributes \(-\beta^2/(2\cdot6.25)\) to the log objective, so the equation for the treatment coefficient’s mode is \(12/\{1+\exp(\beta/2)\}-\beta/6.25=0\).
Numerically solving this equation gives \(\hat{\beta}\approx5.196\). For a check, \(\exp(5.196/2)=\exp(2.598)\approx13.44\), so the likelihood-score term is \(12/(1+13.44)\approx0.8312\). The prior-score term is \(5.196/6.25\approx0.8314\). Their small difference is due to rounding the root.
At \(b_j=0\), the fitted control probability is \(\operatorname{logit}^{-1}(-5.196/2)\approx0.0693\), and the fitted treatment probability is \(\operatorname{logit}^{-1}(5.196/2)\approx0.9307\). The conditional odds ratio is \(\exp(5.196)\approx180.5\). These finite values come from the prior as well as the observed outcomes; they are not the unpenalized maximum-likelihood estimate.
Worked Example: A Ridge-Penalized Fit
For a second fit, maximize the same conditional log likelihood minus the ridge penalty \(\lambda\beta^2/2\), with \(\lambda=0.5\); again, the intercept is not penalized. The score equation is \(12/\{1+\exp(\beta/2)\}-0.5\beta=0\). Its numerical root is \(\hat{\beta}\approx3.521\). Substitution gives a likelihood-score term of approximately \(12/\{1+\exp(1.7605)\}\approx1.7607\), close to the penalty-score term \(0.5(3.521)=1.7605\).
The fitted probabilities at \(b_j=0\) are about \(\operatorname{logit}^{-1}(-1.7605)=0.1467\) for control and \(\operatorname{logit}^{-1}(1.7605)=0.8533\) for treatment. The conditional odds ratio is \(\exp(3.521)\approx33.8\). The ridge coefficient is smaller than the proper-prior estimate because this example’s ridge penalty is stronger in the coefficient score equation: \(0.5\beta\), rather than \(\beta/6.25=0.16\beta\).
A Gaussian prior and a quadratic ridge penalty have the same mathematical form when their variances and penalty strengths correspond. In this comparison they do not correspond: the prior has variance 6.25, whereas the ridge fit uses \(\lambda=0.5\), equivalent in this objective to a zero-centred Gaussian penalty with variance \(1/0.5=2\). “Prior” and “ridge” do not by themselves determine the amount of shrinkage; the scale must be stated.
Worked Example: Revising the Data Collection
Suppose investigators collect four additional observations in two of the clusters: in each of those clusters, one control observation has \(Y=1\), and one treatment observation has \(Y=0\). This is a hypothetical design illustration, not a claim that observations should be altered. The revised data have 14 control observations with 2 successes and 14 treatment observations with 12 successes. Each cluster remains symmetric: where added observations occur, the control and treatment outcomes mirror one another.
With \(\alpha=-\beta/2\), write \(p=\operatorname{logit}^{-1}(\beta/2)\) for the treatment probability and \(1-p\) for the control probability. The likelihood score for \(\beta\) is \(12-14p\). Setting it to zero gives \(p=12/14=6/7\), and therefore \(\hat{\beta}=2\operatorname{logit}(6/7)=2\log(6)\approx3.584\). Unlike the original data, the revised data have a finite unpenalized maximum: each predictor group now contains both outcomes.
The conditional fitted probabilities at \(b_j=0\) are \(1/7\approx0.1429\) for control and \(6/7\approx0.8571\) for treatment. The estimate is finite because the added observations provide outcome variation within both predictor groups—not because a penalty has bounded the coefficient. The conditional odds ratio is \(6^2=36.0\).
What the Comparison Shows—and Does Not Show
For this example, the proper-prior fit gives \(\hat{\beta}\approx5.196\), the chosen ridge fit gives \(\hat{\beta}\approx3.521\), and the revised-data fit gives \(\hat{\beta}\approx3.584\). Their conditional treatment probabilities at a random intercept of zero are approximately 0.9307, 0.8533, and 0.8571, respectively. These are predictions for a cluster with \(b_j=0\), not population-averaged probabilities. A population-averaged probability instead averages \(\operatorname{logit}^{-1}(\alpha+\beta x+b)\) over the random-effects distribution; in general, that average is not \(\operatorname{logit}^{-1}(\alpha+\beta x)\).
The close ridge and revised-data estimates here do not establish that either approach generally reproduces the other. The ridge estimate depends on \(\lambda\); the data-revised estimate depends on which observations are collected and on whether they represent the target population. A proper prior expresses a distributional assumption about plausible fixed effects. A ridge penalty expresses a chosen shrinkage strength. Collecting informative outcomes can address a lack of information in the data, provided the collection process is scientifically defensible. Changing labels, discarding inconvenient observations, or combining categories just to remove separation does not create valid information.
The numerical derivation also depends on the chosen fitting target. In a marginal-likelihood GLMM, the likelihood integrates over the random-intercept distribution rather than setting \(b_j\) to its conditional mode. Marginal estimates and predictions can differ from the conditional-mode values shown here. For real analyses, check which objective the software optimizes, whether the random-effects distribution and estimation method are appropriate, and whether estimates and predictions change meaningfully under reasonable prior or penalty choices. Extreme coefficients, fitted probabilities nearly equal to zero or one, large changes under modest regularization changes, and optimizer instability are reasons to examine the model and data—not automatic evidence that one particular remedy is correct.
Common Pitfall: Treating a Finite Estimate as a Cure-All
A common mistake is to conclude, “The ridge estimate is finite, so the separation problem has been solved and this must be the best estimate.” A finite penalized estimate answers a narrower question: where does the selected penalized objective attain its maximum? It does not show that the penalty is appropriate, that uncertainty is negligible, or that the model’s assumptions fit the data. Similarly, a proper prior makes the posterior well-defined but its influence can be substantial when the likelihood contains little information in the separating direction. Report the remedy and its tuning or prior scale, assess sensitivity, and describe conditional versus marginal predictions accurately.
The strategies are alternatives, not interchangeable repairs. A proper prior contributes explicit prior information; ridge applies quadratic shrinkage; revised data collection seeks additional outcome information. Firth correction is another standard method, but it is not the same as either a normal prior or ridge penalty. The appropriate comparison depends on the scientific question and the origin of the separation.
State which predictor pattern separates which outcomes, and check whether it occurs across all clusters or only some of them.
Distinguish an integrated likelihood fit from a conditional-mode or penalized conditional fit, and specify any prior or penalty strength.
Compare coefficients on the same scale and predictions at the same random-effect value; do not confuse cluster-specific and population-averaged probabilities.
Ask whether finite estimates result from a stated regularization assumption or from additional representative data, and check sensitivity before drawing substantive conclusions.
Check Your Understanding
Answer the questions, then compare your reasoning with the brief solutions.
- Why does the unpenalized treatment coefficient have no finite maximum in the original dataset?
- For the proper-prior fit, what is the treatment probability at \(b_j=0\), and what does it describe?
- What ridge penalty strength \(\lambda\) corresponds to the zero-centred normal penalty with variance 6.25?
- Why does the revised dataset have a finite unpenalized estimate?
- Why is \(\operatorname{logit}^{-1}(\alpha+\beta x)\) generally not the population-averaged prediction in a random-intercept GLMM?
Solutions. (1) The likelihood score \(12/\{1+\exp(\beta/2)\}\) stays positive at every finite \(\beta\), so increasing \(\beta\) continues to improve the likelihood. (2) It is \(\operatorname{logit}^{-1}(5.196/2)\approx0.9307\), the fitted conditional event probability for treatment in a cluster with random intercept zero. (3) Since \(\beta^2/(2\cdot6.25)=\lambda\beta^2/2\), \(\lambda=1/6.25=0.16\). (4) Both predictor groups contain observed successes and failures, so the likelihood no longer improves indefinitely by sending the coefficient toward infinity. (5) The population-averaged prediction integrates the inverse-logit over the distribution of \(b\), and the inverse-logit is nonlinear, so averaging its values is generally not the same as evaluating it at the mean random effect.