Proper Priors for Separated Fixed Effects

Why Put a Proper Prior on a Separated Fixed Effect?

In a binary GLMM, separation can make the likelihood improve as a fixed-effect coefficient grows without bound. The resulting maximum-likelihood estimate may not be finite, even when the model includes random effects. A proper prior changes the inferential target: instead of maximizing the likelihood alone, Bayesian inference combines the likelihood with a prior distribution that assigns finite total probability to the coefficient’s possible values.

This is regularization through probability weighting, not a claim that the data contain information in a direction where they do not. A prior can produce a proper posterior and finite estimates where likelihood-based estimation diverges, but the resulting inference can depend strongly on the prior’s scale and tail behavior.

Model and assumptions: Let $Y_{ij}\in\{0,1\}$ be the binary outcome for observation $i$ in cluster $j$. Conditional on random effects $b$, assume the outcomes are independent Bernoulli variables with $$ P(Y_{ij}=1\mid b,\beta)=\sigma(x_{ij}^{T}\beta+z_{ij}^{T}b),\qquad \sigma(u)=\frac{1}{1+\exp(-u)}. $$ Here $\beta$ is the vector of fixed effects, $b$ has a specified proper random-effects distribution, and any unknown random-effects parameters are collected in $\psi$. A proper prior means a prior distribution with total probability one. For the propriety result below, the joint prior on $(\beta,b,\psi)$, or on $(\beta,\psi)$ after integrating out $b$, must be proper.

The likelihood of any fixed binary data set is bounded above by one: every Bernoulli probability and its complement is at most one, and their product is therefore at most one. Under separation, however, that likelihood may approach a positive limit along a direction in $\beta$ rather than decreasing toward zero. A proper prior gives finite total prior mass in that direction, so an unbounded region of coefficient values cannot by itself make the posterior integral diverge.

Theorem: Suppose the likelihood is measurable, is bounded above by one, and is positive on a set with positive prior probability. If the joint prior for all unknown parameters is proper, the posterior distribution is proper. Indeed, its normalizing constant is positive and no greater than one. This guarantees a posterior probability distribution, not necessarily finite posterior means or variances.

Why the Posterior Is Proper

Let $\theta$ denote all unknown parameters and let $\pi(\theta)$ be their proper joint prior density. The posterior normalizing constant is $C=\int L(\theta)\pi(\theta)\,d\theta$. Since $0\leq L(\theta)\leq 1$, we have $C\leq\int\pi(\theta)\,d\theta=1$. If $L(\theta)>0$ on a set with positive prior probability, then $C>0$. Thus $0<C\leq1$, and the posterior density $L(\theta)\pi(\theta)/C$ integrates to one.

This argument applies whether the likelihood is evaluated conditionally on $b$, or whether $b$ is integrated out to form a marginal likelihood. It also shows why the qualification about all unknown parameters matters: a proper prior on $\beta$ does not repair an improper prior on a variance parameter if the joint posterior remains nonintegrable for that reason.

Specify the full probability model.
State the conditional Bernoulli model, random-effects distribution, and which parameters are unknown.

Check the prior on every unknown.
For the simple bounded-likelihood guarantee, the joint prior must be proper, not only the prior on the separated coefficient.

Separate propriety from summaries.
A proper posterior permits posterior probabilities, but a mean or variance requires a separate tail-integrability check.

Worked Examples

Worked Example: A Quasi-Separated Random-Intercept GLMM

Consider four independent clusters, each with two observations. Use the logistic model $\eta_{ij}=\beta x_{ij}+b_j$, with no intercept, and $b_j\sim N(0,1)$ independently. In every cluster, the observation with $x=1$ is a success and the observation with $x=0$ is a failure. The random-intercept variance is known here to keep attention on $\beta$.

For one cluster, conditional on $b_j=b$, the likelihood contribution is $$ \sigma(\beta+b)\{1-\sigma(b)\} =\sigma(\beta+b)\sigma(-b). $$ The marginal contribution is therefore $m(\beta)=E\{\sigma(\beta+b)\sigma(-b)\}$, and the full marginal likelihood is $L(\beta)=m(\beta)^4$.

As $\beta\to\infty$, $\sigma(\beta+b)\to1$ for every finite $b$. The integrand is bounded by one, so dominated convergence gives $m(\beta)\to E\{\sigma(-b)\}$. The normal distribution is symmetric about zero and $\sigma(-b)=1-\sigma(b)$, hence $E\{\sigma(-b)\}=1/2$. Consequently, $$ \lim_{\beta\to\infty}L(\beta)=\left(\frac12\right)^4=\frac1{16}=0.0625. $$

In the other tail, $\sigma(\beta+b)\to0$ for every finite $b$, again with an integrand bounded by one. Thus $m(\beta)\to0$ and $L(\beta)\to0$ as $\beta\to-\infty$. The likelihood has a positive limit only in the positive direction. The failures at $x=0$ are unchanged by increasing $\beta$, so this is quasi-complete separation, not complete separation: those observations have zero, not strictly positive, signed margin in the separating direction.

With any proper prior on $\beta$, and with the other unknowns given proper priors, the posterior is proper by the bounded-likelihood argument. The positive likelihood limit does not invalidate that conclusion. It does, however, make the prior’s tail behavior important for posterior moments, as the next example shows.

Worked Example: A Proper Cauchy Prior Can Have an Infinite Posterior Mean

Use the likelihood in the preceding example and put a standard Cauchy prior on $\beta$, with density $\pi(\beta)=1/\{\pi(1+\beta^2)\}$. The prior is proper. Since the likelihood is bounded and has positive values, the posterior normalizing constant is finite and positive; therefore the posterior distribution is proper.

But as $\beta\to\infty$, the likelihood tends to $1/16$, while the Cauchy density is asymptotic to $1/(\pi\beta^2)$. The unnormalized posterior’s positive-tail contribution to its absolute first moment is therefore asymptotic to $$ \beta L(\beta)\pi(\beta)\ \sim\ \frac{1}{16\pi\beta}. $$ The integral of $1/\beta$ from any positive constant to infinity diverges. Dividing by the finite posterior normalizing constant cannot remove that divergence, so $E(|\beta|\mid Y)=\infty$. In particular, the posterior mean is not defined as a finite absolute expectation.

For comparison, a $N(0,2^2)$ prior has density proportional to $\exp(-\beta^2/8)$. Its tail decays fast enough that $\int |\beta|L(\beta)\pi(\beta)\,d\beta\leq\int |\beta|\pi(\beta)\,d\beta<\infty$. The normal prior therefore gives a finite posterior first absolute moment in this example. Its density at $\beta=4$ is $\exp(-2)/(2\sqrt{2\pi})\approx0.0270$, illustrating that this prior still assigns positive mass to fairly large coefficients while having lighter tails than the Cauchy prior.

Worked Example: Exact Posterior Weighting With a Finite Prior

For a calculation with no numerical integration, consider one cluster with one success at $x=1$, modelled by $\eta=\beta+b$. Let the random intercept take values $b=-\log 3$ and $b=\log 3$, each with probability $1/2$. This specified discrete random-intercept distribution makes the marginal likelihood exact. Put prior probability $1/2$ on each of $\beta=0$ and $\beta=2$.

For $\beta=0$, the marginal likelihood of the success is $$ L(0)=\frac12\sigma(-\log3)+\frac12\sigma(\log3) =\frac12\left(\frac14+\frac34\right)=0.500. $$ For $\beta=2$, $$ L(2)=\frac12\sigma(2-\log3)+\frac12\sigma(2+\log3) \approx\frac12(0.7112+0.9568)=0.8340. $$ The unnormalized posterior weights are $0.5(0.500)=0.2500$ for $\beta=0$ and $0.5(0.8340)=0.4170$ for $\beta=2$. Their sum is $0.6670$, so $$ P(\beta=2\mid Y=1)=\frac{0.4170}{0.6670}\approx0.625. $$ The data favor the larger coefficient among the two prior-supported values, but the posterior remains a probability distribution over finite values. With only a success at $x=1$, the likelihood can keep increasing as $\beta$ increases; a likelihood-only estimate need not be finite. This example illustrates prior weighting in a separated direction, not a claim that the chosen two-point prior is generally appropriate.

What Regularization Does—and Does Not—Mean

A proper prior makes posterior inference possible under the conditions stated above, even when the likelihood has no finite maximizer. It does not create additional observations or make the separated coefficient well identified by the data. In regions where the likelihood is nearly flat, posterior location and spread can be driven substantially by the prior.

The distinction between a proper posterior and a finite summary is especially important for diffuse heavy-tailed priors. Posterior probabilities over bounded regions may be meaningful even when the posterior mean is infinite. Before reporting a mean, variance, or other moment, check the corresponding integral, such as $\int|\beta|L(\beta)\pi(\beta)\,d\beta$ for the absolute first moment.

This approach differs from maximum likelihood, which uses the likelihood alone and can send a separated coefficient toward infinity. Improper flat priors do not supply finite prior mass and do not automatically resolve posterior impropriety. Penalized likelihood and bias-reduction approaches are alternative methods with different inferential targets; a proper prior should not be described as equivalent to those procedures in every model.

Diagnostics and Common Pitfalls

Confusing propriety with a finite mean. A proper posterior integrates to one, but its tails can still make $E(|\beta|\mid Y)$ infinite.
Assuming a proper prior on one coefficient is enough. Check the joint prior for all unknown parameters, including random-effects parameters and variance components.
Treating finite estimates as data-based identification. When the likelihood approaches a plateau, the prior may be doing much of the work. Inspect whether the posterior is concentrated by the likelihood or mostly shaped by prior mass.
Misclassifying separation. A separating direction with zero margin for some observations and strict margin for others is quasi-complete separation, not complete separation.
Trusting numerical output without tail checks. A sampler returning finite values does not prove that a posterior moment exists. Check likelihood limits and prior tail order analytically where possible.

A practical diagnostic is to examine the fixed-effect direction implicated by the separation and ask what happens to the likelihood in both tails. Then compare that behavior with the prior’s tail density. If the likelihood tends to a positive constant, posterior tail integrability is essentially controlled by the prior in that direction. If the likelihood tends to zero, its rate of decay also matters.

Key takeaway: Proper priors can yield a proper posterior for separated fixed effects when the full model’s prior is proper and the likelihood is bounded. They do not guarantee finite posterior moments, nor do they remove the need to assess how much the prior determines the inference.

Check Your Understanding

Answer each question, then compare with the solutions.

Why does a likelihood bounded by one combine with a proper joint prior to give a finite posterior normalizing constant?
In the four-cluster example, what is the likelihood limit as $\beta\to-\infty$, and why?
Can a proper posterior have an infinite posterior mean? Give the tail argument from the Cauchy example.
What additional prior condition is needed beyond a proper prior on the separated fixed effect?
Why is the four-cluster example quasi-completely rather than completely separated?

Solutions. (1) The normalizing constant is at most $\int\pi(\theta)\,d\theta=1$, and it is positive when the likelihood is positive on a set of positive prior probability. (2) It is zero: the success probability $\sigma(\beta+b)$ tends pointwise to zero, and the integrand is bounded, so dominated convergence applies. (3) Yes. In the positive tail, $L(\beta)\to1/16$ and the Cauchy density is asymptotic to $1/(\pi\beta^2)$, making the absolute-moment integrand proportional to $1/\beta$, whose integral diverges. (4) The joint prior on all unknown parameters must be proper for the bounded-likelihood guarantee. (5) Failures at $x=0$ have zero margin in the increasing-$\beta$ direction; not every observation has a strict margin.