Why Factor Coding Can Mislead a Separation Check
A factor’s coefficient values depend on how its levels are represented. With reference-cell coding, coefficients usually compare each nonreference level with a named baseline. With sum-to-zero contrasts, the coefficients describe deviations under a different constraint. The numerical values—and even which coefficient is zero—can therefore change when the coding changes.
That change in description does not, by itself, change the fitted linear predictors. If two codings represent the same fixed-effect model space, they also represent the same geometric separation of the observed binary outcomes by that design. This distinction matters in GLMMs: a large or apparently divergent coefficient under one coding is not itself proof of separation, and a diagnostic should assess the design rather than rely on a particular coefficient’s size.
Here \(m_i\) includes the intercept column, if there is an intercept, as well as the fixed-effect columns. The signed margin is positive when the direction moves an observation’s linear predictor toward its observed outcome: upward for a success and downward for a failure. The definition is geometric; it does not depend on what a particular coefficient is called.
Model and Scope of the Invariance Claim
For a binary GLMM, a standard conditional model is
The usual GLMM assumptions include conditional independence of the responses given the random effects and the specified fixed and random effects, the Bernoulli response model, and the stated random-effects distribution. The separation definition above concerns a specified binary-response design and its linear-predictor columns. In particular, a fixed-effect separation check uses \(m_{ij}\) and fixed-effect direction \(d\). It is a geometric diagnostic, not by itself a general existence theorem for every integrated, marginal GLMM likelihood.
Suppose two full-rank fixed-effect codings describe the same model space. Write their design matrices as \(M\) and \(M_*=MA\), where \(A\) is an invertible matrix. If \(\theta\) is the coefficient vector for \(M\), then the same linear predictors are obtained with \(\theta_*=A^{-1}\theta\), because \(M_*\theta_*=MAA^{-1}\theta=M\theta\). Thus, for a factor with an intercept and all \(K-1\) contrast columns, reference-cell and full-rank alternative contrasts give different coordinates for the same fitted values.
The equivalence requires the same model space. Dropping a level, omitting an interaction in only one model, imposing a different constraint, or using a rank-deficient parametrization without appropriate handling may change the set of possible predictors. Those are not merely changes of contrast labels.
Why Separation Is Preserved
The invariance follows directly from the signed margins. When \(M_*=MA\), a direction \(d\) in the original coefficient coordinates corresponds to \(d_*=A^{-1}d\) in the new coordinates. For every row,
Every signed margin is therefore identical. A direction satisfying the complete-separation inequalities in one coding satisfies them in the other; the same holds for quasi-complete separation. Applying the inverse transformation proves the converse, so the separation status is the same in both codings.
There is also a likelihood justification for why a separating direction is diagnostically consequential. Conditional on fixed random effects, the Bernoulli log likelihood for linear predictors \(\eta_i\) is \(\ell=\sum_i[y_i\eta_i-\log(1+\exp(\eta_i))]\). Along a direction \(d\), the derivative for observation \(i\) is \((y_i-p_i)m_i^\mathsf{T}d\). If the signed margin is nonnegative, this derivative is nonnegative: for a success, \(m_i^\mathsf{T}d\geq0\) and \(1-p_i>0\); for a failure, \(m_i^\mathsf{T}d\leq0\) and \(-p_i<0\). If at least one margin is strict, that observation contributes a strictly positive derivative at finite coefficients. The likelihood can keep increasing along the direction rather than attaining a finite maximum in that conditional fixed-effect likelihood. Invariance of the margins means this direction cannot be created or removed by an invertible contrast recoding.
Worked Example: Translating Reference-Cell Coefficients
Suppose a three-level factor has levels A, B, and C, with A as the reference. A hypothetical fitted model has intercept \(-1.2\), coefficient \(0.7\) for B versus A, and coefficient \(1.1\) for C versus A. Find the three level-specific logits and express them using contrasts whose level rows are A: \((1,0)\), B: \((0,1)\), and C: \((-1,-1)\).
Under reference-cell coding, the logits are \(\eta_A=-1.2\), \(\eta_B=-1.2+0.7=-0.5\), and \(\eta_C=-1.2+1.1=-0.1\). These are log odds; no probability is calculated without applying the inverse-logit transformation.
Under the alternative coding, write each logit as \(\alpha+\gamma_1c_1+\gamma_2c_2\). The three logits must satisfy \(\eta_A=\alpha+\gamma_1\), \(\eta_B=\alpha+\gamma_2\), and \(\eta_C=\alpha-\gamma_1-\gamma_2\). Their average is \(\alpha=(-1.2-0.5-0.1)/3=-0.6\). Hence \(\gamma_1=\eta_A-\alpha=-1.2-(-0.6)=-0.6\) and \(\gamma_2=\eta_B-\alpha=-0.5-(-0.6)=0.1\).
Checking the last level gives \(-0.6-(-0.6)-0.1=-0.1\), as required. The coefficient descriptions differ, but all three fitted logits are unchanged. The coefficient \(\gamma_1\) is not the B-versus-A contrast: the contrast rows, not the coefficient labels alone, determine its meaning.
Worked Separation Check Under Two Codings
Worked Example: A Separated Two-Level Factor
Consider hypothetical binary data with three successes at factor level A and two failures at level B. Use an intercept and the factor as the only fixed effects. Check separation using both B-reference treatment coding and a sum contrast coded \(x=1\) for A and \(x=-1\) for B.
With B as the reference, each row has design \((1,I_A)\), where \(I_A=1\) for A and \(0\) for B. Choose direction \(d=(-1,2)^\mathsf{T}\). For a B observation, the directional change is \((1,0)d=-1\), which moves a failure’s logit downward. Its signed margin is \((2(0)-1)(-1)=1\). For an A observation, the change is \((1,1)d=1\), which moves a success’s logit upward; its signed margin is \((2(1)-1)(1)=1\). Every observation has a strictly positive signed margin, so the fixed-effect design is completely separated.
With the sum contrast, the rows are \((1,1)\) for A and \((1,-1)\) for B. Direction \(d_*=(0,1)^\mathsf{T}\) gives a change of \(1\) at A and \(-1\) at B. Each of the three successes and two failures again has signed margin \(1\). The fitted direction points toward the outcomes under both codings, although its coefficient coordinates have changed.
For the first direction scaled by a positive number \(t\), the logits move by \(t\) for A and \(-t\) for B. As \(t\) grows, the conditional Bernoulli probabilities approach 1 for A and 0 for B. The example demonstrates fixed-effect design separation; it should not be read as proof that every marginal GLMM with arbitrary random-effects structure has the same likelihood behavior.
Using Contrasts in Diagnostics
A reliable workflow separates the diagnostic question from coefficient interpretation. First verify which observations and columns define the design being checked. Next verify that alternative codings retain the same factor levels, intercept, interactions, and rank. Then use a geometric, linear-programming, or likelihood-based separation diagnostic appropriate to that design. If the diagnosis changes after an allegedly equivalent recoding, investigate rank, convergence, numerical tolerances, and whether the model spaces really match.
Identify the binary observations and specify whether the diagnostic concerns fixed-effect columns or a different conditional design.
Confirm that the recoding is a full-rank change of basis for the same terms, including the intercept and any interactions.
Assess signed margins or an equivalent separation test; do not infer separation solely from one large coefficient or a convergence warning.
Compare model matrices and fitted linear predictors, then check rank deficiency, software tolerances, and optimization behavior.
Different contrast systems remain useful even though they do not change separation status. A reference-cell contrast makes comparisons with a named baseline explicit. Sum-to-zero and other contrasts can make different scientific comparisons more direct. Choose a coding for interpretability, then make sure conclusions about separation are based on the represented design rather than on whether one particular coefficient is large, unstable, or absent under a chosen constraint.
| Question | Reference-cell coding | Equivalent full-rank contrasts |
|---|---|---|
| What changes? | Baseline-relative coefficient descriptions | Coordinates and coefficient interpretations |
| What stays the same? | The model space, possible fitted linear predictors, and separation status | |
| When might separation status differ? | When the model space, included terms, rank, or diagnostic design changes | |
Failure Modes and Limitations
A common mistake is to treat a large coefficient under one coding as evidence that the factor is separated, then expect another coding to show the same large coefficient. The correction is that coefficient magnitudes and meanings are coordinate-dependent. Check whether a separating direction exists in the design; under an invertible recoding, the direction’s coordinates change while its signed margins do not.
Another mistake is to treat any software result as a pure geometric verdict. A convergence warning can occur without separation, while a finite-looking estimate does not by itself establish overlap. Numerical diagnostics can also disagree because of tolerances, rank deficiency, or different design matrices. Compare the matrices and the fitted linear predictors, and distinguish optimization failure from a separating direction.
Finally, invariance of the fixed-effect design does not mean that every part of a GLMM is invariant under every transformation. If a factor is used in the random-effects structure as well, changing its random-slope basis can alter covariance restrictions. For example, a general covariance matrix can be transformed along with a change of basis, but a diagonal covariance restriction need not remain diagonal after that transformation. The fixed-effect contrast argument applies when the fixed-effect model space is unchanged; the random-effects structure must be checked separately.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A factor is recoded using an invertible contrast transformation while retaining the same intercept and terms. Does this alone alter complete or quasi-complete separation?
- For a success with design row \(m_i^\mathsf{T}\), a direction gives \(m_i^\mathsf{T}d=-0.4\). What is its signed margin, and does it point toward the observed outcome?
- Why might two models using different factor codings legitimately have different separation status?
- Why does a large coefficient under one contrast system not, by itself, establish separation?
Solutions. (1) No. If the recoding is a full-rank change of basis, each direction maps to the other coordinates with identical signed margins. (2) The margin is \((2(1)-1)(-0.4)=-0.4\), so it is negative and points away from the success. (3) The codings may not represent the same model space—for example, they may include different interactions, constraints, or factor levels. (4) Coefficient size and meaning depend on the parameterization; separation is determined by whether a direction satisfies the signed-margin inequalities for the design.