Why Design Matters When Outcomes Separate
When a binary GLMM has separation, the observed outcomes align so strongly with one or more predictors that a fixed-effect coefficient may have no finite maximum-likelihood estimate. Changing an optimizer can help with numerical computation, but it cannot create outcome patterns that were not observed. Data collection can address the underlying problem—but only if new observations provide information in the predictor directions where outcomes currently fail to overlap.
The practical question is therefore not simply “How many more observations should be collected?” It is “Which outcome patterns, predictor values, and clusters are missing?” Recruiting more events may help when events are rare, but adding events exclusively where events are already concentrated can leave separation untouched. Sampling over predictor ranges, with outcomes represented within those ranges and across clusters, can be more directly informative.
Separation is a pattern in the relationship between outcomes and predictors, not simply a small-sample label. As discussed in the earlier material on separation diagnostics, it is important to distinguish that pattern from an optimizer or convergence failure. Design planning can use diagnostics such as outcome-by-predictor tables, sparse-cell checks, and cluster-specific outcome patterns to locate the missing overlap. A global event count alone cannot show whether the events occur in the predictor regions that matter.
Why Opposing Outcomes Can Matter More Than More Events
A short likelihood argument clarifies the distinction. Consider a logistic regression component with linear predictor \(\eta_i=\mathbf{x}_i^T\boldsymbol{\beta}\). Suppose a direction \(d\) in coefficient space separates the outcomes: for each observation, \((2y_i-1)\mathbf{x}_i^Td\geq 0\), with a strict inequality for at least one observation. Along \(\boldsymbol{\beta}(t)=\boldsymbol{\beta}+td\), the derivative of the log likelihood is
For an event, \(y_i=1\), so its contribution is \((1-p_i(t))\mathbf{x}_i^Td\), which is nonnegative under the separation condition. For a non-event, \(y_i=0\), the condition implies \(\mathbf{x}_i^Td\leq 0\), and its contribution is \(-p_i(t)\mathbf{x}_i^Td\), also nonnegative. If at least one observation has a strict inequality, its contribution is positive at finite \(t\). Thus the likelihood can keep increasing as the coefficient moves in direction \(d\), instead of attaining a finite maximum.
This argument explains the geometry for the logistic fixed-effect likelihood; it is not a proof that every integrated GLMM likelihood has exactly the same form or behavior. In a GLMM, cluster effects, random slopes, and the integration over their distribution also matter. Still, the argument gives a useful design principle: observations that oppose the separating direction can weaken the pattern; observations that reinforce it cannot. They must be collected in relevant covariate and cluster settings, not merely added somewhere in the data.
Worked Example: Adding Events Where Events Are Missing
Suppose a hypothetical study has one binary predictor \(x\), coded \(-1\) for a lower-risk range and \(+1\) for a higher-risk range. Initially, all 10 observed non-events have \(x=-1\), and all 10 events have \(x=+1\). Researchers plan to recruit 10 additional events, but all are expected to have \(x=+1\). Does this plan oppose the separating pattern?
Check the observed pattern. The outcome is perfectly aligned with \(x\): events are at \(+1\) and non-events at \(-1\). Along an increasing slope \(\beta\), with intercept \(\alpha\) held fixed, the event linear predictor \(\alpha+\beta\) increases and the non-event predictor \(\alpha-\beta\) decreases. Both changes make the observed outcomes more likely.
Check what the proposed recruitment changes. The event count rises from 10 to 20, a doubling. But all 10 added events are at \(x=+1\), where events already occur; no added observation opposes the separating direction. More generally, if the event count were increased from 10 to 30 in the same way, it would have tripled, since \(30/10=3\), and the separating pattern would still remain.
The likelihood derivative in the slope direction confirms the point. If there are \(m\) events at \(+1\) and \(c\) non-events at \(-1\), then \(d\ell/d\beta=m(1-p_{+})+c p_{-}>0\) for finite \(\alpha,\beta\), where \(p_{+}\) and \(p_{-}\) are the fitted probabilities at \(x=+1\) and \(x=-1\). Increasing \(m\) adds positive contributions to this derivative; it does not make the likelihood peak at a finite slope. Recruiting additional events at lower \(x\), or non-events at higher \(x\), would more directly challenge the pattern, provided those observations are sampled from a design appropriate to the scientific question.
Planning Recruitment Across Predictor Ranges
A useful design review starts by identifying ranges or combinations of predictors with only events or only non-events. For a continuous predictor, divide its range into scientifically meaningful intervals for planning and tabulate both outcomes in each interval. This does not mean that binning should replace the continuous predictor in the fitted model. It is a practical way to reveal where outcome support is missing. For factors, examine outcome counts within levels and relevant combinations of predictors.
In a GLMM, the review should also account for clusters. If all events occur in one set of clinics and all non-events in another, pooled outcome overlap may conceal a cluster-specific pattern. Where feasible, recruitment should span multiple clusters and relevant predictor values within clusters. This can improve the information available about fixed effects and can help separate a predictor association from differences between clusters. It does not guarantee finite estimates: other predictors, interactions, random slopes, or unobserved combinations can still separate the data.
Worked Example: Checking Outcome Coverage by Predictor Range
A hypothetical multiclinic study is planning recruitment for a binary outcome and a continuous exposure. For design review, the exposure is divided into low, middle, and high ranges. Current counts are: low, 0 events and 12 non-events; middle, 4 events and 4 non-events; high, 12 events and 0 non-events. The pattern is strongly aligned with exposure, although the middle range has both outcomes.
The current number of ranges containing both outcomes is 1 of 3, so the observed two-outcome coverage is \(1/3=0.333\), or 33.3%. The study proposes recruiting six additional participants in the low range and six in the high range, with three events and three non-events in each range. The resulting counts would be low: 3 events and 15 non-events; middle: 4 and 4; high: 15 events and 3 non-events.
After the planned recruitment, all three ranges contain both outcomes, so the coverage is \(3/3=1\), or 100%. The data now include outcomes that oppose the original extreme pattern in both tails. This is a more targeted design improvement than merely enlarging the middle range. In practice, the allocation must be feasible under the outcome-generating and follow-up process; the table is a planning target, not a guarantee that a prescribed number of events will occur.
Because this is a multiclinic GLMM, the team should also check that these outcomes are distributed across clinics rather than concentrated in one or two. Coverage in a pooled table alone does not establish within-clinic overlap or rule out separation in a random-slope direction.
Recruiting Events, Sampling Design, and the Target Model
Recruiting more events can be useful when events are scarce and their absence leaves important predictor regions unsupported. The goal is not to achieve an arbitrary event total, but to obtain events over predictor ranges and clusters where the model needs information. If event recruitment is based on outcome status, however, the sampling mechanism must be considered before interpreting the fitted model.
For example, in ordinary logistic regression, sampling that depends only on outcome status can preserve certain slope relationships under standard case-control assumptions, while changing the intercept and the population event probability. A GLMM has additional complications: cluster sampling, random-effect distributions, and the relation between selection and covariates can affect what likelihood is appropriate. Outcome-dependent recruitment should therefore be reflected in the analysis design, rather than treated as if it were an ordinary random sample. A prospective design that samples participants across predictor ranges and observes their outcomes can often make the target population and sampling process clearer.
Worked Example: Distributing Recruitment Across Clinics
A hypothetical study has six clinics. At a high exposure range, the currently observed outcomes are mostly events; at a low exposure range, they are mostly non-events. The study can enroll 12 additional participants. Consider two plans: Plan A enrolls all 12 from one clinic at the high exposure range; Plan B enrolls two participants in each of six clinics, with the allocation within each clinic guided by the aim of covering low and high exposure ranges and following participants for the binary outcome.
Compare the designs. Plan A adds 12 observations in one already event-heavy setting. Even if several are events, it may add little information about outcomes at low exposure or about whether the pattern recurs across clinics. Plan B distributes observations across all six clinics and deliberately covers both exposure ranges. It gives the design an opportunity to observe outcomes that challenge the current alignment within more than one cluster.
This comparison is about where information may be added, not a claim that Plan B guarantees a particular number of events. The realized outcomes are unknown, and each clinic may still have only one outcome type. Before analysis, the team should tabulate the realized outcome counts by exposure range and clinic, and then check whether the fitted model’s sparse patterns and separation diagnostics have improved. If no opposing outcomes are observed, the design has not achieved its intended remedy.
Diagnostics, Alternatives, and Limitations
After new data are collected, reassess the pattern rather than assuming that recruitment fixed it. Useful checks include:
- Tabulate events and non-events over meaningful predictor ranges and factor levels; identify empty or nearly empty outcome cells.
- Check whether overlap occurs in more than one cluster and, where relevant, over the range of random-slope predictors.
- Revisit score, information, profile-likelihood, and convergence diagnostics already used to distinguish separation from numerical failure.
- Compare the planned recruitment target with the realized data. A target of balanced coverage is not evidence that balanced coverage was achieved.
If the data remain separated or too sparse for stable estimation, the team must report that limitation and consider an analysis suited to the scientific goal. The earlier material on ridge penalization, proper priors, and Firth correction concerns estimation approaches that can regularize or modify estimation when the observed data do not support ordinary finite estimates. These approaches are not substitutes for careful design, and they do not make an unrepresentative sample representative.
An optimizer change is appropriate when the model is numerically difficult but the data support a finite solution. It is not a data-collection remedy: changing starting values, tolerances, or optimization routines cannot supply opposing outcomes. Conversely, seeing a warning does not prove that collecting more data is the only solution; first determine whether the issue is separation, model misspecification, or numerical failure. Ridge penalization, for example, changes the estimation objective, whereas a design remedy changes which observations contribute information.
Several limitations remain. Recruitment may be constrained by event rarity, cost, follow-up, or ethics; investigators should not manipulate exposure or outcome solely to manufacture overlap. Very small numbers of opposing observations may technically disrupt a simple separation pattern yet still leave estimates highly uncertain. Additional observations also cannot correct incorrect covariate measurement, an inappropriate random-effects structure, or selection that changes the target population. Data collection remedies are most effective when planned before recruitment and linked to the estimand and sampling process.
Use outcome-by-predictor and outcome-by-cluster summaries to identify where one outcome is absent or rare.
Plan recruitment that can add relevant opposing outcomes across predictor ranges and clusters, subject to the actual sampling and follow-up process.
Repeat the coverage and separation checks after recruitment. Do not infer success from the planned sample size or event count alone.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study increases its event count from 8 to 24, and every added event lies in the same predictor range as the original events. By what factor did the event count change, and does that fact alone show that separation is resolved?
- In a predictor range, there are 0 events and 15 non-events. What kind of observation would more directly challenge this local outcome pattern?
- A pooled table shows both outcomes at each exposure range, but one clinic has only events and another only non-events. Why is the pooled table insufficient for assessing a clustered design?
- Why can an optimizer change help with convergence failure but not repair a separating data pattern?
Solutions. (1) The count tripled because \(24/8=3\). This does not show that separation is resolved; if all new events reinforce the existing pattern, the missing opposing outcome remains. (2) An event observed in that predictor range would oppose the local pattern, provided it is obtained through a sampling process appropriate to the study. (3) The pooled table can hide cluster-specific separation or a lack of within-clinic outcome overlap; examine outcomes by both predictor range and clinic. (4) An optimizer changes how the likelihood is numerically searched, not which outcomes and covariate values were observed. It cannot add information that is absent from the data.