Start With Membership, Not Model Fit
A random-effects structure describes which observations share a grouping unit. That structure determines which observations receive a common random contribution to their linear predictors, and therefore which dependence the model can represent. The decision between nested and crossed effects should begin with the dataset’s membership relations: which unit belongs to which other unit, and can a unit appear in more than one combination?
For example, classrooms are ordinarily nested within schools if each classroom belongs to one school during the study. By contrast, students and test items are crossed when students answer items and each item is answered by multiple students. A dataset may contain only a subset of all possible student-item combinations and still have crossed membership. The classification describes the relationship between the units, not whether every possible combination was observed.
A useful GLMM formulation is \(g(\mu_r)=\mathbf{x}_r^\mathsf{T}\boldsymbol{\beta}+\sum_f\mathbf{z}_{rf}^\mathsf{T}\mathbf{b}_f\), where \(Y_r\) is the response for observational unit \(r\), \(\mu_r=E(Y_r\mid\mathbf{b})\), \(g\) is the link, and \(\mathbf{b}_f\) contains the random effects for grouping factor \(f\). The conditional response distribution and link specify the response model; the membership pattern determines which entries of the design vectors \(\mathbf{z}_{rf}\) are shared across observations.
The examples below use random intercepts to make the structural consequences transparent. They assume the response distribution is correctly specified conditional on the random effects, observations are conditionally independent given those effects, and each random intercept has mean zero. For the covariance calculations, distinct factor-specific random effects are assumed independent, with the stated variances. These assumptions are about the model, not evidence that a particular factor is nested or crossed.
Translate Membership Into a Model
For a random-intercept factor, an observation receives the intercept belonging to its unit. If two observations share a unit, they share that random contribution; if they do not, they do not share that contribution. With independent random intercepts, the covariance of the random portions of two linear predictors is the sum of the variances for the units they share. This result follows by expanding the covariance: terms for distinct independent, mean-zero random effects have covariance zero, while a shared effect contributes its variance.
Here \(z_{rj}\) and \(z_{sj}\) indicate whether observations \(r\) and \(s\) share unit \(j\). This is a covariance on the random-effects contribution to the linear predictor. With a nonlinear link, it is not generally the covariance of the observed responses.
Worked Example: Classrooms Nested Within Schools
Suppose a hypothetical study records a binary outcome for students in two schools. School A has classrooms A1 and A2; School B has classrooms B1 and B2. Each student belongs to one classroom and therefore one school. A logit GLMM with random intercepts is
where \(a_s\) is the school intercept and \(c_{k(s)}\) is the intercept for classroom \(k\) within school \(s\). Assume independent mean-zero school and classroom effects with variances \(\tau_S^2=0.25\) and \(\tau_C^2=0.49\). The membership records check the nesting assumption: each listed classroom is assigned to exactly one school.
For two students in classroom A1, both effects are shared. The covariance of their random linear-predictor contributions is \(0.25+0.49=0.74\). For students in A1 and A2, the school effect is shared but the classroom effects are not, so the covariance is \(0.25\). For students in A1 and B1, neither effect is shared, so it is \(0\) under the stated independence assumptions. The residual variation of the binary response is not added to these calculations: they describe covariance in the random part of the logit linear predictor.
For crossed random intercepts, an observation belongs to one unit of each factor, and units of one factor may recur with multiple units of the other. A sparse set of combinations does not change that relationship. It can, however, reduce information about some variance components or make comparisons depend strongly on the observed pattern.
Worked Example: Clinics Crossed With Providers
A hypothetical outpatient dataset records one binary outcome per visit. Its observed visits include: Clinic A with Provider 1; Clinic A with Provider 2; Clinic B with Provider 1; Clinic B with Provider 3. Provider 1 works at both clinics, so providers are not nested within clinics. Clinics and providers are crossed in the observed membership pattern, although not every possible combination is present.
A random-intercept logit model is
Suppose, for illustration, the clinic and provider intercepts are independent, mean zero, and have variances \(\tau_H^2=0.20\) and \(\tau_P^2=0.36\). Two visits at Clinic A by different providers share only the clinic effect, giving covariance \(0.20\) between their random linear-predictor contributions. Visits to Provider 1 at different clinics share only the provider effect, giving covariance \(0.36\). Two visits with both the same clinic and the same provider share both effects, giving \(0.20+0.36=0.56\). These are random-predictor covariances, not observed-outcome covariances.
The membership check is decisive: Provider 1 appears at Clinics A and B. Treating providers as nested within clinics would represent that provider as different clinic-specific units, contrary to the recorded identity.
A Practical Decision Procedure
Before fitting candidate models, construct a membership table or equivalent record with one row per observational unit and one column for each proposed grouping factor. For each factor pair, ask whether a unit of one factor appears with multiple units of the other. Confirm that identifiers refer to the same unit throughout the analysis period. If memberships change over time, specify the period and the unit definition before deciding the structure.
State what each response row represents, and identify the grouping units attached to that row.
If each lower-level unit has exactly one higher-level parent in the analysis, a nested specification is supported. If a lower-level unit occurs under multiple parents, that nesting description is not supported.
If units of both factors recur across combinations, represent their shared membership with crossed effects. Do not require a complete table of all combinations.
For each observation, name the random effects it receives. Check that observations sharing a unit receive the same effect and observations from distinct units do not receive it accidentally.
Check whether the data support estimation of the chosen variance components and whether the fitted model behaves adequately. Fit statistics compare specified models; they do not establish the membership facts.
Worked Example: Why AIC Cannot Validate a False Nesting Claim
Return to the hypothetical clinic-provider records. Assume the visits, outcome, fixed-effects specification, and likelihood basis are identical in two fitted models. The recorded assignments show that Provider 1 works at both clinics, so a model that treats providers as clinic-specific units misdescribes membership. Suppose the fitted output is as follows: the correctly crossed model has log likelihood \(-205\) and \(k=7\) estimated parameters; the incorrectly nested model has log likelihood \(-203\) and \(k=6\). These numbers are illustrative hypothetical output, not estimates from a real study.
Using \(\mathrm{AIC}=-2\ell+2k\), the crossed model has \(-2(-205)+2(7)=410+14=424\). The nested model has \(-2(-203)+2(6)=406+12=418\). AIC is lower by 6 for the nested fit, even though its membership description contradicts the observed provider assignments.
This comparison does not make the nested model structurally correct. AIC summarizes a trade-off between likelihood and parameter count among the candidate probability models; it does not check whether the random-effect labels correspond to the real units. The lower AIC may reflect the fit of a different, misspecified covariance structure. The membership record must be resolved on substantive and design grounds, not by choosing whichever number is smallest.
Diagnostics, Comparisons, and Limits
Several warning signs can reveal a structure that deserves review. A factor treated as nested may have an identifier appearing under multiple proposed parents. Conversely, a supposedly crossed factor may have been assigned unique labels within each parent, concealing a unit that is actually shared. A model matrix or membership listing can expose these inconsistencies before fitting. After fitting, convergence problems, variance estimates near a boundary, or unstable estimates can indicate inadequate information or other model difficulties; none alone proves that the nesting decision is wrong.
The main alternative structures make different claims about shared variation. A nested specification assigns distinct lower-level effects to units within each parent. A crossed specification assigns the same effect to a recurring unit across its multiple combinations. Treating a genuinely crossed factor as nested can split one unit into several artificial units and remove dependence between observations that share the real unit. Treating genuinely nested units as crossed can instead impose sharing across parents where none exists. In each case, the resulting likelihood and uncertainty calculations are for the encoded structure, not for the membership the analyst intended.
If the data contain no repeated use of a factor level across combinations, the observed table may not distinguish a crossed design from a nested one. Design knowledge, sampling records, and the definition of the units then matter. A sparse or disconnected pattern can also limit what is estimable; the earlier material on disconnected graphs in crossed classifications is relevant when the observed combinations form separate components. A good fit statistic cannot create information about unobserved connections.
The random-intercept illustrations also have limits. They presume the chosen grouping units and response model are appropriate and use a simple variance structure. Random slopes, dependence beyond shared intercepts, and misspecified response distributions require additional modeling decisions. Correct membership is necessary for a defensible random-effects structure, but it is not sufficient for an adequate GLMM.
Common Mistake
Check Your Understanding
For each question, decide from the stated membership what structure is supported. Solutions are provided below the questions.
- Each laboratory technician tests samples from several hospitals, and each hospital’s samples are tested by several technicians. Are technician and hospital effects nested or crossed?
- Each school has its own classrooms, and every classroom belongs to exactly one school. What structure does this support?
- A dataset includes only some combinations of students and test items. Each observed item is answered by several students, and each student answers several items. Does the missing set of combinations make the factors nested?
- An incorrectly nested model has a lower AIC than a model matching the recorded membership. What does that establish, and what does it not establish?
Solutions. (1) They are crossed: technicians recur across hospitals and hospitals recur across technicians. (2) Classrooms are nested within schools because each classroom has exactly one school parent. (3) No. The factors are crossed despite incomplete combinations; missing combinations make the observed table sparse, not nested. (4) It establishes only that the nested candidate has the lower AIC among those fitted candidates under the stated comparison. It does not establish correct membership; the recorded assignments still contradict the nested description.