Common Mistakes in Fitting Crossed Effects as Nested

Why false nesting matters

Many GLMM data sets have observations classified by more than one grouping factor. A response may come from a particular subject and a particular item, such as a participant answering a question, a clinician assessing a case, or a laboratory evaluating a sample. If subjects recur across items and items recur across subjects, the factors are crossed. Treating one factor as nested within the other can make observations that share a real subject or item appear to come from unrelated groups. That changes the dependence the model represents, not merely the way the data are labelled.

The practical consequence is that the fitted model may miss a source of association, assign variation to the wrong grouping level, or calculate uncertainty as if there were more independent information than the design provides. The key question is not whether a factor's labels can be written inside another factor's labels. It is whether each level of one factor occurs with levels of the other factor in the actual sampling structure.

Definition: Two factors are crossed when levels of each factor occur with levels of the other; they need not occur in every possible combination. Factor A is nested within factor B when each level of A belongs to only one level of B, so an A level cannot recur across B levels. A GLMM that treats crossed factors as nested assigns separate grouping identities to combinations or occurrences that should share a subject effect, an item effect, or both.

For example, a set of participants may each respond to several test items, while each test item is answered by several participants. Participants and items are crossed even if some participant–item combinations are missing. Missing combinations or disconnected portions of the membership structure do not, by themselves, make a recurring item belong to only one participant. The earlier material on empty cells and disconnected graphs is useful for describing which combinations are observed; the nesting decision still depends on actual membership.

What dependence does a crossed model represent?

Consider a Gaussian identity-link model as a transparent way to see the grouping structure. For observation $r$, let $s(r)$ denote its subject and $j(r)$ its item. Suppose

Y_r = \boldsymbol{x}_r^\top\boldsymbol{\beta} + b_{s(r)} + u_{j(r)} + \varepsilon_r, \qquad b_s \sim N(0,\tau_s^2),\quad u_j \sim N(0,\tau_j^2),\quad \varepsilon_r \sim N(0,\sigma^2).

Assume that the subject effects, item effects, and residuals are mutually independent; effects within each factor are independent across levels; and observations are conditionally independent given the random effects. The fixed-effects part and the random-effects distributions are also assumed to be adequately specified. Under these assumptions, two distinct responses have covariance from whichever random effects they share.

Result: For two distinct observations $r$ and $q$ in this Gaussian random-intercept model, $$ \operatorname{Cov}(Y_r,Y_q) = I\{s(r)=s(q)\}\tau_s^2 + I\{j(r)=j(q)\}\tau_j^2, $$ where $I\{\cdot\}$ equals 1 when its condition holds and 0 otherwise. If they share both subject and item, both terms contribute. The residual variance $\sigma^2$ contributes to the variance of each response, not to covariance between distinct observations.

This covariance result follows by subtracting the fixed means from both responses and expanding the covariance. Independent effects contribute zero covariance; a random effect appearing in both responses contributes its variance. Thus a pair with the same subject but different items has covariance $\tau_s^2$; a pair with the same item but different subjects has covariance $\tau_j^2$; and a pair sharing both has covariance $\tau_s^2+\tau_j^2$. The same sharing logic applies to GLMMs on the linear-predictor scale. For non-Gaussian responses, response-scale covariances generally require integrating over the random effects and are not given by this simple additive formula.

Worked examples: seeing what false nesting removes

Worked Example: One Subject and One Item Are Shared Separately

Suppose a hypothetical Gaussian study has subject variance $\tau_s^2=0.36$, item variance $\tau_j^2=0.16$, and residual variance $\sigma^2=0.48$. Four distinct responses come from these combinations: subject A with items 1 and 2, and subject B with items 1 and 2. Find the marginal covariance and correlation for three pairs.

For responses from A–1 and A–2, the subject is shared but the item is not. Therefore, $\operatorname{Cov}(Y_{\mathrm{A1}},Y_{\mathrm{A2}})=0.36$. Each response has marginal variance $0.36+0.16+0.48=1.00$, so the correlation is $0.36/\sqrt{1.00(1.00)}=0.36$.

For responses from A–1 and B–1, the item is shared but the subject is not. Their covariance is $0.16$, and their correlation is $0.16/1.00=0.16$.

For responses from A–1 and B–2, neither factor is shared. Their covariance and correlation are both zero under this model. A specification that treats each item as unique to its subject would also assign zero covariance to A–1 and B–1, even though those observations share item 1. It has erased the item-level dependence by changing item 1 into two distinct groups.

Worked Example: A Crossed Membership Table Is Not a Hierarchy

A hypothetical experiment records one response for each of these participant–item pairs: P1–I1, P1–I2, P2–I1, P2–I2, and P3–I2. Is item nested within participant? Is participant nested within item?

Item I1 occurs for both P1 and P2, and item I2 occurs for P1, P2, and P3. Therefore, neither item level is exclusive to one participant: item is not nested within participant. Participant P1 occurs with both I1 and I2, and P2 also occurs with both, so participant is not nested within item either. The observed factor structure is crossed, although the P3–I1 cell is empty.

If item were coded as a participant-specific group, the entries P1–I1 and P2–I1 would be assigned different group identities. That coding contradicts the fact that both responses used I1. If participant were coded as an item-specific group, P1’s responses would likewise be split into separate participant identities. The missing P3–I1 response does not alter either conclusion.

Worked Example: The Standard Error Can Be Too Small

Suppose a hypothetical balanced Gaussian study has 8 subjects, each responding to all 4 items, for $8\times4=32$ observations. Assume the fitted crossed random-intercept model has subject variance $0.16$, item variance $0.25$, and residual variance $0.75$. Compare the model-based standard error of the overall mean with an independence calculation that treats all 32 responses as independent with their marginal variance.

Under the crossed model, the grand mean gives each subject a weight of $1/8$, each item a weight of $1/4$, and each residual a weight of $1/32$. Independence of the random effects therefore gives $\operatorname{Var}(\bar{Y})=0.16/8+0.25/4+0.75/32=0.02000+0.06250+0.02344=0.10594$. The standard error is $\sqrt{0.10594}=0.3255$.

The marginal variance of one response is $0.16+0.25+0.75=1.16$. Treating all 32 responses as independent gives $\operatorname{Var}(\bar{Y})=1.16/32=0.03625$, so the standard error is $\sqrt{0.03625}=0.1904$.

In this stipulated calculation, the independence standard error is much smaller because it counts repeated observations on the same subjects and items as though they contributed independent random-effect information. This calculation illustrates the consequence of ignoring both grouping sources; it is not a claim that every incorrectly fitted nested model produces exactly this standard error. The size and direction of the error depend on the design and the fitted model.

How nested specifications distort the model

A nested specification can fail in more than one way. If items are made unique within subjects, then responses from different subjects that used the same item no longer share an item random effect. If subjects are made unique within items, repeated responses from one subject across items no longer share a subject random effect. A specification using a new random effect for each subject–item combination retains only pair-specific grouping: it cannot, by itself, represent the broader sharing of subjects across items or items across subjects.

For crossed random intercepts, the structure is commonly written schematically as a subject random intercept plus an item random intercept, for example $(1\mid\text{subject})+(1\mid\text{item})$ in software using that formula convention. A term for the subject–item combination, such as $(1\mid\text{subject:item})$, defines a different grouping factor. It may be appropriate when there is genuine additional dependence among repeated observations within a combination, but it is not a substitute for the two shared crossed effects.

A fixed-effects model for item can be an alternative when inference concerns a specified finite set of items and there is adequate information to estimate their separate effects. It does not automatically represent a population distribution of item effects or support the same generalization to new items as a random-effects model. The choice between fixed and random effects is a modeling decision; neither option makes a crossed design nested.

Checking a fit and recognizing failure modes

Before fitting, make a membership table or graph with one row per observation and columns for each grouping factor. Check whether each subject identifier and item identifier consistently refers to the same entity across rows. Then ask whether any level of one factor appears with more than one level of the other. This simple audit often catches a problem that a model formula alone can hide.

Check identities.
Confirm that a repeated label means the same subject or item wherever it occurs. Do not create participant-specific item IDs unless the items truly differ by participant.

Check membership.
List which subjects encounter which items. Recurrence of an item across subjects or a subject across items is evidence against nesting in that direction.

Match the random-effects terms to sharing.
For each term, ask which observations are meant to share that random effect. A combination-level term alone cannot create subject-wide or item-wide sharing.

Inspect residual dependence and fit behavior.
Look for remaining patterns among observations sharing a subject or item, and check estimates, uncertainty, convergence, and boundary variances. These checks can reveal a poor fit, but a plausible diagnostic plot does not prove the grouping structure is correct.

Common warning signs include unexpectedly many grouping levels after identifiers have been combined, a random-effect variance estimated at zero after a theoretically shared factor was split into separate groups, or uncertainty that changes sharply when the correct shared factor is added. None is decisive alone: a variance near zero can be genuine, and a model can have convergence problems for reasons unrelated to nesting. Design knowledge and identifier checks are essential.

Common mistake: “Each response has one subject and one item, so item is nested within subject.” This confuses the fact that each row has one value for each factor with a nesting relationship. The correction is to inspect recurrence across rows: if the same item appears for multiple subjects, it is not nested within subject. A response belongs to one subject–item pair, but the factors can still be crossed.

Limits and interpretation

The crossed random-intercept model is not automatically the right model for every crossed design. It assumes the specified random intercepts adequately describe grouping-related heterogeneity; it does not represent subject-specific or item-specific slopes unless those are included. Nor does a random intercept account for every form of within-subject time dependence, spatial structure, or repeated-measure process. Those features require appropriate model terms and assumptions.

In non-Gaussian GLMMs, conditional independence means responses are independent given the random effects and covariates; after integrating out shared random effects, responses can be marginally associated. A mistaken nested structure changes which responses share those latent effects. Thus the core diagnostic question remains: which observations share a subject, which share an item, and which share both? The answer should come from the design, not from convenience in writing the model.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a study, each clinician rates several cases and each case is rated by several clinicians. Are clinician and case nested or crossed? What membership fact decides?
Under the Gaussian crossed random-intercept model, two different responses share a subject but not an item. State their covariance.
A modeller creates a unique item ID for every subject–item encounter. What dependence between different subjects' responses to the same original item is lost?
Why can treating all observations as independent understate the standard error of an overall mean in a fully crossed design?

Solutions. (1) They are crossed: a case recurs across clinicians and clinicians recur across cases. (2) The covariance is $\tau_s^2$, the subject random-intercept variance, because the pair shares the subject effect but not the item effect. (3) The responses no longer share an item random effect, so the model omits their item-level covariance. (4) Responses sharing a subject or item are correlated through the corresponding random effect; counting them as independent overstates the amount of independent information and can make the calculated standard error too small.