A Crossed Participant-Item Experiment in R

From Experimental Design to Model Formula

In a participant-item experiment, each participant responds to multiple items, and each item is answered by multiple participants. The observations therefore share information in two directions: responses from the same participant may resemble one another, and responses to the same item may resemble one another. A logistic generalized linear mixed model (GLMM) can represent both sources of dependence while estimating an experimental condition effect.

The practical challenge is to make the data structure agree with the design. Each row should represent one observed binary response, and the participant and item columns must identify the actual experimental units that responses share. A formula with two random-intercept terms is appropriate only when those identifiers genuinely describe crossed sources of variation.

Model definition: Let $Y_{pi}$ be the binary response for participant $p$ on item $i$, with $Y_{pi}=1$ for a success and $Y_{pi}=0$ otherwise. A crossed random-intercept logistic GLMM is $$ Y_{pi}\mid b_p,u_i \sim \operatorname{Bernoulli}(\pi_{pi}), \qquad \operatorname{logit}(\pi_{pi})=\beta_0+\beta_1 x_{pi}+b_p+u_i, $$ where $x_{pi}$ is a fixed-effect predictor, $b_p\sim N(0,\sigma^2_{\text{participant}})$ is the participant random intercept, and $u_i\sim N(0,\sigma^2_{\text{item}})$ is the item random intercept. The random effects are assumed independent of one another and of the fixed predictors. Conditional on the random effects and predictors, responses are independent Bernoulli observations.

The logit is $\operatorname{logit}(\pi)=\log\{\pi/(1-\pi)\}$, so the model describes conditional log odds: the condition effect compares odds while holding the participant and item random intercepts fixed. The assumptions do not require all observed responses to be marginally independent. Instead, responses sharing a participant or an item can be associated through their shared random intercept.

“Crossed” describes the design, not just the appearance of a formula. Participant levels recur across items, and item levels recur across participants. In contrast, if every item belongs to only one participant, items are nested within participants. The same item identifier should be used for every response to that actual item, even when different participants answered it.

Representing the Observations in R

For trial-level binary data, use one row per response. At minimum, the data frame needs a binary outcome, a participant identifier, an item identifier, and any fixed-effect predictors. If there are multiple trials for the same participant-item pair, those trials are separate rows, provided each is a distinct observed response and the model’s conditional-independence assumption is reasonable.

Worked Example: Checking the Crossed Row Structure

Suppose a hypothetical experiment records whether each response is correct. Participant P01 answers items I01 and I02; participant P02 also answers I01 and I02. The following four records show the essential structure.

participant	item	condition	correct
P01	I01	control	1
P01	I02	treatment	0
P02	I01	treatment	1
P02	I02	control	1

There is one row per response, not one row per participant or per item. P01 appears on two rows, as does P02; I01 and I02 also recur across participants. Thus both participant and item are shared grouping factors. The response column contains only 0 and 1, making the individual-trial Bernoulli specification appropriate.

In R, store the columns as variables named participant, item, condition, and correct. Check that each row is an observation and that IDs are not missing. The four records are only a structural illustration: four responses are not enough for reliable estimation of this model. A real analysis needs enough participants, items, and responses to estimate the fixed effect and both variance components.

In R’s lme4 package, the formula translates the design into the conditional mean model. The term $(1\mid participant)$ adds a participant-specific random intercept; $(1\mid item)$ adds an item-specific random intercept. Their separate terms express crossed grouping factors. The response family specifies the binary distribution and logit link.

R specification: Set the control condition as the reference level before fitting. Then fit glmer(correct ~ condition + (1 | participant) + (1 | item), data = dat, family = binomial(link = "logit")). The fixed-effect coefficient for treatment is the treatment-versus-control conditional log odds ratio.

Before fitting, verify that the outcome is coded as intended, the condition labels map to the intended contrast, and the participant and item identifiers are consistent across the data. For example, if an item is represented as “1” separately within each participant even though those labels refer to different stimuli, the item column incorrectly declares them to be the same item. Conversely, if the same stimulus is assigned different IDs for different participants, the model cannot recognize its shared item effect.

Fitting and Interpreting the Condition Effect

A typical fit in R can be written as follows. The reference level makes the coefficient interpretation explicit; with treatment contrasts, the intercept then refers to the control condition for random effects equal to zero.

R: library(lme4); dat$condition <- relevel(factor(dat$condition), ref = "control"); m <- glmer(correct ~ condition + (1 | participant) + (1 | item), data = dat, family = binomial(link = "logit"))

The random intercepts shift the log odds for participants and items relative to the model’s overall intercept. The condition coefficient is not an odds ratio until exponentiated. Nor is it a population-averaged effect: it is conditional on the random effects. In particular, $\exp(\beta_1)$ is the multiplicative change in the odds of success for treatment versus control, holding participant and item effects fixed.

Worked Example: Converting the Condition Coefficient to Odds

Suppose a hypothetical fitted model has a treatment coefficient of $0.70$. Consider a participant and item whose estimated random intercepts are $b_p=0.20$ and $u_i=-0.10$, and an intercept of $\beta_0=-0.40$. These values illustrate how to use a fitted model; they are not claimed as estimates from the four-row table above.

For control, $x_{pi}=0$, so the linear predictor is $-0.40+0+0.20-0.10=-0.30$. The corresponding conditional success probability is $$ \pi_{\text{control}}=\frac{\exp(-0.30)}{1+\exp(-0.30)}\approx 0.4256. $$

For treatment, $x_{pi}=1$, so the linear predictor is $-0.40+0.70+0.20-0.10=0.40$, giving $$ \pi_{\text{treatment}}=\frac{\exp(0.40)}{1+\exp(0.40)}\approx 0.5987. $$

The conditional odds ratio is $\exp(0.70)\approx 2.014$. Thus, for the same participant and item random intercepts, the treatment condition has about 2.01 times the odds of success as control under this illustrative model. The conditional probability difference here is about $0.5987-0.4256=0.1731$, or 17.31 percentage points. That probability difference depends on the chosen random effects; it is not a constant implied by the coefficient alone.

Why the Two Random Intercepts Matter

The two random-intercept terms represent distinct forms of clustering. A participant who tends to succeed more often than average contributes a shift shared across that participant’s responses. An item that tends to be easier or harder than average contributes a shift shared across responses to that item. Omitting either term can leave the corresponding dependence unrepresented and can make uncertainty for fixed effects misleading.

The rationale can be seen directly from the linear predictor. Two different responses from participant $p$ share $b_p$; two responses to item $i$ share $u_i$. Thus the model assigns a common latent shift to records sharing either unit. The standard random-intercept model assumes these shifts are normally distributed with mean zero, and estimates their variances from the pattern of observations across units.

Worked Example: Diagnosing an Item-Identifier Error

Imagine a hypothetical experiment in which every participant sees the same 12 images, and the analysis data contain an item column numbered 1 through 12. The same image is always given the same number. Since each image recurs across participants, the formula should include $(1\mid item)$, in addition to the participant intercept.

Now suppose instead that each participant sees a different set of 12 images, and the item labels 1 through 12 are reused locally. Label “1” for P01 and label “1” for P02 refer to different images. If both are entered as item = 1, the model treats distinct stimuli as one shared item and imposes a false dependence structure. The correction is to create globally unique identifiers for the actual images, such as P01_I01 and P02_I01, if the stimuli are truly participant-specific.

If those items really are distinct and nested within participants, then the design is not crossed for items. A nested structure can be represented, when justified by the scientific design, using an item-within-participant grouping factor such as $(1\mid participant:item)$. In the original crossed-image design, that interaction-style identifier would instead incorrectly split each shared image into participant-specific copies. Identifiers must therefore follow the experimental units, not merely the labels used in a spreadsheet.

Diagnostics, Alternatives, and Limits

After fitting, inspect the fixed-effect estimates, random-effect variances, convergence messages, and fitted model structure. In lme4, summary(m) displays estimates and standard errors, VarCorr(m) displays the random-effect variance estimates, and isSingular(m) checks for a singular fit. A singular fit means at least one estimated random-effect variance is on or near the boundary at zero; it is not automatically proof of a coding error, but it can indicate that the data do not support the full random-effects structure well.

A convergence warning should not be dismissed simply because the printed coefficients look plausible. Check that the model matrix and grouping identifiers are correct, that the data contain variation in the binary outcome across the relevant design, and that the fitted structure is supported by the number of participants and items. Complete or near-complete separation can also make fixed-effect estimates unstable or very large. A model cannot reliably estimate variation for a grouping factor with too few levels or insufficient replication.

The principal alternatives answer different questions or impose different dependence structures:

Ordinary logistic regression omits both random intercepts. It is simpler, but treats responses as conditionally independent given fixed predictors and does not represent participant or item clustering.
A participant-only random-intercept model captures participant clustering but not shared item difficulty. It may be suitable only if item-level dependence is absent or outside the target model.
A nested model is appropriate when item units are genuinely unique to participants, rather than shared across them. Using a nested structure for shared stimuli changes the assumed data-generating structure.
A population-averaged approach can target marginal associations rather than participant- and item-conditional effects. It is not interchangeable with the conditional GLMM coefficient.

The random-intercept model is also a simplification. It assumes participants and items differ in baseline log odds but have the same condition effect. If the experimental design and data support variation in the condition effect, a random-slope structure may be scientifically relevant, but this tutorial’s formula does not include one. More complex structures require adequate replication and careful assessment; adding terms mechanically does not guarantee a better model.

Key takeaway: For a crossed participant-item binary experiment, use one row per response, stable identifiers for each actual participant and item, and separate random-intercept terms for both factors. Confirm that the formula matches the design before interpreting its conditional treatment coefficient.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions.

In a crossed design, each participant answers several items and each item is answered by several participants. What grouping terms represent random intercepts for both sources?
A treatment coefficient from a logistic GLMM is $0.50$. What is the conditional odds ratio, approximately, and what does it compare?
Each participant sees different items, but item labels 1 through 10 are reused for every participant. What is wrong with using the reused label as a crossed item identifier?
What does a singular-fit check help identify, and does a singular fit by itself prove that the data were entered incorrectly?

Solutions. (1) Use $(1\mid participant)+(1\mid item)$, assuming the identifiers refer to genuinely shared participants and items and the random-intercept model is appropriate. (2) $\exp(0.50)\approx 1.649$. It compares treatment with the reference condition in conditional odds, holding the random effects fixed. (3) The same label is falsely treated as the same item across participants. Give each distinct stimulus a unique identifier; if items are truly participant-specific, represent the nested design rather than a crossed item factor. (4) It identifies whether estimated random-effect variances are on or near the boundary at zero. That can signal limited support for the specified random-effects structure, but it does not by itself establish a data-entry error.