Empty Cells in Cross-Classified Designs

Why Empty Combinations Need Careful Interpretation

In a cross-classified design, each observation belongs to a level of one factor and a level of another, but neither factor is nested within the other. Examples include responses from subjects to items, or ratings from raters of targets. Not every subject–item or rater–target combination must occur. A missing combination may simply be unobserved; it does not by itself invalidate a model. The important question is whether the combinations that are observed provide enough information to distinguish the contributions of the crossed factors.

This distinction matters because a model can include two random effects in its formula while the observed data contain little or no information for estimating their variance components separately. The issue is not the percentage of empty cells alone. It is the pattern of observed sharing: which observations share a level of the first factor, which share a level of the second, and whether those patterns differ in informative ways.

Model and Information in the Observed Combinations

Definition: For observation $i=1,\ldots,n$, let $a(i)$ and $b(i)$ denote its levels of two crossed factors. A cross-classified random-intercept GLMM has conditional mean $\mu_i=E(Y_i\mid u,v)$ satisfying $$ g(\mu_i)=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}+u_{a(i)}+v_{b(i)}. $$ Here $g$ is the link function, $\mathbf{x}_i$ is the fixed-effect design vector, and the random intercepts are commonly modeled as $u_a\sim N(0,\tau_A^2)$ and $v_b\sim N(0,\tau_B^2)$, mutually independent across levels and between factors. Conditional on all random effects and the fixed predictors, responses are assumed independent and follow the chosen GLMM response distribution.

The model uses only observed records. An empty factor combination contributes no response and therefore no direct likelihood contribution; it is not a response of zero. For example, an absent subject–item pair is not evidence that the subject gave a zero score for that item. The random effects for its subject and item may still be informed by other observed combinations involving those levels.

A useful exact criterion can be stated for a Gaussian random-intercept model. Let $Z_A$ and $Z_B$ be the incidence matrices that indicate the observed levels of the two factors, and suppose the residual errors have variance $\sigma^2$ and are independent of the random effects. The marginal covariance matrix is $$ V=\tau_A^2 Z_A Z_A^\mathsf{T}+\tau_B^2 Z_B Z_B^\mathsf{T}+\sigma^2 I. $$ The two random-effect variances are structurally distinguishable from this covariance model only if the covariance patterns they generate can be told apart after accounting for the residual pattern. In particular, if $Z_A Z_A^\mathsf{T}$ and $Z_B Z_B^\mathsf{T}$ are identical on the observed rows, the data covariance depends on $\tau_A^2+\tau_B^2$, not on the two components separately. In general, linear dependence among the component covariance patterns prevents their separate identification; near-dependence makes estimation unstable.

This is a structural statement about what the observed design can distinguish under the model. A formally distinguishable model may still yield imprecise estimates in a small sample. In non-Gaussian GLMMs the marginal response covariance is not generally given by that Gaussian formula, but the same practical concern remains: the likelihood must contain information that differentiates the effects. Estimates, uncertainty, and likelihood curvature should therefore be examined rather than inferred from the formula alone.

Worked Example: Empty Pairs That Do Not Remove the Sharing Patterns

Suppose a hypothetical Gaussian study records two responses in each observed subject–item combination. There are three subjects and three items, and the observed combinations are (S1,I1), (S1,I2), (S2,I2), (S2,I3), (S3,I1), and (S3,I3). The other three combinations are empty. Assume the random-intercept model above, with independent residual errors.

Even with those empty combinations, each subject appears with two items, and each item appears with two subjects. Thus the observed data include pairs of observations that share a subject but not an item, and pairs that share an item but not a subject. Under the Gaussian model, their covariances have different sources:

\begin{aligned} \operatorname{Cov}(Y_i,Y_j)&=\tau_A^2 &&\text{if }a(i)=a(j),\ b(i)\ne b(j),\\ \operatorname{Cov}(Y_i,Y_j)&=\tau_B^2 &&\text{if }b(i)=b(j),\ a(i)\ne a(j),\\ \operatorname{Cov}(Y_i,Y_j)&=\tau_A^2+\tau_B^2 &&\text{if }a(i)=a(j),\ b(i)=b(j),\ i\ne j. \end{aligned}

For instance, two responses in (S1,I1) share both random intercepts, so their covariance is $\tau_A^2+\tau_B^2$. Responses from (S1,I1) and (S1,I2) share S1 but not an item, so their covariance is $\tau_A^2$. The empty (S1,I3) cell does not remove either kind of observed comparison.

To illustrate the separation numerically, suppose hypothetical covariance summaries from the observed design were 0.36 response-units squared for pairs sharing a subject only, and 0.16 response-units squared for pairs sharing an item only. The corresponding component estimates by these summaries are $\hat{\tau}_A^2=0.36$ and $\hat{\tau}_B^2=0.16$. The implied covariance for two responses in the same observed cell is $0.36+0.16=0.52$ response-units squared. These figures are illustrative covariance summaries, not a claim that such summaries are reliable estimates in every data set. Their purpose is to show why the two types of observed sharing carry distinct information.

The design therefore has missing combinations but retains distinct covariance patterns. A fitted model would still require checks of uncertainty and fit; the mere presence of both comparison types does not guarantee precise estimation.

When the Observed Data Cannot Separate the Components

Worked Example: Missing Combinations That Confound Two Variances

Consider a hypothetical Gaussian design with two subjects and two items, but only the combinations (S1,I1) and (S2,I2) are observed. Suppose there is one response in each observed combination. The other two combinations are empty. Each observation has its own subject level and its own item level; no two observations share either factor level.

Consequently, $Z_A Z_A^\mathsf{T}=I$ and $Z_B Z_B^\mathsf{T}=I$ on the two observed rows. With independent residual errors, the marginal covariance is $$ V=\tau_A^2 I+\tau_B^2 I+\sigma^2 I =(\tau_A^2+\tau_B^2+\sigma^2)I. $$

The arithmetic shows the problem: the data covariance contains the sum of all three variances, with no observed pair covariance that isolates a subject or item contribution. For example, $(\tau_A^2,\tau_B^2)=(0.4,0.2)$ and $(0.2,0.4)$, in response-units squared, give the same covariance matrix if $\sigma^2$ is unchanged. The likelihood cannot use these observations to decide which factor accounts for more variance. A software fit might return values, but those values do not establish that the separate components have been learned from the data.

This is an exact lack of separation in this design under the stated model, not merely a small-sample warning. More observations in the same two combinations would improve knowledge of their combined variability but would not create observations sharing just one of the factors. To distinguish the components, the observed design would need additional combinations that generate different sharing patterns.

Diagnosing Weak Separation in Sparse Designs

Between the well-differentiated design and the exactly confounded design lies a common practical case: the components are technically distinguishable, but only a few observations contribute to the patterns that separate them. Empty cells matter in this case because they may leave too few informative comparisons, even when each factor level appears several times.

Count observed combinations and replication.
Make a table of observed counts by factor combination. Separate an empty cell from a cell with one observation: a single observation contributes to the model, but it cannot provide a within-cell pair for estimating shared covariance.

Check the observed sharing patterns.
Ask whether there are repeated observations sharing only factor A, repeated observations sharing only factor B, and, where useful, repeated observations sharing both. Do not use the number of empty cells as a substitute for this check.

Inspect estimation and uncertainty.
Look for large standard errors, unstable estimates across reasonable fitting choices, boundary estimates, or a nearly singular estimated covariance structure. These are warnings of weak information, not proof by themselves that a component is structurally unidentified.

Compare what the model predicts with the observed response patterns.
Check whether the fitted model captures variation among levels of each factor and whether residual behavior is compatible with the response distribution and conditional-independence assumptions. A sparse design can make these checks inconclusive as well as make estimation unstable.

For Gaussian models, the covariance-pattern expression gives a direct design-based check. For other GLMMs, numerical likelihood diagnostics are especially important because the relationship between random-effect variances and observed response dependence depends on the link and response distribution. A flat profile likelihood or strong sensitivity to small data changes suggests weak separation. No single diagnostic turns sparse data into informative data.

Worked Example: Quantifying a Separation Check

Suppose a hypothetical Gaussian design has observed pairs that share only a subject, with estimated covariance 0.30 response-units squared, and pairs that share only an item, with estimated covariance 0.12 response-units squared. Suppose the estimated covariance for two distinct responses in the same observed cell is 0.48 response-units squared. Under the model, these patterns suggest $\hat{\tau}_A^2=0.30$ and $\hat{\tau}_B^2=0.12$. The implied same-cell covariance is $$ \hat{\tau}_A^2+\hat{\tau}_B^2=0.30+0.12=0.42 \quad\text{response-units squared}. $$

The observed same-cell summary, 0.48, differs from this simple implied value by $0.48-0.42=0.06$ response-units squared. This discrepancy is not, on its own, a formal test or proof of model failure: the summaries are estimated from finite data and have sampling variability. It does show why all three patterns can be informative. The first two distinguish the component variances, while the third provides a check on the additive covariance structure. If the design had no pairs sharing only a subject or only an item, this calculation could not separately estimate the two components in this way.

Modeling Responses and Common Pitfalls

The appropriate response to empty cells depends on the scientific target and on what information remains in the observed combinations. In a random-effects model, levels with observations elsewhere can still contribute information even if some of their combinations are absent. In a fixed-effects model for every factor combination, some contrasts involving unobserved cells may not be estimable from the data. Switching to fixed effects is therefore not an automatic cure; it changes the model and the quantities being estimated.

Other options include collecting observations in combinations that distinguish the factors, simplifying the random-effects structure when a component cannot be supported, or using a scientifically justified alternative model. A Bayesian prior can regularize estimation, but it does not add observed comparisons: when the likelihood cannot distinguish two components, the posterior distinction may depend strongly on the prior. Any such choice should be reported as a limitation of the available design.

Mistake: “There are empty cells, so the crossed model cannot be fitted.” Correction: empty cells are not automatically a problem. Check whether the observed records still provide informative sharing patterns and whether the fitted components are estimated with useful precision.
Mistake: “The model includes two random effects, so their variances must be separately estimable.” Correction: a formula specifies a model; it does not guarantee that the observed design distinguishes its parameters.
Mistake: “An empty cell is a zero response.” Correction: it is an unobserved combination. Treating it as a zero changes the data and can distort the likelihood.
Mistake: “A boundary estimate proves that the corresponding factor has no variability.” Correction: a boundary estimate can reflect limited information or model issues. Interpret it alongside uncertainty, design structure, and fit.

The central limitation is that no estimation method can recover a component contrast that the observed design does not encode. Conversely, an incomplete table does not imply failure when the remaining observations preserve the necessary distinctions. Assess the design first, then interpret the fitted model in light of that assessment.

Check Your Understanding

Answer each question, then compare with the solutions below.

In a crossed Gaussian random-intercept model, what covariance is expected between two different observations sharing a subject but not an item?
Two observations each come from a different subject and a different item, with no repeated levels across the records. What does their covariance tell you about separating the two random-intercept variances?
A combination is empty, but its subject and item both appear in other observed combinations. Is the missing combination itself evidence of a zero response?
Why can a prior stabilize estimates in a weakly separated model without adding information from the observed design?

Solutions. (1) The covariance is $\tau_A^2$, because the observations share the subject random intercept and do not share an item intercept. (2) Under independent random effects and residuals, their covariance is zero. That pair supplies no shared-effect covariance for either component; separation must come from other observed pairs with different sharing patterns. If no such patterns occur, the two variances may not be separately identifiable. (3) No. The combination contributes no response and is not a zero; the subject and item may still be informed by their other observed records. (4) A prior contributes external distributional assumptions to the posterior. It can regularize a flat or weak likelihood, but it does not create observed comparisons that distinguish the components, so the posterior separation can depend strongly on the prior.