Raters Crossed With Targets

Why Separate Rater and Target Variation?

A rating reflects more than the target being assessed. Some raters tend to give higher scores than others, some targets tend to receive higher scores, and individual ratings still vary for reasons not represented by either tendency. A model that treats every rating as independent can miss the dependence created when the same rater assesses several targets or the same target receives several ratings.

Raters and targets are crossed when raters assess multiple targets and targets receive ratings from multiple raters. This is different from a nested arrangement in which each target belongs to just one rater, or each rater assesses only one target. The crossed structure gives the model a way to distinguish a rater's general scoring tendency from a target's general tendency to receive high or low ratings.

Model and Assumptions

For a continuous rating analyzed with a Gaussian model, a common random-intercept specification is

Here \(\beta_0\) is the population-average intercept, \(a_i\) is the deviation associated with rater \(i\), \(b_j\) is the deviation associated with target \(j\), and \(\varepsilon_{ij}\) is the remaining rating-level deviation. The usual model assumes that these three sets of terms are mutually independent, and that effects within each set are independent draws from the stated distributions. The model also assumes that the conditional mean structure is adequate and that, given the effects, rating-level errors are independent with constant variance.

The Gaussian model is appropriate when treating the rating scale as approximately continuous is defensible. For a non-Gaussian response, the generalized mixed model specifies a conditional response distribution and a link. In general form,

The predictor \(x_{ij}^{\mathsf T}\beta\) can include measured covariates, while \(g\) is the link appropriate to the response distribution. For example, a binary rating can use a Bernoulli distribution with a logit link. The rater and target effects then act additively on the log-odds scale, not directly on the probability scale.

The random-effects interpretation assumes the raters and targets can be regarded as draws from populations of interest. If the analysis is specifically about a fixed, named set of raters, and no inference to a wider rater population is intended, rater indicators as fixed effects may be more appropriate. The choice is scientific as well as statistical: it determines what the rater effects represent.

What the Components Explain

In the Gaussian model, \(\sigma_a^2\) describes variation among raters in their overall scoring tendency; \(\sigma_b^2\) describes variation among targets in their overall ratings; and \(\sigma^2\) describes remaining rating-level variation. These are variances in squared rating units. For example, if ratings are recorded in points, each variance is in points squared.

The model's covariance pattern follows directly from its shared terms. Two different ratings from the same rater share \(a_i\). Two different ratings of the same target share \(b_j\). If they share neither, they share neither random intercept. To see this, take two distinct observations and use independence and the zero means of the components:

The indicator \(\mathbf{1}(i=i')\) equals 1 when the two observations share a rater and 0 otherwise; the target indicator works the same way. For distinct observations, the residual terms do not contribute to their covariance under the model. A rating's variance is the sum of all three components, \(\sigma_a^2+\sigma_b^2+\sigma^2\).

Generalized Ratings: A Binary Example

For binary responses, the conditional response variation is described by the Bernoulli distribution. There is no separate additive Gaussian residual \(\varepsilon_{ij}\) in the usual logistic random-intercept specification. Instead, conditional on the random effects, \(Y_{ij}\) is Bernoulli with probability \(p_{ij}\), and its conditional variance is \(p_{ij}(1-p_{ij})\). Rater and target variation is modeled on the link scale; the Bernoulli variation is on the response scale.

Reading the Design Before Fitting

A rating table should be examined as a rater-by-target membership structure. In a complete crossing, each rater assesses every target. In an incomplete crossing, some pairs are absent, but raters can still assess multiple targets and targets can still receive ratings from multiple raters. The model can use such data if the observed connections provide information about both classifications. If targets are rated by only one rater, target differences cannot in general be separated cleanly from that rater's tendency without additional design information or assumptions.

Diagnostics, Alternatives, and Limitations

The model is only useful if its structure and assumptions are plausible. Check that the recorded membership pattern matches the scientific process: a rater effect should represent a tendency that recurs across the targets that rater scores, and a target effect should recur across its raters. For a Gaussian fit, inspect residual patterns and whether the assumed spread is plausible across fitted values and groups. Inspect the estimated random-effect distributions for marked departures from the assumed normal shape, while recognizing that a small number of raters or targets makes such checks weak.

Warning signs include convergence problems, estimates at or near zero for a variance component, and instability when a few raters or targets are removed. These can reflect a weakly informative design, too few levels to estimate a population variance reliably, or a model that asks the data to separate effects they cannot distinguish. For generalized models, also check whether the chosen conditional response distribution and link are sensible for the rating outcome. A Gaussian residual diagnostic does not apply to a Bernoulli model in the same way, because its conditional response variation is specified by the Bernoulli distribution.

The crossed random-intercept model is not the only reasonable approach. Treating raters as fixed effects is useful when inference concerns only the named raters; it does not estimate a rater-population variance in the same way. A model with only target effects ignores recurring rater tendencies, while a model with only rater effects ignores recurring target differences. A nested model is suitable only when the actual assignment structure is nested, not merely because the data table is inconveniently incomplete.

The simple model also has limits. It assumes each rater has one general additive tendency across targets and each target has one general additive tendency across raters. If a rater behaves differently for different kinds of targets, or if a particular rater-target pairing has a persistent special effect, the random-intercept model may not capture that structure. With only one rating per rater-target pair, a distinct rater-by-target interaction cannot be separated from rating-level residual variation without further assumptions or replication. Model choice should therefore follow the rating process, not just a desire to include more random effects.