Why Ordinal Residual Displays Need More Than One View
For an ordinal outcome, categories have a natural order, but the distances between them are not generally meaningful. A residual display should therefore reveal whether a fitted model reproduces the distribution across ordered categories, not treat the category labels as measurements on an interval scale. This distinction matters especially when some categories are rare: a model can match the overall direction of responses while missing a particular category or a threshold in the ordering.
The displays in this tutorial compare observed category frequencies with probabilities fitted by an ordinal GLMM. They are diagnostic summaries, not automatic tests of model adequacy. Their interpretation depends on which fitted probabilities are used, how observations are grouped, and how much information is available in sparse categories.
Model Probabilities and Residual Definitions
Let \(Y_{ij}\) be the ordinal response for observation \(i\) in cluster \(j\), taking one of the ordered values \(1,\ldots,K\). Let \(x_{ij}\) denote its covariates and \(b_j\) the cluster random effects. An ordinal GLMM supplies fitted category probabilities \(p_{ijk}(b_j)=P(Y_{ij}=k\mid x_{ij},b_j)\), with \(p_{ijk}(b_j)\geq 0\) and \(\sum_{k=1}^{K}p_{ijk}(b_j)=1\). Its fitted cumulative probability at cutpoint \(q\) is \(F_{ijq}(b_j)=\sum_{k=1}^{q}p_{ijk}(b_j)\).
A diagnostic can use probabilities conditional on estimated cluster effects, or probabilities marginalized over the random-effects distribution. Conditional probabilities ask how well the model describes observations given the estimated effects for their clusters. Marginal probabilities ask how well it describes responses for the population of clusters. Choose the version to match the question, and state which was used; mixing the two can create apparent discrepancies that reflect the prediction target rather than model failure.
Groups \(G\) might be pre-specified covariate strata or bins along a predictor relevant to the model. The fitted proportion is an average of observation-specific predictions, not necessarily the prediction for an “average” observation. Since both observed and fitted proportions sum to one across categories, their residuals sum to zero: \(\sum_k D_{Gk}=0\). A positive residual in one category must be balanced by negative residuals elsewhere.
Worked Example: Comparing Category Proportions
Suppose a hypothetical ordinal GLMM has three categories: low, moderate, and high. For group A, there are 20 observations, with observed counts \((12,6,2)\). The average fitted probabilities are \((0.55,0.30,0.15)\). Calculate the observed and fitted proportions and the category residuals.
The observed proportions are \((12/20,6/20,2/20)=(0.60,0.30,0.10)\). The average fitted probabilities already give the fitted proportions, so \(D_{A}=(0.60-0.55,\ 0.30-0.30,\ 0.10-0.15)=(0.05,0,-0.05)\).
Thus the observed low-category proportion is 0.05 higher than fitted, the moderate-category proportions agree, and the observed high-category proportion is 0.05 lower. The residuals add to zero, as required. These are descriptive differences; with only 20 observations, they do not by themselves establish that the model is misspecified.
For group B, suppose the observed counts are \((5,8,7)\), again out of 20, and average fitted probabilities are \((0.30,0.40,0.30)\). The observed proportions are \((0.25,0.40,0.35)\), so the residuals are \((-0.05,0,0.05)\). The display would show a deficit in the low category and an excess in the high category for group B, the reverse pattern from group A.
Cumulative Residuals Show Where the Ordering Shifts
Category-specific displays isolate each category, but adjacent categories can be sparse and their residuals can fluctuate. Cumulative displays instead ask whether the model reproduces the proportion at or below each ordinal cutpoint. For group \(G\) and cutpoint \(q\), define the observed cumulative proportion as \(O_{G,\leq q}/n_G\), where \(O_{G,\leq q}=\sum_{(i,j)\in G}I(Y_{ij}\leq q)\). Its fitted counterpart is \(\bar F_{Gq}=n_G^{-1}\sum_{(i,j)\in G}F_{ijq}\). Their difference is the cumulative residual \(C_{Gq}=O_{G,\leq q}/n_G-\bar F_{Gq}\).
A positive \(C_{Gq}\) means that more responses fell at or below cutpoint \(q\) than the model fitted; a negative value means fewer did. With \(K\) categories, there are \(K-1\) informative cutpoints. The cumulative residual is zero at the two endpoints \(q=0\) and \(q=K\), because both the observed and fitted cumulative probabilities are respectively zero and one there.
Worked Example: Reading Cumulative Discrepancies
In a hypothetical group of 12 observations with three ordered categories, the observed counts are \((7,3,2)\). Suppose the average fitted category probabilities are \((0.50,0.25,0.25)\). Calculate the cumulative residuals at both cutpoints.
At the first cutpoint, the observed proportion at or below category 1 is \(7/12=0.5833\), while the fitted cumulative probability is \(0.50\). Thus \(C_{G1}=0.5833-0.50=0.0833\).
At the second cutpoint, the observed proportion at or below category 2 is \((7+3)/12=0.8333\), while the fitted cumulative probability is \(0.50+0.25=0.75\). Thus \(C_{G2}=0.8333-0.75=0.0833\).
The first category has a count residual of \(7-12(0.50)=1\); the second has \(3-12(0.25)=0\); and the third has \(2-12(0.25)=-1\). In proportion units these are \(1/12,0,-1/12\). The identity \(D_{Gk}=C_{Gk}-C_{G,k-1}\) gives \(D_{G2}=0.0833-0.0833=0\) and \(D_{G3}=0-0.0833=-0.0833\), matching the category residuals. Both cutpoints show an excess of lower responses overall, while the category display locates the deficit in category 3.
Randomized Quantile Residuals for Individual Outcomes
A category residual summarizes a group, while a randomized quantile residual assigns a continuous residual to an individual discrete outcome. For an observation with fitted cumulative distribution \(F_{ij}\), let \(F_{ij}(y^-)=P(Y_{ij}<y)\) and \(F_{ij}(y)=P(Y_{ij}\leq y)\). Draw \(U_{ij}\) uniformly from the interval \((F_{ij}(Y_{ij}^-),F_{ij}(Y_{ij}))\), then set \(r_{ij}=\Phi^{-1}(U_{ij})\), where \(\Phi^{-1}\) is the standard normal quantile function. The interval has width equal to the fitted probability of the observed category.
When the predictive distribution is correct and known, this construction gives a standard normal marginal residual: the observed category selects a cumulative-probability interval, and uniform randomization fills that interval. In a fitted GLMM, parameters and random effects are estimated, so residuals need not be independent standard normal values. In particular, marginal residuals from observations in the same cluster may be dependent. Treat their plots as checks for patterns, not as proof that normal-theory tests apply.
Worked Example: Calculating Randomized Residuals
Consider four hypothetical observations with fitted category probabilities for categories 1 through 4. Their observed categories and probability vectors are: observation 1, category 3 and \((0.10,0.20,0.50,0.20)\); observation 2, category 1 and \((0.20,0.30,0.35,0.15)\); observation 3, category 4 and \((0.15,0.25,0.40,0.20)\); observation 4, category 2 and \((0.10,0.40,0.35,0.15)\). For one illustrative set of independent uniform draws, use \(U=(0.62,0.08,0.91,0.30)\).
For observation 1, category 3 occupies the cumulative interval \((0.10+0.20,\ 0.10+0.20+0.50)=(0.30,0.80)\). The draw \(0.62\) is inside it, giving \(r_1=\Phi^{-1}(0.62)\approx0.3055\).
For observation 2, category 1 occupies \((0,0.20)\), so \(r_2=\Phi^{-1}(0.08)\approx-1.405\). For observation 3, category 4 occupies \((0.80,1)\), so \(r_3=\Phi^{-1}(0.91)\approx1.341\). For observation 4, category 2 occupies \((0.10,0.50)\), so \(r_4=\Phi^{-1}(0.30)\approx-0.5244\). The resulting residuals are therefore approximately \((0.3055,-1.405,1.341,-0.5244)\).
These values are one randomized realization, not unique residuals for the four outcomes. Repeating the randomization changes them. A practical display checks for systematic departures—such as an asymmetric pattern or residual behavior that changes with fitted probabilities—rather than interpreting a single random draw as evidence against the model.
Constructing and Interpreting Displays
For category-specific and cumulative displays, compare observed proportions with average fitted probabilities in groups defined in advance or chosen for a clear diagnostic purpose. A plot can show observed and fitted proportions together, or plot their differences against a predictor or fitted probability. For randomized quantile residuals, useful displays include a histogram or a normal quantile plot, as well as a plot against a predictor or fitted quantity chosen to reveal a plausible lack of fit. No single view checks every feature of the model.
Sparse outcomes require particular restraint. If a category is rare, a small change in its count can produce a large change in its observed proportion. Conversely, an expected count near zero makes a small absolute discrepancy look large when expressed as a ratio. Avoid treating a noisy cell as decisive evidence. When uncertainty bands are needed, simulation-based checks should reproduce the fitted model’s clustering and prediction target; a reference band that assumes independent observations can be misleading.
| Display | Most direct question | Main caution |
|---|---|---|
| Category-specific proportions | Which category is over- or under-represented? | Rare categories yield noisy proportions. |
| Cumulative proportions | Is the fitted distribution too high or too low across a cutpoint? | Adjacent cutpoints are related, not independent checks. |
| Randomized quantile residuals | Do individual outcomes show a pattern relative to the fitted distribution? | Randomization and estimated effects affect the plotted values. |
These displays complement, rather than replace, other approaches. Raw category residuals remain easy to interpret in the response scale but are discrete and constrained to sum to zero. Cumulative residuals use the outcome ordering but can obscure which specific category contributes to a discrepancy. Randomized quantile residuals provide a continuous scale but add random variation. Conditional predictions and residuals focus on within-cluster fit given estimated effects; marginal predictions address population-average response distributions. Select the target first and use more than one compatible display when the data permit.
Assumptions, Diagnostics, and Common Pitfalls
Interpret these checks in light of the model and data structure. The response categories must have a meaningful order; the fitted probabilities must correspond to the ordinal model actually fitted; and the grouping or plotting scale must be stated. For conditional checks, the estimated random effects used to form predictions should be appropriate to the observations being assessed. For marginal checks, probabilities should integrate over the random-effects distribution rather than simply substitute a zero random effect. In either case, model estimation and sparse data make visual departures uncertain.
A display may point to an inadequate category-probability pattern, an unsuitable link or threshold structure, or a mismatch between the chosen prediction target and the scientific question. A visual pattern does not identify which assumption failed. Follow-up assessment should consider the design and model specification, and use simulations that preserve the fitted GLMM structure when a reference distribution is needed. These displays do not by themselves assess leverage, influence, dispersion, or cluster-level residual patterns.
- Mistake: treating category labels as equally spaced scores. A residual based on the numeric category code imposes distances that the ordinal scale does not supply. Compare category probabilities or cumulative probabilities instead.
- Mistake: interpreting a positive residual as a formal rejection. A positive difference is an observed-minus-fitted discrepancy, not a p-value. Consider the sample size, sparsity, dependence, and uncertainty before drawing a conclusion.
- Mistake: assuming one randomized residual is definitive. Its value depends on a random draw within the outcome’s cumulative-probability interval. Inspect the pattern and, where consequential, check sensitivity to repeated randomizations or simulation-based diagnostics.
- Mistake: comparing conditional observations with marginal predictions. These answer different questions. Align the prediction type with the intended diagnostic before interpreting a gap.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a group of 10 observations, category 1 is observed 4 times. Its average fitted probability is 0.30. What is the category-specific residual?
- Observed cumulative proportion at or below category 2 is 0.70, while the fitted cumulative probability is 0.80. Interpret the cumulative residual.
- Why do category-specific residuals across all categories in a group sum to zero?
- An observation is in category 2, whose fitted cumulative-probability interval is \((0.25,0.65)\). If the randomized draw is \(U=0.45\), state the residual formula and its approximate sign.
Solutions. (1) The observed proportion is \(4/10=0.40\), so the residual is \(0.40-0.30=0.10\), an excess of 0.10 in category 1. (2) The residual is \(0.70-0.80=-0.10\): the observed proportion at or below category 2 is 0.10 lower than fitted. (3) Both observed proportions and fitted probabilities sum to one, so their category-wise differences sum to \(1-1=0\). (4) The residual is \(r=\Phi^{-1}(0.45)\), approximately \(-0.1257\), which is slightly negative.