Diagnostics for Ordinal Response Models

Why Diagnose Ordinal Calibration?

An ordinal response has categories with a meaningful order, such as none, mild, moderate, and severe. A model can estimate plausible average category frequencies yet still miss important features: it may underpredict severe responses at late times, for example, while overpredicting them earlier. Diagnostics should therefore examine more than a single overall fit summary.

This tutorial focuses on calibration: whether the model’s predicted category probabilities agree with observed category frequencies at relevant cutpoints and across covariates and time. These are descriptive checks of model fit, not proof that the model is correct. In repeated-measures NLME data, the level at which predictions are made—conditional on subject random effects or averaged over their distribution—must also match the question being asked.

Predictions and Residuals for an Ordinal NLME Model

Let \(Y_{ij}\) be the ordinal response for subject \(i\) at observation \(j\), taking values \(1,\ldots,K\). Let \(x_{ij}\) denote covariates and \(b_i\) the subject-specific random effects. For illustration, consider a cumulative-logit proportional-odds model:

Here \(\theta_1<\cdots<\theta_{K-1}\) are ordered thresholds, \(\beta\) contains fixed-effect coefficients, and \(b_i\) follows the model’s specified random-effects distribution. Write \(F_{ijk}=P(Y_{ij}\leq k\mid b_i,x_{ij})\), with \(F_{ij0}=0\) and \(F_{ijK}=1\). The conditional category probability is \(p_{ijk}=F_{ijk}-F_{ij,k-1}\). This difference is essential: the model predicts probabilities for categories, not just probabilities of being below each threshold.

For a population calibration question, compare observed frequencies with population-averaged probabilities. Conditional predictions can help assess how well fitted subject trajectories align with their own observations, but estimates of \(b_i\) are informed by those same observations. That reuse can make conditional fit look better than performance for a new subject. State which prediction is used, and avoid treating either kind as a substitute for the other.

At an observation with predicted category probabilities \(p_{ij1},\ldots,p_{ijK}\), define the category residual \(R_{ijk}=I(Y_{ij}=k)-p_{ijk}\). For a cutpoint \(k\), define the cumulative residual \(C_{ijk}=I(Y_{ij}\leq k)-F_{ijk}\). Under a correctly specified model and the matching conditioning or averaging scheme, each residual has expectation zero. The category residuals at one observation sum to zero because both the observed indicators and predicted probabilities sum to one.

Cumulative-Probability Calibration

An ordinal scale has \(K-1\) cumulative cutpoints. For each observation and cutpoint \(k\), the model predicts \(F_{ijk}=P(Y_{ij}\leq k)\). Calibration can be assessed by comparing these predictions with the observed indicators \(I(Y_{ij}\leq k)\). A practical plot groups observations with similar predicted cumulative probabilities, then plots the observed proportion at or below the cutpoint against the mean predicted probability in each group. A well-calibrated model should show points near the 45-degree line, allowing for sampling variability.

Make a separate calibration display for each cutpoint, or label the cutpoints clearly if combining them. A model may predict the first cutpoint well but miscalibrate the probability of being at or below a higher category. For repeated observations, grouping all records together describes record-level calibration; it does not make the records independent. Stratification by subject-level or time-varying features can reveal patterns hidden by an overall plot.

Stratified Residual Summaries

A residual summary by covariate or time asks whether discrepancies are systematic in a scientifically relevant part of the data. For a chosen cutpoint, average \(C_{ijk}\) within a stratum. A positive average means more observations fell at or below the cutpoint than the model predicted; a negative average means fewer did. For category-specific summaries, average \(R_{ijk}\): a positive value indicates that the category occurred more often than predicted in that stratum.

Choose strata before inspecting results when possible, and use bins that preserve enough observations for stable proportions. For a continuous covariate or time, plot observed and predicted proportions across sensible ranges or use a smooth summary with uncertainty. Examine covariate and time patterns jointly when a time trend might differ across covariate groups. Sparse categories or small strata can produce noisy apparent departures.

A randomized quantile residual offers another observation-level summary. For observed category \(y\), draw \(U\) uniformly from the interval \((F_{ij,y-1},F_{ij,y})\), using probabilities on the same conditional or population-averaged scale as the analysis. Then set \(r_{ij}=\Phi^{-1}(U)\), where \(\Phi\) is the standard normal cumulative distribution function. Under a correctly specified predictive distribution, this randomization produces standard-normal marginal residuals. Because the added uniform draw makes the values random, examine patterns over repeated draws or use stable summaries rather than overinterpreting individual residuals. Repeated outcomes can still be dependent, so a normal-looking histogram alone does not diagnose within-subject structure.

Justification, Diagnostics, and Limits

The zero-mean property of the residuals follows directly from the definition. For category \(k\), conditional on the covariates and random effect, \(I(Y_{ij}=k)\) is a Bernoulli indicator with expectation \(p_{ijk}\). Therefore:

Likewise, \(I(Y_{ij}\leq k)\) has expectation \(F_{ijk}\), so \(E(C_{ijk}\mid b_i,x_{ij})=0\). Averaging over the random-effects distribution preserves zero expectation when the predictions are integrated over that same distribution. This explains why persistent nonzero residual averages across meaningful strata are diagnostically relevant. It does not supply an independent-observation standard error: dependence between records from the same subject must be respected in formal uncertainty calculations.

Look for repeated patterns rather than isolated deviations: cumulative plots consistently above or below the diagonal, residual means changing with time, or category discrepancies concentrated at a covariate level. Also check whether the fitted model produces valid ordered cumulative probabilities and nonnegative category probabilities, and whether conclusions are driven by very small groups. A failure of the proportional-odds structure may appear as different calibration patterns across cutpoints; the stratified plots can help identify where the discrepancy occurs.

These diagnostics have limits. Overall frequencies can hide subgroup misfit; residual averages can hide opposing patterns within a stratum; and randomized residuals depend partly on the randomization. In-sample comparisons are not independent validation, particularly when subject-specific random effects are estimated from the same records being assessed. Nor does a successful calibration plot establish that the model’s random-effects distribution, dependence assumptions, or scientific interpretation is correct.

Compared with raw category counts alone, cumulative-probability plots use the ordering of the response and assess calibration at each cutpoint. Compared with a single residual histogram, stratified summaries can locate covariate- or time-specific discrepancies. Neither replaces careful examination of the model’s assumptions or independent assessment of predictive performance. The ordered categories are central to these tools; treating them as equally spaced numerical scores and relying only on \(Y_{ij}-E(Y_{ij})\) can obscure which category boundary is miscalibrated.