Checking Sparse Ordinal Categories

Why Check Sparse Ordinal Categories?

An ordinal GLMM can reproduce the overall distribution of responses while predicting one infrequent category poorly. This matters when that category represents a clinically or scientifically important state, or when its probability is used for prediction. A category seen only a few times is also difficult to assess with a large-sample goodness-of-fit test: the usual approximations may be unreliable, and one unusual observation can have a large effect on the apparent discrepancy.

A direct diagnostic is to compare the observed frequency of each category with frequencies generated from the fitted model. Cumulative proportions provide a complementary view: they compare the numbers at or below each ordinal threshold. The two views help show whether a discrepancy is isolated to a rare category or is part of a broader shift in the ordinal distribution.

Model and Diagnostic Quantities

Suppose \(Y_{ij}\) is the ordinal outcome for observation \(j\) in cluster \(i\), with possible categories \(1,\ldots,K\). For a cumulative-logit random-intercept model, one common specification is

Here \(\theta_1<\cdots<\theta_{K-1}\) are cutpoints, \(x_{ij}\) is the predictor vector, \(\beta\) is the fixed-effect coefficient vector, and \(b_i\) is a cluster random intercept with its specified distribution. Other ordinal GLMMs may use a different link or random-effects structure; the frequency checks below use the fitted model’s category probabilities and do not require this particular link.

Let \(Y_{ij}^{(r)}\) be the outcome in replicated data set \(r\), generated under the fitted model for the observed design, with \(r=1,\ldots,B\). Calculate \(O_k^{(r)}\), \(A_k^{(r)}\), and \(A_k^{(r)}/n\) in every replication. The observed count or proportion can then be placed beside the distribution of its replicated counterpart: for example, its simulated mean, standard deviation, and central predictive interval. A replication must preserve the cluster structure relevant to the question. Conditional replications assess outcomes for the fitted clusters, while unconditional replications include new draws of the random effects and address variation across clusters in the population. Choose the type to match the prediction target rather than selecting whichever gives a more reassuring comparison.

Why the Two Views Are Complementary

For a cumulative-link model, write \(F_{ij,k}(b_i)=P(Y_{ij}\leq k\mid b_i)\), with \(F_{ij,0}(b_i)=0\) and \(F_{ij,K}(b_i)=1\). The conditional probability of category \(k\) is the difference between adjacent cumulative probabilities:

If the target is a population-average prediction, integrate this probability over the random-effects distribution to obtain \(q_{ij,k}\). The expected category frequency is then \(\sum_{ij}q_{ij,k}\). The justification is simple: \(O_k\) is a sum of indicator variables, and linearity of expectation gives \(E(O_k)=\sum_{ij}E\{I(Y_{ij}=k)\}=\sum_{ij}P(Y_{ij}=k)\). This calculation does not require the outcomes to be independent. In practice, simulation is useful because it also gives the distribution of the counts while retaining the dependence induced by shared random effects.

Cumulative frequencies reorganize the same outcomes around the ordered cutpoints. Because \(A_k=\sum_{\ell=1}^{k}O_\ell\), a discrepancy in several adjacent category counts can show up as a systematic difference at one or more thresholds. But these checks are not independent: cumulative frequencies overlap, and the category counts must sum to \(n\). Treat them as related diagnostic views, not as separate independent tests.

Reading Cumulative Proportions

Category counts answer, “How often did each exact response occur?” Cumulative proportions answer, “How often was the response at or below each threshold?” The latter can reveal an overall tendency toward lower or higher categories that is harder to see by looking at a single rare category. Compare observed cumulative proportions with the simulated distribution at each threshold, and look for a coherent pattern rather than counting every threshold as a separate discovery.

Checking a Particularly Rare Category

When an exact category is rare, its predictive distribution may be strongly skewed and concentrated on a few integer values. A mean and standard deviation can summarize that distribution, but the empirical interval and tail fraction are often more informative. Always inspect the counts themselves: a simulated mean of 1.1 does not imply that a count of 1.1 is possible in any individual data set.

Assumptions, Diagnostics, and Common Pitfalls

A meaningful comparison depends on a valid setup. Confirm that category labels and ordering are correct, that the observed and replicated data use the same observations and denominator, and that replications were generated from a fitted model that converged and used the intended random-effects structure. Check that the chosen conditional or unconditional replication matches the target. If the data contain unequal numbers of observations per cluster, the replication and frequency calculation must retain those observed cluster sizes when that is part of the target.

Inspect the whole replicated distribution, not only a mean. A frequency can have a skewed or discrete predictive distribution, and adjacent category counts are constrained because they sum to \(n\). If many categories and cutpoints are checked, at least one may look extreme by chance. A simulation tail fraction is a diagnostic summary, not automatically a calibrated frequentist p-value; it does not give the probability that the model is true or that the null hypothesis is true. The estimate also has Monte Carlo error. For a tail fraction \(\hat{r}=r/B\), its approximate Monte Carlo standard error is \(\sqrt{\hat{r}(1-\hat{r})/B}\), so a small number of replications can make an apparent tail probability unstable.

The simulation-based checks are especially useful when ordinary large-sample approximations are questionable, but they are not a substitute for a well-specified model. They can flag mismatch without identifying its source, and results may depend on whether predictions are conditional on observed clusters or average over the random-effects distribution. A low tail fraction also does not account automatically for selecting the category to inspect after seeing the data. Follow an unusual pattern by checking model assumptions, coding, cluster structure, and the scientific meaning of the rare response.