Ordinal Outcomes Require Ordered Category Probabilities

Why the Order of Categories Matters

A response such as pain rated “none,” “mild,” “moderate,” or “severe” is not an event count: the values name categories, not numbers of occurrences. But the categories are not interchangeable labels either. “Severe” indicates a higher response level than “moderate.” A model for this outcome should represent that ordering, while avoiding an unsupported assumption that the gaps between adjacent categories are equal.

For a GLMM, it is useful to think about the response probabilities conditional on the observed predictors and the relevant random effect. Let \(Y_{ij}\) be the response for observation \(i\) in cluster \(j\), and let \(b_j\) denote that cluster’s random effect. If the response has \(K\) categories, its conditional distribution assigns a probability to each category for given covariates and \(b_j\). The ordering of the categories is part of how those probabilities are represented; it is not a claim that the labels have numerical distances.

Definition: An ordinal response takes values in a finite set of \(K\) categories with a meaningful order, written \(c_1<c_2<\cdots<c_K\). The order tells us which category is higher or lower, but does not specify the size of the difference between adjacent categories. For given covariates and random effects, the category probabilities \(p_k=P(Y_{ij}=c_k\mid\text{covariates},b_j)\) must each be between 0 and 1 and must sum to 1.

A critical distinction is that the individual category probabilities \(p_1,\ldots,p_K\) do not have to increase or decrease as the categories rise. For example, the middle category may be the most common. The order constraint applies to cumulative probabilities: the probability of being at or below a category cannot decrease as the boundary moves upward.

The Probability Requirements Imposed by Order

Write the cumulative probability through category \(c_k\) as \(F_k=P(Y_{ij}\leq c_k\mid\text{covariates},b_j)\). For \(K\) ordered categories, there are \(K-1\) nontrivial cumulative probabilities. Since each event \(\{Y_{ij}\leq c_k\}\) is contained in the next event \(\{Y_{ij}\leq c_{k+1}\}\), their probabilities must obey the following inequalities.

0\leq F_1\leq F_2\leq\cdots\leq F_{K-1}\leq 1

These inequalities give a direct check on whether a set of model-predicted cumulative probabilities can describe an ordinal response. They also show how the category probabilities follow from the cumulative ones: the probability of an intermediate category is the increase in cumulative probability at that category. The first category collects the probability up to the first boundary, and the last collects the remaining probability.

p_1=F_1,\qquad p_k=F_k-F_{k-1}\ \ (2\leq k\leq K-1),\qquad p_K=1-F_{K-1}

This relationship is a basic justification for requiring an ordinal model to produce ordered cumulative probabilities. If a cumulative probability falls when its boundary moves up, the difference for the intervening category is negative. A negative probability is impossible. Conversely, if the cumulative probabilities lie between 0 and 1 and are nondecreasing, the differences are nonnegative and the category probabilities sum to 1.

\sum_{k=1}^{K}p_k =F_1+\sum_{k=2}^{K-1}(F_k-F_{k-1})+(1-F_{K-1}) =1

The ordering requirement does not mean that a particular category must be more probable than the category below it. It means the cumulative chance of being at or below successive category boundaries must be nondecreasing. That distinction matters when checking fitted predictions: a pattern such as \(p_1<p_2>p_3\) can be entirely valid, while decreasing cumulative probabilities cannot.

Worked Example: Checking Four Category Probabilities

Suppose a hypothetical GLMM predicts, for a patient with specified covariates and cluster random effect, probabilities of 0.20 for “none,” 0.35 for “mild,” 0.30 for “moderate,” and 0.15 for “severe.” Check whether these probabilities are valid and find the cumulative probabilities.

Each category probability is between 0 and 1. Their sum is \(0.20+0.35+0.30+0.15=1.00\), so they form a valid distribution over the four categories. Accumulating from the lowest category gives

F_1=0.20,\qquad F_2=0.20+0.35=0.55,\qquad F_3=0.20+0.35+0.30=0.85

The cumulative probabilities satisfy \(0\leq0.20\leq0.55\leq0.85\leq1\). Notice that the category probabilities themselves are not monotone: “mild” is more probable than “none,” but “moderate” is less probable than “mild.” That is not a violation of ordinal structure.

How Order Enters the Likelihood

For a single observation, the likelihood contribution is the probability assigned to the category actually observed. If \(Y_{ij}=c_k\), that contribution is \(p_k\). Thus the observed category is treated as a category—not as a numeric score whose distance from another score determines the contribution.

For example, if observations are conditionally independent given the covariates and cluster random effects, the conditional likelihood for the observed responses is the product of their corresponding conditional category probabilities. This independence condition is part of the assumed response model; a random effect may represent dependence among observations in the same cluster, but conditional independence must still be appropriate after conditioning on it. The model’s predicted probabilities must be valid for each relevant covariate and random-effect value.

Key requirement: An ordinal response model must assign valid probabilities to the ordered categories and preserve the ordering of cumulative probabilities. A likelihood that assigns a separate probability to each category can describe category frequencies, but unless its structure uses the known order, it does not make the response model order-aware.

The order does not determine a unique likelihood or a unique relationship between predictors and probabilities. It constrains what the probabilities mean and what a coherent model must preserve. Different ordinal likelihoods can encode the category ordering in different ways; the choice depends on the response process and is a separate modeling decision. The probability checks above apply regardless of that choice.

Worked Example: From Cumulative Predictions to an Observed-Data Likelihood

Consider a hypothetical four-category response with predicted cumulative probabilities \(F_1=0.20\), \(F_2=0.55\), and \(F_3=0.85\). A small hypothetical sample has two responses in category 1, one in category 2, one in category 3, and none in category 4. Calculate the category probabilities and the likelihood of these observed category counts.

First take differences between adjacent cumulative probabilities, including the probability remaining above the last boundary:

p_1=0.20,\quad p_2=0.55-0.20=0.35,\quad p_3=0.85-0.55=0.30,\quad p_4=1-0.85=0.15

For the particular ordered sequence of four recorded outcomes, the conditional likelihood contribution is

$$ L=(0.20)^2(0.35)^1(0.30)^1(0.15)^0 =0.0042 $$

If only the category counts are recorded, rather than a particular order of the four observations, there are \(4!/(2!1!1!)=12\) sequences with those counts. The probability of those counts is therefore \(12(0.0042)=0.0504\). The calculation uses category probabilities derived from ordered cumulative probabilities; it does not use numerical distances between category labels.

Order Without Equal Spacing

Ordered categories provide a ranking, not a measurement scale. If the response is coded as 1, 2, 3, and 4 for convenience, this coding records the category order. It does not establish that the change from category 1 to 2 is the same size as the change from category 3 to 4. Nor does it establish that a one-unit predictor change has a constant effect on a numerical response score.

That is why the category-probability representation is useful: it concerns the chance of each possible response, without needing to assign distances to category labels. A model may use the order to organize how probabilities relate across the categories, but it must not quietly turn ranks into equally spaced measurements unless that assumption is separately justified. In particular, a response such as “none, mild, moderate, severe” does not become continuous merely because its categories are numbered.

Worked Example: A Valid Distribution With Unequal Category Gaps

Suppose a hypothetical symptom scale uses the ordered labels “absent,” “noticeable,” “limiting,” and “incapacitating.” At one predictor setting, a model gives cumulative probabilities \(0.28\), \(0.62\), and \(0.91\). Find the category probabilities and explain what can and cannot be concluded about the category labels.

The cumulative probabilities are nondecreasing and lie between 0 and 1. The corresponding category probabilities are

p_1=0.28,\quad p_2=0.62-0.28=0.34,\quad p_3=0.91-0.62=0.29,\quad p_4=1-0.91=0.09

They are all nonnegative and sum to \(0.28+0.34+0.29+0.09=1.00\), so they form a valid distribution. The response is ordered from “absent” to “incapacitating,” but these probabilities say nothing about the size of the gap from “absent” to “noticeable” relative to the gap from “limiting” to “incapacitating.” The categories are ordered labels, not measured distances.

Assumptions, Diagnostics, and Alternatives

Before using an ordinal likelihood in a GLMM, check that the response has a finite set of categories and that a defensible ordering can be stated. The category definitions should be mutually exclusive, so one observation belongs to one response category, and the listed categories should cover the possible responses being modeled. The conditional probabilities must sum to one and be nonnegative at the covariate and random-effect values relevant to the model. If observations within a cluster are assumed conditionally independent, that assumption should be plausible after accounting for the random effects.

Several practical checks help identify problems:

List the categories in their substantive order before coding them. A code order that differs from the intended response order can make model output misleading.
Check predicted cumulative probabilities across every adjacent boundary. A decrease indicates that at least one implied category probability is negative.
Check category probabilities as well as cumulative probabilities. They must be nonnegative, no greater than 1, and sum to 1, allowing only small rounding discrepancies.
Review category definitions and observed responses for ambiguity or missing response levels. A model cannot repair categories that do not represent the response process coherently.
Compare predicted category frequencies with the data as a diagnostic, while recognizing that overall agreement alone does not establish that the ordering assumptions or the random-effects structure are adequate.

An order-ignoring multicategory likelihood is a possible alternative when categories have no meaningful order. It can assign probabilities to labels without using a ranking. For ordered outcomes, that approach may fail to use information already present in the response definition. Conversely, imposing an order where none is defensible adds structure without substantive support. Treating category codes as a numeric response is a different alternative, but it brings in assumptions about numerical spacing and effects that do not follow from category order alone.

Order-aware models also have limits. Respecting the order does not guarantee that a chosen model fits well, that its random-effects assumptions are appropriate, or that its effects have a causal interpretation. Nor does order alone dictate a single link function or a single likelihood. Those choices require attention to how the response categories arise and to the model’s assumptions; the first requirement is simply that the probability representation be coherent and genuinely reflect the known order.

Common mistake: “The response categories are ordered, so their probabilities must rise from the lowest category to the highest.” This confuses category probabilities with cumulative probabilities. Individual category probabilities can rise and fall; what must be nondecreasing is the cumulative probability of being at or below successive category boundaries.

Check Your Understanding

Answer each question, then compare with the solutions below.

A response has categories \(c_1<c_2<c_3\). Its cumulative probabilities are \(0.30\) and \(0.72\). Find the three category probabilities.
For four ordered categories, a model predicts cumulative probabilities \(0.25\), \(0.60\), and \(0.52\). Explain why these cannot describe a valid ordinal response distribution.
Must the probabilities of successive individual categories increase as their labels rise? Explain.
What does coding four ordered categories as 1, 2, 3, and 4 establish, and what does it not establish?

Solutions. (1) \(p_1=0.30\), \(p_2=0.72-0.30=0.42\), and \(p_3=1-0.72=0.28\). These are nonnegative and sum to 1. (2) The cumulative probability falls from \(0.60\) to \(0.52\). The implied probability for category 3 is \(0.52-0.60=-0.08\), which is impossible. (3) No. Individual category probabilities may increase or decrease; cumulative probabilities must be nondecreasing across category boundaries. (4) The coding can represent the category ranking. It does not establish equal spacing between categories or justify treating the response as a quantitative measurement.