Why Ordered-Response Diagnostics Need to Respect Order
An ordered response has categories with a meaningful ranking, such as “low,” “moderate,” and “high,” but the distances between categories are not generally known. Diagnostics should therefore ask whether the fitted model respects that ranking and reproduces the probabilities of the ordered outcomes. Treating category labels as arbitrary nominal outcomes can miss problems; treating them as equally spaced numerical scores can impose an unjustified structure.
This tutorial focuses on four checks: threshold ordering, the proportional-odds restriction, residual and calibration patterns, and sensitivity to how categories are coded. None of these checks proves a model correct. Their purpose is to identify specific ways in which the fitted model may fail and to guide careful follow-up.
These diagnostics assume the response categories have been ordered meaningfully and the covariates and observations have been specified appropriately. Standard likelihood-based tests also rely on suitable regularity conditions, including adequate information in the data; sparse categories or small samples can make their approximations unreliable. If observations are clustered or otherwise dependent, uncertainty calculations must account for that dependence.
Threshold Ordering and Implied Probabilities
Thresholds divide the cumulative response scale. Since \(P(Y_i\leq j\mid x_i)\) cannot decrease as \(j\) increases, the fitted cumulative probabilities must be nondecreasing at every covariate value. In an ordered logit model, the logistic function is increasing, so ordered thresholds guarantee ordered cumulative probabilities. Taking adjacent differences then gives nonnegative category probabilities.
Many estimation procedures parameterize the thresholds so that their order is enforced automatically. In that case, a threshold-order diagnostic is still useful as a check on the reported estimates, numerical convergence, and the probabilities actually computed. If thresholds are estimated separately or by an unconstrained procedure, crossing thresholds are a substantive warning: the resulting cumulative probabilities can cross, implying a negative probability for one or more categories.
Worked Example: Checking Thresholds and Category Probabilities
Suppose a hypothetical four-category ordered logit model has estimated thresholds \(\hat{\tau}_1=-1.10\), \(\hat{\tau}_2=0.35\), and \(\hat{\tau}_3=1.20\). At \(x_i^\prime\hat{\beta}=0\), check their order and calculate the fitted category probabilities.
Threshold order. The estimates satisfy \(-1.10<0.35<1.20\), so they are in the required order. At this covariate value the cumulative probabilities are the logistic transforms of the thresholds: $$ \hat{F}_1=\frac{1}{1+\exp(1.10)}\approx0.2497,\quad \hat{F}_2=\frac{1}{1+\exp(-0.35)}\approx0.5866,\quad \hat{F}_3=\frac{1}{1+\exp(-1.20)}\approx0.7685. $$
Category probabilities. Take adjacent differences, including the endpoints zero and one: $$ (\hat{p}_1,\hat{p}_2,\hat{p}_3,\hat{p}_4) =(0.2497,\;0.5866-0.2497,\;0.7685-0.5866,\;1-0.7685) \approx(0.2497,\;0.3369,\;0.1819,\;0.2315). $$ The probabilities are nonnegative and sum to approximately \(1.0000\), allowing for rounding.
If an unconstrained fit instead produced \(\hat{\tau}_1=0.42\) and \(\hat{\tau}_2=0.31\), then at any fixed covariate index the estimated cumulative probability at cutpoint 1 would exceed the one at cutpoint 2. Their difference, which should be the probability of category 2, would be negative. That is not a valid ordered probability distribution; it is a model or estimation warning, not an interpretable prediction.
Assessing the Proportional-Odds Restriction
Proportional odds requires that covariates have the same slope at every cumulative cutpoint. A useful diagnostic compares the proportional-odds model with a less restrictive model that allows one or more slopes to differ across cutpoints. Likelihood-ratio, Wald, or score tests can assess the equality restrictions when the models, estimates, and covariance calculations meet the relevant regularity conditions. A joint test asks whether the restrictions fail overall; variable-specific tests can help locate which slopes are driving a discrepancy.
A rejection is evidence against the restriction as specified, not a proof that every proportional-odds prediction is poor. Conversely, a non-rejection may reflect limited information, especially when some categories are rare. Inspect predicted probabilities and covariate-specific patterns as well as a test statistic. The less restrictive model must itself produce coherent, ordered cumulative probabilities in the range of covariates being considered.
Worked Example: Testing Equality of Two Cumulative Slopes
Consider a hypothetical partial proportional-odds fit for an outcome with three ordered categories and one covariate, years of experience. Its cumulative logits are \(\operatorname{logit}\{P(Y_i\leq j\mid x_i)\}=\tau_j-\beta_jx_i\), with estimates \(\hat{\beta}_1=0.80\) and \(\hat{\beta}_2=0.30\). Suppose their estimated standard errors are \(0.20\) and \(0.18\), and their estimated covariance is \(0.009\). Test the proportional-odds restriction \(\beta_1=\beta_2\) using a Wald test.
The estimated difference is \(0.80-0.30=0.50\). Its estimated variance, accounting for the covariance, is $$ \widehat{\operatorname{Var}}(\hat{\beta}_1-\hat{\beta}_2) =0.20^2+0.18^2-2(0.009)=0.0544, \qquad SE(\hat{\beta}_1-\hat{\beta}_2)=\sqrt{0.0544}\approx0.2332. $$
Thus the Wald statistic is \(z=0.50/0.2332\approx2.144\), or \(z^2\approx4.60\), compared with a \(\chi^2\) distribution with one degree of freedom. The approximate two-sided p-value is \(0.032\). At the 5% level, this is evidence against equal slopes for experience in this illustrative test, subject to the test’s model and large-sample assumptions.
The sign convention matters for interpretation. Under the stated model, increasing experience by one year changes the cumulative log odds at the first cutpoint by \(-0.80\), and at the second by \(-0.30\). The two slopes are unequal in magnitude, so their cumulative-log-odds changes differ. Reversing both signs would not change the equality test, but reporting the changes as \(+0.80\) and \(+0.30\) would be inconsistent with the specified model.
Residual Patterns and Calibration
Residuals for an ordinal response are not ordinary continuous errors: the observed outcome is discrete, and the fitted model supplies a probability for each possible category. One useful diagnostic is the randomized quantile residual. Let \(\hat{F}(y\mid x_i)\) be the fitted conditional cumulative distribution and let \(V_i\) be independently drawn from a uniform distribution on \((0,1)\). For an observed category \(y_i\), define $$ U_i=\hat{F}(y_i-1\mid x_i)+V_i\{\hat{F}(y_i\mid x_i)-\hat{F}(y_i-1\mid x_i)\},\qquad r_i=\Phi^{-1}(U_i). $$
If the conditional distribution is correctly specified and its parameters were known, this randomization spreads each discrete probability mass uniformly over its cumulative-probability interval; \(U_i\) is uniform on \((0,1)\), and \(r_i\) follows a standard normal distribution. With estimated parameters, these are approximate diagnostic reference patterns, not an exact finite-sample guarantee. Plot residuals against fitted values and important covariates, and inspect their overall distribution. Trends, changing spread, or repeated extreme residuals can signal a missing predictor, functional-form problem, or poorly fitted regions. Because the residuals use randomization, check that a pattern is not an artifact of one random draw, for example by repeating the diagnostic with new draws.
Calibration checks whether predicted probabilities agree with observed frequencies. For ordered outcomes, examine category-specific frequencies and cumulative frequencies, by covariate or risk groups when feasible. A model might reproduce the probability of being at or below category 2 while systematically underpredicting category 1. Looking only at a single overall accuracy rate can conceal this distinction.
Worked Example: Reading Cumulative Calibration and Randomized Residuals
For two hypothetical groups of 100 observations each, a fitted three-category model gives the following average predicted category probabilities and observed counts.
| Group | Predicted probabilities for categories 1, 2, 3 | Observed counts for categories 1, 2, 3 |
|---|---|---|
| A | 0.20, 0.50, 0.30 | 25, 45, 30 |
| B | 0.50, 0.30, 0.20 | 45, 35, 20 |
For group A the expected counts are \((20,50,30)\), and for group B they are \((50,30,20)\). In group A, the observed proportion in category 1 is \(25/100=0.25\), compared with a predicted \(0.20\); the cumulative proportions at or below category 2 are both \(0.70\), since \((25+45)/100=0.70\) and \(0.20+0.50=0.70\). In group B, the observed proportion in category 1 is \(0.45\), compared with \(0.50\), while both observed and predicted proportions at or below category 2 are \(0.80\). This pattern suggests that the first-category probabilities differ from observed frequencies in opposite directions across groups, even though the cumulative comparison at the second cutpoint matches in both.
For illustration, the grouped Pearson discrepancy is $$ \sum\frac{(O-E)^2}{E} =\frac{(25-20)^2}{20}+\frac{(45-50)^2}{50} +\frac{(45-50)^2}{50}+\frac{(35-30)^2}{30} =1.25+0.50+0.50+0.8333=3.083. $$
This number summarizes the discrepancy in these grouped counts; it is not, by itself, a valid p-value. A formal reference distribution would require accounting for how groups were formed, parameters estimated, and any dependence or other design features. To examine individual residuals, each observed category has a fitted cumulative-probability interval. Randomized quantile residuals place observations within those intervals; repeated unusually large residuals in group A, for example, would warrant checking the group’s fitted probabilities and covariates rather than inferring a cause from the grouped discrepancy alone.
Sensitivity to Category Coding
An ordinal model uses the order of categories, not numerical distances between their labels. Relabelling categories as “poor,” “fair,” and “good” rather than 1, 2, and 3 changes nothing if their order is preserved and the model is fitted to the same responses. Reversing the order also preserves the underlying information, but reverses coefficient signs under the convention used here. In contrast, collapsing adjacent categories changes the response definition and the thresholds; it is a sensitivity analysis, not a harmless recoding.
For a model with \(K\) categories, define the reversed response as \(Y_i^*=K+1-Y_i\). Then $$ P(Y_i^*\leq j\mid x_i)=1-P(Y_i\leq K-j\mid x_i), \qquad \operatorname{logit}\{P(Y_i^*\leq j\mid x_i)\} =-\tau_{K-j}+x_i^\prime\beta. $$
This has the same proportional-odds form with \(\tau_j^*=-\tau_{K-j}\) and \(\beta^*=-\beta\). Therefore, fitted probabilities for the corresponding reversed categories remain the same; the coefficient signs change because “at or below” now refers to the opposite end of the scale.
Worked Example: Reversing and Collapsing an Ordered Scale
Suppose a hypothetical five-category satisfaction response is coded from 1 (very dissatisfied) to 5 (very satisfied), and a fitted coefficient for a predictor is \(\hat{\beta}=0.40\) in the convention \(\operatorname{logit}\{P(Y_i\leq j\mid x_i)\}=\tau_j-\beta x_i\). Interpret the result under the original and reversed coding.
Under the original coding, a one-unit increase in the predictor changes each cumulative log odds \(\operatorname{logit}\{P(Y_i\leq j\mid x_i)\}\) by \(-0.40\), holding other covariates fixed. If the categories are reversed to 1 (very satisfied) through 5 (very dissatisfied), the corresponding coefficient is \(-0.40\). Its cumulative-log-odds change is then \(-(-0.40)=+0.40\). The probabilities for corresponding satisfaction levels have not changed; only the direction used to define cumulative outcomes has changed.
Now suppose categories 1 and 2 are combined into “dissatisfied.” The response has fewer categories and different cutpoints, and the probability being modeled at the new cutpoint is not identical to every original cumulative probability. Refit the model and compare substantive predictions, uncertainty, and diagnostics. A change in results could reflect the coarser outcome definition, not merely a coding convention.
Using Diagnostic Evidence Carefully
A coherent diagnostic assessment combines model structure with observed patterns. Thresholds should be ordered; predicted category probabilities should be nonnegative and sum to one; proportional-odds restrictions should be assessed rather than assumed to have passed merely because estimation converged; and residual and calibration patterns should be examined across relevant covariates. When categories are sparse, emphasize the uncertainty and stability of the patterns instead of relying on a single asymptotic test.
Common mistake: “The proportional-odds test was not significant, so the restriction is true and the model is validated.” A non-significant test does not establish the null hypothesis; it says the data did not provide sufficiently strong evidence against the restriction at the chosen level under the test’s assumptions. Check whether the sample has enough information to detect meaningful departures, examine predicted probabilities and calibration, and report the limitation. Similarly, crossing thresholds in an unconstrained fit should not be dismissed as a minor coefficient quirk: it can imply invalid probabilities.
Check Your Understanding
Answer these questions using the model convention and diagnostic principles described above.
- Why can crossing estimated thresholds produce a negative fitted probability for a category?
- Under \(\operatorname{logit}\{P(Y_i\leq j\mid x_i)\}=\tau_j-x_i^\prime\beta\), what is the change in cumulative log odds for a one-unit increase in a covariate whose coefficient is \(0.60\)?
- What does a non-significant test of equal cumulative slopes establish, and what does it not establish?
- Why might category-specific calibration reveal a problem that a cumulative comparison at the last cutpoint misses?
- What changes to coefficient signs and fitted probabilities should result from reversing all category labels while preserving their order in reverse?