Model Comparison When Ordinal Alternatives Are Not Nested

Why Non-Nested Ordinal Models Need Different Comparisons

When ordinal responses are modeled, a cumulative-link model and a baseline-category multinomial model can describe different ways that predictors relate to category probabilities. For more than two response categories, their usual forms are generally nonnested: neither is simply the other with some parameters fixed. A larger maximized likelihood for one model is therefore not, by itself, a likelihood-ratio test result.

For a cumulative-link model, the model is commonly expressed through cumulative probabilities, such as \(\operatorname{logit}P(Y\leq j\mid x)=\alpha_j-x^\mathsf{T}\beta\), with ordered cutpoints \(\alpha_j\). A baseline-category multinomial logit instead models contrasts such as \(\log\{P(Y=j\mid x)/P(Y=K\mid x)\}\) for \(j=1,\ldots,K-1\). These specifications use the same ordinal outcome but impose different structures on its category probabilities. When comparing them, use methods that compare predictive or penalized fit without assuming a nested-model relationship.

What Makes a Comparison Fair?

A fair comparison targets the same observed response, on the same sampling units, with a comparable likelihood. For example, compare models fit to the same \(n\) independent individuals and the same response categories. When using information criteria, the likelihoods must use compatible conventions, including any likelihood constants. Do not compare a conditional likelihood for one model to a marginal likelihood for another as though they were the same target.

Count all freely estimated parameters in the model: slopes, intercepts, cutpoints, and any estimated nuisance parameters included in the likelihood. For grouped multinomial data, the sample size relevant to BIC is generally the total number of independent observations represented by the counts, not merely the number of rows in the grouped table.

Information Criteria

Let \(\ell(\hat\theta)\) denote the maximized log-likelihood and \(k\) the number of freely estimated parameters. The Akaike information criterion and Bayesian information criterion are

For either criterion, smaller is preferred among the candidates being compared. The likelihood term rewards fit; the penalty discourages using extra parameters. AIC is commonly used to compare estimated predictive information loss, while BIC applies a larger complexity penalty as \(n\) grows and has a model-selection interpretation under particular large-sample assumptions. Neither criterion is a hypothesis test, and neither establishes that a selected model is true.

The comparison can be expressed directly. For models \(A\) and \(B\),

Thus a gain in maximized log-likelihood must be large enough to offset the extra-parameter penalty for AIC to favor the more complex model. For BIC, the corresponding penalty difference is \((k_A-k_B)\log n\). Because the penalties differ, AIC and BIC can rank models differently without either calculation being erroneous.

Predictive Log Scores

A direct way to compare predictions is to evaluate probabilities assigned to responses that were not used to fit the model. If a fitted model assigns probability \(\hat p_i(y_i)\) to the response actually observed for held-out individual \(i\), its average negative log score is

where \(m\) is the number of held-out observations. Lower values are better. Equivalently, the average log score without the minus sign is better when larger. A valid comparison evaluates both models on the same held-out observations, with predictions produced without using those observations to fit the models.

To verify the result, let the true category probabilities be \(p_1,\ldots,p_K\), and let a model report \(q_1,\ldots,q_K\), where both vectors sum to one and \(q_j>0\) whenever \(p_j>0\). The difference between the expected log score under \(p\) and the expected log score under \(q\) is

For \(t>0\), \(\log t\leq t-1\). Apply this with \(t=q_j/p_j\) for categories with \(p_j>0\):

The first inequality is an equality only when \(q_j/p_j=1\) for every category with \(p_j>0\). For the displayed difference also to be zero, the reported probability assigned outside the true support must be zero. Hence equality occurs exactly when \(q=p\). This proves the expected-score claim. In practice the true probabilities are unknown, so held-out log scores estimate predictive performance rather than reveal it exactly. A model assigning zero probability to an observed category receives a log score of negative infinity for that observation.

Cross-Validation and Uncertainty in the Ranking

When a single validation set would be small or unstable, use cross-validation. Divide the observations into folds, fit each candidate model on the other folds, and score its predictions on the held-out fold. Aggregate the held-out negative log scores, preferably weighting observations appropriately if fold sizes differ. Each model must be refit without the observations it is asked to predict; scoring training-set predictions would reward fit to data already used for estimation.

Cross-validation estimates predictive performance for models trained on the corresponding reduced training samples. Its result can vary with the fold assignment and sampling variability. A small difference in average score should therefore be interpreted alongside the variation across folds and the practical importance of the difference. The fold scores are not automatically independent observations, so a simple standard error computed across folds is only a rough description, not a universal formal test.

Why the Usual Likelihood-Ratio Chi-Square Test Does Not Apply

For nested models, the likelihood-ratio statistic is formed by subtracting maximized log-likelihoods, with the larger model containing the smaller one. Under regularity conditions—such as an interior null parameter, identifiability, and a correctly specified likelihood—Wilks’ theorem gives an asymptotic chi-square reference distribution, with degrees of freedom equal to the difference in parameter dimensions.

That justification requires the smaller model to be a restriction of the larger model. For nonnested alternatives, there is no such restriction relationship, and the difference in parameter counts does not supply the degrees of freedom of a chi-square distribution. Consequently, calculating \(2\{\ell_B(\hat\theta_B)-\ell_A(\hat\theta_A)\}\) and comparing it to a chi-square distribution with \(k_B-k_A\) degrees of freedom is not justified merely because the models have different numbers of parameters. The statistic may even be negative if the model labelled \(B\) has the smaller maximized likelihood.

There may be specialized tests for particular nonnested hypotheses, but they require their own assumptions and derivations. They are not obtained by applying the ordinary nested-model likelihood-ratio rule. For the comparison in this tutorial, AIC or BIC can summarize penalized in-sample fit, while validation or cross-validated log scores target predictive performance.

Common Pitfalls and Practical Choice

• Treating a larger likelihood as a test result. Maximized likelihood alone rewards flexibility. Use a criterion or predictive evaluation rather than assigning a nested-model chi-square reference distribution.
• Comparing incompatible likelihoods. Ensure the response, observations, likelihood target, and likelihood conventions match. Otherwise, a numerical criterion difference may not represent a meaningful comparison.
• Counting only slopes. Cutpoints, intercepts, and other estimated parameters contribute to \(k\); omitting them can change AIC or BIC.
• Overstating a small predictive-score difference. Held-out scores vary with the validation sample and fold assignment. Report the scale of the difference and its variability.
• Assuming the criteria answer the same question. AIC, BIC, and held-out scores use different ideas and can rank models differently. Choose the comparison to match the scientific or predictive goal.

A practical comparison begins by checking that both candidates address the same response data. Use AIC or BIC when a penalized likelihood ranking is useful; use held-out or cross-validated log scores when predictive probabilities are the primary concern. Present the criterion, the parameter counts or validation design, and the size of the score difference. For nonnested ordinal alternatives, these are comparisons of fit or prediction—not ordinary likelihood-ratio chi-square tests.