Likelihood for Individual Ordinal Responses

From Cumulative Probabilities to a Likelihood

For an ordinal response, the categories have a meaningful order, but the observed response for each individual is one category, not a cumulative event. A cumulative link model specifies probabilities such as the chance of being in category $j$ or below. To evaluate the likelihood of observed data, we first convert those cumulative probabilities into probabilities for the individual categories, then multiply the contributions across observations under the model’s independence assumptions.

Let $Y_i$ take one of the ordered categories $1,\ldots,J$, and let $x_i$ denote the covariates for observation $i$. Write the model’s conditional cumulative probabilities as $C_{ij}(\theta)=P_\theta(Y_i\leq j\mid x_i)$, for $j=1,\ldots,J-1$. The endpoint probabilities are set by convention: $C_{i0}(\theta)=0$ and $C_{iJ}(\theta)=1$. The category probability is the difference between two adjacent cumulative probabilities.

Definition: For observation $i$, the conditional probability of category $k$ is $$ \pi_{ik}(\theta)=P_\theta(Y_i=k\mid x_i) =C_{ik}(\theta)-C_{i,k-1}(\theta),\qquad k=1,\ldots,J. $$ For observed responses $y_1,\ldots,y_n$, the individual-response likelihood is the joint conditional probability of those responses, viewed as a function of $\theta$. Under conditional independence given the full covariate design, and the assumption that each conditional response distribution depends only on its own $x_i$, it is $$ L(\theta)=\prod_{i=1}^n \pi_{i,y_i}(\theta). $$

The condition that the cumulative probabilities are ordered, $0=C_{i0}(\theta)\leq C_{i1}(\theta)\leq\cdots\leq C_{i,J-1}(\theta)\leq C_{iJ}(\theta)=1$, ensures that every category probability is nonnegative. The category probabilities also sum to one: the adjacent differences telescope from zero to one. Thus, the model supplies a valid conditional probability distribution for each response.

Worked Example: Likelihood From Three Cumulative Probabilities

Suppose each response has three categories, and for each of three individuals the model gives $C_{i1}=0.20$ and $C_{i2}=0.70$. The observed responses are $y_1=1$, $y_2=2$, and $y_3=3$. Assume the responses are conditionally independent given the covariates, and each individual’s conditional distribution depends only on that individual’s covariates.

The category probabilities for any of these individuals are $$ \pi_{i1}=C_{i1}=0.20,\qquad \pi_{i2}=C_{i2}-C_{i1}=0.70-0.20=0.50,\qquad \pi_{i3}=1-C_{i2}=1-0.70=0.30. $$

The observed first response contributes $0.20$, the second contributes $0.50$, and the third contributes $0.30$. Therefore, $$ L(\theta)=0.20(0.50)(0.30)=0.030. $$ The value $0.030$ is the joint probability of this particular response pattern under the specified conditional probabilities and independence assumption.

Likelihood and Log-Likelihood

The likelihood combines one category probability for each individual: the contribution is the probability of the category actually observed, not the probability of every category and not the cumulative probability at the observed category. The log-likelihood is obtained by taking the logarithm of the product. When all observed category probabilities are positive, the logarithm changes the product into a sum, which is often easier to analyze and optimize.

Result: Under the conditional-independence and individual-covariate assumptions in the definition, $$ \ell(\theta)=\log L(\theta) =\sum_{i=1}^n \log \pi_{i,y_i}(\theta) =\sum_{i=1}^n \log\!\left\{C_{i,y_i}(\theta)-C_{i,y_i-1}(\theta)\right\}, $$ provided every observed category has positive probability. If at least one observed category has probability zero, then $L(\theta)=0$ and $\ell(\theta)=-\infty$, using the extended-real convention $\log 0=-\infty$.

The same expression can be written with indicators. Define $I(Y_i=k)$ to equal one if $Y_i=k$ and zero otherwise. For any observed data set, $$ \ell(\theta)=\sum_{i=1}^n\sum_{k=1}^J I(y_i=k)\log\pi_{ik}(\theta). $$ For each individual, exactly one indicator equals one, so only the log probability of the observed category remains. If a zero probability has a zero indicator, its term contributes nothing; if the observed category has probability zero, its term is $-\infty$.

Obtain the cumulative probabilities.
For each individual, evaluate the model’s cumulative probabilities at the observed covariates and include the endpoints zero and one.

Difference adjacent values.
For the observed category $y_i$, calculate $C_{i,y_i}(\theta)-C_{i,y_i-1}(\theta)$.

Combine contributions.
Multiply the observed-category probabilities for the likelihood, or add their logarithms for the log-likelihood, using the stated conditional-independence assumptions.

Worked Example: A Cumulative Logit Likelihood

Consider a three-category cumulative logit model using the convention $\operatorname{logit}\{C_{ij}(\theta)\}=\alpha_j-\beta x_i$, where $\operatorname{logit}(p)=\log\{p/(1-p)\}$. Take $\alpha_1=\log(0.20/0.80)$, $\alpha_2=\log(0.70/0.30)$, and $\beta=0.40$. These cutpoints satisfy $\alpha_1<\alpha_2$, so the cumulative probabilities are ordered. Three independent responses have covariate values and outcomes $(x_1,y_1)=(0,1)$, $(x_2,y_2)=(1,3)$, and $(x_3,y_3)=(1,2)$.

At $x=0$, the cumulative probabilities are $C_1=0.20$ and $C_2=0.70$, since the specified cutpoints are their logits. Thus, the probability of category 1 for the first individual is $0.20$.

At $x=1$, the two cumulative logits are $$ \alpha_1-\beta=-1.3863-0.40=-1.7863,\qquad \alpha_2-\beta=0.8473-0.40=0.4473. $$ Applying the inverse logit gives $C_1\approx0.1435$ and $C_2\approx0.6100$. Therefore, for either individual with $x=1$, $$ \pi_2\approx0.6100-0.1435=0.4665,\qquad \pi_3\approx1-0.6100=0.3900. $$ The second individual’s observed category is 3, and the third individual’s is 2. Conditional independence therefore gives $$ L(\theta)\approx0.20(0.3900)(0.4665)=0.036387. $$ The log-likelihood is $$ \ell(\theta)\approx\log(0.20)+\log(0.3900)+\log(0.4665) \approx-1.6094-0.9416-0.7625=-3.3135. $$ This calculation uses a different cumulative distribution at $x=1$ than at $x=0$; each likelihood contribution is evaluated at that individual’s covariate value.

Why the Factorization Holds

The product form is not just a convenient way to write the likelihood. It follows from conditional independence, but the conditioning and covariate assumptions must be stated precisely. Let the full observed covariate design be $X=(x_1,\ldots,x_n)$. Assume that, conditional on $X$, the responses $Y_1,\ldots,Y_n$ are mutually independent. Also assume the individual-specific conditional distribution property $P_\theta(Y_i=k\mid X)=P_\theta(Y_i=k\mid x_i)$ for every $i$ and category $k$. Then the joint conditional probability factors into the individual contributions.

Derivation: For any fixed response vector $y=(y_1,\ldots,y_n)$, conditional independence given the full design yields $$ P_\theta(Y_1=y_1,\ldots,Y_n=y_n\mid X) =\prod_{i=1}^n P_\theta(Y_i=y_i\mid X). $$ The individual-covariate property replaces each factor by its probability conditional on $x_i$, so $$ P_\theta(Y_1=y_1,\ldots,Y_n=y_n\mid X) =\prod_{i=1}^n P_\theta(Y_i=y_i\mid x_i) =\prod_{i=1}^n\pi_{i,y_i}(\theta). $$ Viewed as a function of $\theta$, this is $L(\theta)$. Taking logarithms gives the sum of the individual log contributions whenever the joint probability is positive. $\square$

Conditional independence given $X$ alone does not imply that the distribution of $Y_i$ given $X$ depends only on $x_i$. For example, an individual’s response distribution could depend on another person’s covariate even if the responses are conditionally independent. In that case a product factorization may still hold given $X$, but its factors must be written as $P_\theta(Y_i=y_i\mid X)$; they cannot automatically be replaced by probabilities conditional only on $x_i$. The usual individual-response cumulative link specification includes the individual-covariate property explicitly.

Worked Example: Comparing Two Parameter Values

Suppose three independent individuals have observed categories $1,2,3$, respectively. Under parameter value $\theta_A$, their probabilities for the observed categories are $0.40,0.50,0.40$. Under $\theta_B$, those probabilities are $0.50,0.30,0.50$. These can arise from valid ordered cumulative probabilities: for the category-2 observation, the respective cumulative pairs may be $(0.20,0.70)$ and $(0.30,0.60)$; the category-1 probabilities can be first cumulative probabilities $0.40$ and $0.50$, while the category-3 probabilities can be one minus second cumulative probabilities $0.40$ and $0.50$.

The likelihoods for the same observed response vector are $$ L(\theta_A)=0.40(0.50)(0.40)=0.080,\qquad L(\theta_B)=0.50(0.30)(0.50)=0.075. $$ Their log-likelihoods are $\ell(\theta_A)=\log(0.080)\approx-2.526$ and $\ell(\theta_B)=\log(0.075)\approx-2.590$. Thus, these observations have a slightly greater likelihood under $\theta_A$. This comparison concerns the likelihood of the observed data under the two parameter values; it does not by itself establish that either parameter value is a good model for other data.

A Basic Probability Check

The adjacent-difference construction is valid whenever the cumulative probabilities are ordered. Conversely, if the category probabilities form a valid distribution, their cumulative sums necessarily are ordered. This equivalence is a useful check on both a model specification and a likelihood calculation.

Result: For a fixed individual, numbers $C_0=0,C_1,\ldots,C_{J-1},C_J=1$ define category probabilities $\pi_k=C_k-C_{k-1}$ that are nonnegative and sum to one if and only if $0=C_0\leq C_1\leq\cdots\leq C_J=1$.

Proof. First suppose the cumulative values are ordered. Then $C_k-C_{k-1}\geq0$ for every $k$, and $$ \sum_{k=1}^J\pi_k=\sum_{k=1}^J(C_k-C_{k-1})=C_J-C_0=1. $$ Thus the differences are a probability distribution. Conversely, suppose $\pi_k\geq0$ for all $k$ and $\sum_{k=1}^J\pi_k=1$. Defining $C_j=\sum_{k=1}^j\pi_k$ gives $C_j-C_{j-1}=\pi_j\geq0$, so the $C_j$ are nondecreasing; also $C_0=0$ and $C_J=1$. $\square$

Common Pitfalls and Interpretation

Using a cumulative probability as a category probability. If $Y_i=2$, its contribution is $C_{i2}-C_{i1}$, not $C_{i2}$. The cumulative probability $C_{i2}$ includes both categories 1 and 2.
Factoring with the wrong conditioning statement. Conditional independence given the full design $X$ gives a product of terms conditional on $X$. To write each term conditional only on $x_i$, also assume that each individual’s response distribution given $X$ depends only on $x_i$.
Taking the log before checking probabilities. A negative adjacent difference signals unordered cumulative probabilities and is not a valid category probability. A zero probability assigned to an observed category makes the likelihood zero and the log-likelihood $-\infty$.
Multiplying probabilities for categories that were not observed. Each individual contributes one probability: the probability of that individual’s observed category.

Key takeaway: For each observed ordinal response, difference the cumulative probability at its category and the cumulative probability immediately below it. Multiply those probabilities under the stated conditional-independence and covariate assumptions, or sum their logarithms to obtain the log-likelihood.

Check Your Understanding

Use adjacent differences and the stated assumptions to answer these questions.

For a four-category response with cumulative probabilities $C_1=0.15$, $C_2=0.55$, and $C_3=0.85$, what is the probability of category 3?
Two independent observations have observed-category probabilities $0.25$ and $0.60$. Find their joint likelihood and log-likelihood.
State the two assumptions needed to replace a product of probabilities conditional on the full covariate design by terms conditional on each individual’s covariates.
What is the log-likelihood contribution when an observed category has probability zero?
Explain why the probability of category $k$ is not generally the cumulative probability $P(Y_i\leq k\mid x_i)$.