Individual And Grouped Multinomial Likelihoods

From Category Records to Likelihoods

A multinomial logit model assigns probabilities to a set of possible response categories. The observed data may be stored as one category label per subject, or summarized as counts in each category. These are two data formats for the same model when the observations being grouped share a category-probability vector. The distinction matters: a likelihood for one particular ordered sequence is not numerically the same as the probability of the entire count pattern.

Let $Y_i$ denote the response for observation $i$, taking one of $K$ categories $1,\ldots,K$. Under a model with parameter vector $\theta$, write $p_{ij}(\theta)=P(Y_i=j)$, where $\sum_{j=1}^K p_{ij}(\theta)=1$. These probabilities may depend on the observation's covariates, for example through the softmax map. Throughout, assume the $Y_i$ are independent. For likelihood formulas involving logarithms, we consider parameter values at which the probabilities of the observed categories are positive; the product likelihood itself also makes sense when some probabilities are zero.

Definition: For an individual categorical response $Y_i$, define $y_{ij}=1$ if $Y_i=j$, and $y_{ij}=0$ otherwise. Thus exactly one of $y_{i1},\ldots,y_{iK}$ equals 1. The individual-response likelihood for independent observations is $$ L(\theta)=\prod_{i=1}^n\prod_{j=1}^K p_{ij}(\theta)^{y_{ij}}. $$ The likelihood is the probability of the observed labeled responses, expressed as a function of $\theta$.

In this expression, the indicator for the observed category selects its probability: if observation $i$ falls in category $r$, its contribution is $p_{ir}(\theta)$, because the other factors are raised to the zero power. Independence is what permits the probabilities for separate observations to be multiplied.

Likelihood for Grouped Category Counts

Suppose a group contains $m$ independent observations, each with the same category probabilities $p_1(\theta),\ldots,p_K(\theta)$. Let $N_j$ be the number in category $j$, so $\sum_{j=1}^K N_j=m$. The probability of one particular ordered sequence with these counts is $\prod_j p_j(\theta)^{N_j}$. But the observed count vector does not specify an ordering. There are $m!/\prod_j N_j!$ ordered sequences with those counts.

Definition: A count vector $(N_1,\ldots,N_K)$ from $m$ independent categorical observations with a common probability vector $(p_1(\theta),\ldots,p_K(\theta))$ has the multinomial likelihood $$ L_{\mathrm{count}}(\theta) = \frac{m!}{\prod_{j=1}^K N_j!} \prod_{j=1}^K p_j(\theta)^{N_j}. $$ The factor $m!/\prod_j N_j!$ is the multinomial coefficient: it counts how many ordered response sequences have the observed category counts.

Theorem: If $m$ responses are independent and share the same category probabilities $p_1(\theta),\ldots,p_K(\theta)$, then their count vector has probability $$ P(N_1=n_1,\ldots,N_K=n_K) = \frac{m!}{\prod_{j=1}^K n_j!} \prod_{j=1}^K p_j(\theta)^{n_j}, \qquad \sum_{j=1}^K n_j=m. $$ Proof: Fix a count vector $(n_1,\ldots,n_K)$. Independence and the common probability vector imply that any particular ordered sequence with these counts has probability $\prod_j p_j(\theta)^{n_j}$. To count such sequences, choose the $n_1$ positions for category 1, then the $n_2$ positions for category 2 from the remaining positions, and continue. The product of these choices is $$ \binom{m}{n_1}\binom{m-n_1}{n_2}\cdots = \frac{m!}{n_1!n_2!\cdots n_K!}. $$ The events corresponding to distinct ordered sequences are disjoint. Summing their equal probabilities therefore gives the stated count-vector probability. $\square$

The theorem requires a common probability vector as well as independence. Independence alone is not sufficient for a multinomial count distribution. If probabilities vary across observations, different sequences with the same counts can have different probabilities, so the simple “number of sequences times one sequence probability” argument no longer works.

Worked Example: One Ordered Sequence

There are three categories with probabilities $p_1(\theta)=0.5$, $p_2(\theta)=0.3$, and $p_3(\theta)=0.2$. Four independent responses are observed in the order $1,2,1,3$. Find the likelihood for this particular ordered data.

The observed categories contribute probabilities $p_1,p_2,p_1,p_3$, respectively. Thus $$ L= p_1^2p_2p_3 =(0.5)^2(0.3)(0.2) =0.25(0.06) =0.015. $$ This is the probability of the particular sequence $1,2,1,3$. It is not yet the probability of observing two category-1 responses, one category-2 response, and one category-3 response in any order.

Worked Example: A Multinomial Count Likelihood

Eight independent responses share the category probabilities $(0.25,0.5,0.25)$, and the observed counts are $(4,3,1)$. Find the likelihood for the count vector.

The multinomial coefficient is $$ \frac{8!}{4!3!1!} =\frac{40320}{24\cdot 6\cdot 1} =280. $$ The probability of one particular sequence with these counts is $$ (0.25)^4(0.5)^3(0.25)^1 =(0.25)^5(0.5)^3 =\frac{1}{1024}\cdot\frac{1}{8} =\frac{1}{8192}. $$ There are 280 such sequences, so the count likelihood is $$ 280\left(\frac{1}{8192}\right) =\frac{35}{1024} \approx 0.03418. $$ The likelihood of a particular ordering is $1/8192$; the likelihood of the count event is $35/1024$. They differ because the latter includes every ordering with the stated counts.

Why the Two Likelihood Forms Encode the Same Model

For a common probability vector, an individual data set determines its count vector. Its labeled-data likelihood contains the product of category probabilities raised to their counts. The count likelihood contains exactly that same product, multiplied by a factor that depends on the counts but not on $\theta$. Consequently, the two likelihoods describe the same parameter-dependent category-probability model, though they assign different probabilities to different data events.

Theorem: Suppose $m$ independent labeled responses share category probabilities $p_j(\theta)$, and their observed counts are $n_j$. Let $L_{\mathrm{seq}}(\theta)$ be the likelihood of the particular observed sequence and $L_{\mathrm{count}}(\theta)$ the likelihood of the count vector. Then $$ L_{\mathrm{seq}}(\theta)=\prod_{j=1}^K p_j(\theta)^{n_j}, \qquad L_{\mathrm{count}}(\theta) = \frac{m!}{\prod_{j=1}^K n_j!}L_{\mathrm{seq}}(\theta). $$ The multiplier does not depend on $\theta$. Hence, for any two parameter values $\theta_a,\theta_b$ for which the likelihoods are positive, $$ \frac{L_{\mathrm{count}}(\theta_a)}{L_{\mathrm{count}}(\theta_b)} = \frac{L_{\mathrm{seq}}(\theta_a)}{L_{\mathrm{seq}}(\theta_b)}. $$ Proof: Each observed response contributes the probability of its category. Since category $j$ occurs $n_j$ times, multiplication gives $L_{\mathrm{seq}}(\theta)=\prod_j p_j(\theta)^{n_j}$. By the count-vector theorem, the probability of the count event is this sequence probability multiplied by $m!/\prod_j n_j!$. That coefficient is fixed by the observed counts and is independent of $\theta$. It therefore cancels from the displayed ratio, proving the result. $\square$

This result means that likelihood comparisons across parameter values are unchanged by switching between the labeled sequence and count representations, provided the observations share the same probability vector and the same observed counts are being used. It does not mean that the two event probabilities are equal. One concerns a specific ordering; the other combines all possible orderings.

Worked Example: Grouping Within Covariate Profiles

Suppose six independent responses come from two covariate profiles, with three observations at each profile. At profile A, the category probabilities are $(0.6,0.3,0.1)$, and the counts are $(2,1,0)$. At profile B, the probabilities are $(0.2,0.5,0.3)$, and the counts are $(0,1,2)$. Find the likelihood for the grouped observations.

Within profile A, all three responses share one probability vector, so the count likelihood is $$ \frac{3!}{2!1!0!}(0.6)^2(0.3)^1(0.1)^0 =3(0.36)(0.3) =0.324. $$ Within profile B, the count likelihood is $$ \frac{3!}{0!1!2!}(0.2)^0(0.5)^1(0.3)^2 =3(0.5)(0.09) =0.135. $$ The two profiles contain disjoint independent observations, so their likelihood contributions multiply: $$ L=0.324(0.135)=0.04374. $$ The same calculation from a specific ordering within each profile gives $0.108(0.045)=0.00486$. The two multinomial coefficients contribute $3\cdot3=9$, and $9(0.00486)=0.04374$, as required.

Grouping When Probabilities Differ

In a regression model, observations with different covariates will generally have different category probabilities. Grouping is then appropriate within sets of observations that share a probability vector, such as identical covariate profiles under the model. If profiles are indexed by $g$, let $m_g$ be the number of observations in profile $g$, and let $n_{gj}$ be its count in category $j$. Under independence, the grouped likelihood is $$ L(\theta) = \prod_g \left[ \frac{m_g!}{\prod_{j=1}^K n_{gj}!} \prod_{j=1}^K p_{gj}(\theta)^{n_{gj}} \right], \qquad \sum_{j=1}^K n_{gj}=m_g. $$ Each factor is a multinomial likelihood for one profile. The product is valid because the groups consist of disjoint independent observations.

By contrast, summing counts across profiles and treating the total as one multinomial count vector generally loses the probability structure. In the worked example, the two profiles have different category probabilities. A particular sequence of category labels across all six observations has probability determined by which profile produced each response; total counts alone do not supply one common probability vector for the multinomial formula.

Key takeaway: Use a multinomial likelihood for a group of independent categorical observations only when all members of that group share the same category-probability vector. For individual records, multiply their observed-category probabilities. When grouping is valid, the count likelihood adds the parameter-free multinomial coefficient to the corresponding individual likelihood.

Common Mistake

Wrong reasoning: “The observations are independent, so their category counts must have a multinomial distribution. I can add counts from all covariate profiles and use one pooled probability vector.”

Independence justifies multiplying the probabilities for a fully specified sequence; it does not make those probabilities identical across observations. The multinomial count formula also requires a common category-probability vector. If profiles have different vectors, the probability of a count pattern must account for how categories are allocated among the profiles. Group within common-probability profiles and multiply those group likelihoods, or retain the individual-response likelihood.

Check Your Understanding

Use the distinction between a particular sequence and a count event to answer the following.

For five independent responses with a common three-category probability vector, what factor converts the likelihood of a particular sequence with counts $(2,2,1)$ to the likelihood of the count vector?
Why does the count likelihood include a multinomial coefficient when the likelihood of a particular labeled sequence does not?
State the assumptions needed to use a multinomial likelihood for a group of categorical observations.
Two covariate profiles have different category-probability vectors. Explain why their observations can be grouped within profiles but should not generally be pooled into one multinomial count vector.
If two likelihood representations differ only by a factor fixed by the observed counts, what happens to their likelihood ratio across two parameter values?