Multinomial Models for Nominal Responses

Why Use a Multinomial Model?

Some outcomes have more than two possible values, but their categories have no natural order. A hypothetical treatment-choice outcome might be “oral,” “injection,” or “no treatment”; a hypothetical adverse-event classification might contain several distinct event types. Treating these responses as ordinal would impose an ordering that the categories do not have. Fitting separate binary models, meanwhile, can yield probabilities that do not form a coherent distribution across all categories.

A baseline-category multinomial model handles the categories jointly. It compares each non-reference category with one selected reference category, while a normalization step converts all comparisons into probabilities that sum to one. In a mixed-effects version, subject-specific random effects allow those category comparisons to vary between subjects.

Definition: Let $Y_{ij}$ be a nominal response for subject $i$ at occasion $j$, with $K$ mutually exclusive and collectively exhaustive categories. Select category $K$ as the reference. For each $k=1,\ldots,K-1$, a baseline-category multinomial mixed-effects model specifies $$ \log\left(\frac{P(Y_{ij}=k\mid b_i)}{P(Y_{ij}=K\mid b_i)}\right) =\eta_{ijk} =x_{ij}^{T}\beta_k+z_{ij}^{T}b_{ik}. $$ Here $x_{ij}$ and $z_{ij}$ are covariate vectors, $\beta_k$ is the fixed-effect coefficient vector for category $k$, and $b_{ik}$ is the subject-specific random-effect vector for that category contrast. The reference category has linear predictor $\eta_{ijK}=0$. Random effects may be specified jointly, commonly as a multivariate normal vector with mean zero and covariance matrix $\Omega$.

From Category Comparisons to Probabilities

The model’s linear predictors are log-odds relative to the reference category, not probabilities themselves. To obtain probabilities, exponentiate each non-reference predictor to get its odds relative to category $K$. The reference category has relative weight one. Dividing every weight by their sum gives the conditional probability for each category:

P(Y_{ij}=k\mid b_i) =\frac{\exp(\eta_{ijk})} {1+\sum_{\ell=1}^{K-1}\exp(\eta_{ij\ell})}, \quad k=1,\ldots,K-1, \qquad P(Y_{ij}=K\mid b_i) =\frac{1} {1+\sum_{\ell=1}^{K-1}\exp(\eta_{ij\ell})}.

These are conditional, or subject-specific, probabilities: they condition on the subject’s random effects as well as the covariates. Their sum is one because the common denominator is the sum of all $K$ relative weights. The marginal probability for a new subject is different: it averages these conditional probabilities over the random-effects distribution. In general, that average is not obtained by setting $b_i=0$, because the probability transformation is nonlinear.

Justification: Write the conditional probabilities as $p_k$, and let $p_K$ be the reference-category probability. The model gives $p_k/p_K=\exp(\eta_{ijk})$, so $p_k=p_K\exp(\eta_{ijk})$. Since probabilities sum to one, $p_K+\sum_{k=1}^{K-1}p_k=1$. Substituting for $p_k$ yields $p_K[1+\sum_{k=1}^{K-1}\exp(\eta_{ijk})]=1$. Solving gives the displayed probability formula, and substitution then gives every $p_k$.

Interpreting the Model

For a continuous covariate $x$, the coefficient $\beta_{kr}$ is the change in the log-odds of category $k$ versus category $K$ for a one-unit increase in that covariate, holding the other covariates and random effects fixed. Its exponent, $\exp(\beta_{kr})$, is the corresponding conditional odds ratio. The units of the covariate matter: if dose is entered in 10 mg increments, its coefficient describes a 10 mg increase.

The model does not require one common covariate effect across the non-reference categories. Each category contrast has its own coefficient vector. To compare two non-reference categories $k$ and $m$, subtract their log-odds contrasts:

\log\left(\frac{P(Y_{ij}=k\mid b_i)}{P(Y_{ij}=m\mid b_i)}\right) =\eta_{ijk}-\eta_{ijm}.

Thus, a covariate’s effect on the odds of $k$ versus $m$ is the difference between its coefficients in those two category-specific predictors. Choosing a different reference category changes the coefficient parameterization and its interpretation, but does not by itself change the model’s fitted probabilities when the model is re-expressed equivalently.

Worked Example: Calculating Three Category Probabilities

Suppose a hypothetical response has categories A, B, and C, with C as the reference. For a particular covariate pattern and specified random effects, suppose the fitted predictors are $\eta_A=0.7$ and $\eta_B=-0.4$. Calculate all three conditional probabilities.

The relative weights for A, B, and C are $\exp(0.7)$, $\exp(-0.4)$, and $1$. Numerically, these are approximately $2.0138$, $0.6703$, and $1$. The denominator is $2.0138+0.6703+1=3.6841$.

Therefore, $P(A\mid b_i)=2.0138/3.6841\approx0.5466$, $P(B\mid b_i)=0.6703/3.6841\approx0.1820$, and $P(C\mid b_i)=1/3.6841\approx0.2714$. The probabilities sum to approximately $1.0000$, allowing for rounding.

These probabilities describe the subject with the stated predictors and random effects. They are not population-averaged probabilities unless the random-effects distribution has been integrated over.

Random Effects and Subject-Specific Comparisons

A random intercept for category $k$ shifts that subject’s log-odds of category $k$ relative to the reference. A random slope allows the subject-specific association between a covariate and that contrast to vary. Because multiple category contrasts may be related, their random effects can be modeled jointly, with covariance parameters in $\Omega$. The random-effects structure should reflect the intended sources of between-subject heterogeneity while remaining estimable from the data.

For observations from the same subject, the shared random effects induce dependence through their common subject-specific probabilities. Conditional on the random effects and modeled covariates, a categorical observation has the probabilities given above. The assumption that conditional responses are independent across occasions, if used to form a subject likelihood, is an additional modeling assumption; it should not be confused with marginal independence after the random effects are integrated out.

Worked Example: A Category-Specific Random Intercept

Consider a hypothetical three-category outcome with category 3 as the reference. For one subject, let the predictors be $\eta_1=b_i$ and $\eta_2=-1$, where $b_i$ is a random intercept in the category 1 versus category 3 contrast. First calculate the probabilities when $b_i=-0.8$.

The relative weights are $\exp(-0.8)=0.449328964$, $\exp(-1)=0.367879441$, and $1$. The denominator is $1+0.449328964+0.367879441=1.817208405$, or approximately $1.8172084$. Thus the probabilities for categories 1, 2, and 3 are approximately $0.2473$, $0.2024$, and $0.5503$, respectively.

For comparison, setting $b_i=0$ gives predictors $0$ and $-1$. The weights are $1$, $0.367879441$, and $1$, with denominator $2.367879441$. The resulting probabilities are approximately $0.4223$, $0.1554$, and $0.4223$.

The random intercept changes the first category’s odds relative to category 3 by a factor of $\exp(-0.8)\approx0.4493$, while leaving the category 2 versus category 3 odds unchanged. The probabilities for all categories still change because they share the same normalizing denominator.

Worked Example: Category-Specific Dose Effects

Suppose a hypothetical model describes response categories A, B, and C, with C as the reference. Let dose be measured in 10 mg increments and let $r$ be a renal-function covariate measured in its stated index units. For a subject with random effects set to zero, suppose

\eta_A=-1+0.8(\text{dose})-0.5r, \qquad \eta_B=0.2-0.3(\text{dose})+0.4r.

At dose $=1$ (10 mg) and $r=1$, $\eta_A=-1+0.8-0.5=-0.7$, and $\eta_B=0.2-0.3+0.4=0.3$. The weights for A, B, and C are $\exp(-0.7)=0.4966$, $\exp(0.3)=1.3499$, and $1$. Their exact sum is $\exp(-0.7)+\exp(0.3)+1\approx2.846444$, which rounds to $2.8464$. Hence $P(A\mid b_i=0)\approx0.1745$, $P(B\mid b_i=0)\approx0.4742$, and $P(C\mid b_i=0)\approx0.3513$.

Holding renal function and random effects fixed, a 10 mg dose increase multiplies the odds of A versus C by $\exp(0.8)\approx2.226$, while multiplying the odds of B versus C by $\exp(-0.3)\approx0.7408$. For A versus B, the dose coefficient is $0.8-(-0.3)=1.1$, so the conditional odds of A versus B are multiplied by $\exp(1.1)\approx3.004$ per 10 mg increase. These are conditional odds comparisons, not direct multipliers of the category probabilities.

Assumptions, Diagnostics, and Alternatives

The model is appropriate when each response belongs to exactly one of a specified set of unordered categories. The categories must be defined consistently, and the selected reference category must be meaningful enough to support clear reporting. The covariate effects enter the category-specific log-odds as specified; the random-effects distribution and covariance structure are also assumptions, not consequences of using a multinomial likelihood. As in other mixed-effects models, estimation relies on the data being informative enough to distinguish fixed effects, random-effect variation, and residual sampling variation.

Signs of a poorly supported or failing model include convergence problems, estimates with very large magnitude, large standard errors, or random-effect variances near a boundary. Very sparse categories or covariate patterns can produce unstable estimates and separation-like behavior, where a predictor nearly or completely distinguishes a category contrast. Check category counts and covariate coverage before interpreting a precise-looking coefficient. Also verify that fitted probabilities are finite, lie between zero and one, and sum to one for each evaluated covariate and random-effects pattern.

The baseline-category model is the natural choice for unordered categories and allows category-specific effects. A cumulative-link model is preferable when the response has a substantively meaningful order and its ordinal structure is part of the question. For a two-category outcome, the multinomial model reduces to a binary logistic model. These alternatives answer different questions: nominal categories should not be assigned an artificial order simply to use an ordinal model.

A common mistake is to exponentiate each category-specific predictor and report $\exp(\eta_k)$ as the probability of category $k$. It is instead the odds of category $k$ relative to the reference category. The probability is that weight divided by the sum of all category weights, including the reference weight of one. Another mistake is to interpret the probabilities at $b_i=0$ as population averages. They describe a subject at the zero random effect; population averaging requires integration over the random-effects distribution.

Key takeaway: A baseline-category multinomial mixed-effects model specifies a separate log-odds contrast for each non-reference category. Exponentiated predictors are relative odds; normalize all category weights to obtain a complete, coherent set of conditional probabilities.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

With three categories and category 3 as reference, the predictors are $\eta_1=0$ and $\eta_2=\log(2)$. What are the probabilities of categories 1, 2, and 3?
A dose coefficient for category A versus the reference is $-0.4$ per 5 mg. Interpret its exponentiated value as a conditional odds ratio.
Why is an ordinal cumulative-link model not automatically appropriate for “oral,” “injection,” and “no treatment” categories?
What changes, and what should remain unchanged, if an equivalent model is reparameterized using a different reference category?

Solutions. 1. The weights are $1$, $2$, and $1$, with denominator $4$, so the probabilities are $0.25$, $0.50$, and $0.25$. 2. $\exp(-0.4)\approx0.6703$: conditional on other covariates and random effects, a 5 mg increase multiplies the odds of A versus the reference by about $0.6703$, a decrease of about 33.0%. 3. Those categories have no inherent order; imposing one would add an unsupported structure to the response. 4. The coefficient contrasts and their interpretations change with the reference, while fitted probabilities for the categories should remain unchanged under an equivalent reparameterization.