Deriving The Multinomial Logit Score

Why the Score Has an Observed-Minus-Fitted Form

A multinomial logit model assigns a probability to every response category, while its regression coefficients determine how those probabilities vary with predictors. The score records how the log-likelihood changes when a coefficient is perturbed. Its useful structure is simple to state: each category-specific score block is a sum of design vectors weighted by observed category counts minus fitted category counts.

The formula applies to individual responses and to grouped multinomial counts. In the grouped case, the fitted count is the group size multiplied by the fitted category probability; using a single indicator in place of the observed count would give the wrong contribution.

Model and Log-Likelihood

Let there be $J$ response categories, with category $J$ chosen as the baseline. For observation or group $i$, write $y_{ij}$ for the number of responses in category $j$, and $n_i=\sum_{j=1}^J y_{ij}$ for the total number of responses. For an individual response, $n_i=1$ and each $y_{ij}$ is an indicator.

For each nonbaseline category $j=1,\ldots,J-1$, let $x_{ij}$ be its design vector and $\beta_j$ its coefficient vector. The linear predictor is $\eta_{ij}=x_{ij}^{T}\beta_j$; the baseline predictor is set to zero. Define $D_i=1+\sum_{h=1}^{J-1}\exp(\eta_{ih})$. The fitted category probabilities are $\pi_{ij}=\exp(\eta_{ij})/D_i$ for $j<J$, and $\pi_{iJ}=1/D_i$.

Definition: The score block for $\beta_j$ is the vector of partial derivatives of the log-likelihood with respect to the entries of $\beta_j$, holding the other coefficient blocks fixed. The full score is the vector formed by stacking these blocks for $j=1,\ldots,J-1$.

For independent grouped multinomial observations, the log-likelihood, apart from no omitted parameter-dependent terms, can be written as

\ell(\beta)=\sum_i\left\{\sum_{j=1}^{J-1}y_{ij}\eta_{ij}-n_i\log D_i\right\}+C,

where $C=\sum_i\log\{n_i!/\prod_{j=1}^{J}y_{ij}!\}$ for grouped counts, and $C=0$ for individual categorical observations. The constant does not depend on the regression coefficients, so it contributes zero to the score.

First Derivation: Differentiating the Softmax

The derivative of a category probability with respect to a nonbaseline linear predictor explains why the score separates into a residual for each category. For any $k\in\{1,\ldots,J\}$ and $j<J$, the derivative is

\frac{\partial \pi_{ik}}{\partial \eta_{ij}} =\pi_{ik}\bigl(\mathbf{1}\{k=j\}-\pi_{ij}\bigr).

Here $\mathbf{1}\{k=j\}$ equals one when $k=j$ and zero otherwise. To verify the identity for $k<J$, differentiate $\pi_{ik}=\exp(\eta_{ik})/D_i$. The numerator has derivative $\mathbf{1}\{k=j\}\exp(\eta_{ik})$, and $\partial D_i/\partial\eta_{ij}=\exp(\eta_{ij})$. The quotient rule therefore gives

\frac{\partial \pi_{ik}}{\partial \eta_{ij}} =\frac{\mathbf{1}\{k=j\}\exp(\eta_{ik})D_i-\exp(\eta_{ik})\exp(\eta_{ij})}{D_i^2} =\pi_{ik}\bigl(\mathbf{1}\{k=j\}-\pi_{ij}\bigr).

For $k=J$, $\pi_{iJ}=1/D_i$, so its derivative is $-\exp(\eta_{ij})/D_i^2=-\pi_{iJ}\pi_{ij}$, which is the same formula because $\mathbf{1}\{J=j\}=0$. This establishes the identity for every category. In particular, increasing one nonbaseline predictor raises its own probability and lowers the probabilities of the other categories.

Main Result: The Category-Specific Score

Theorem: Suppose the grouped response vectors are independent multinomial observations with totals $n_i$, and the fitted probabilities follow the baseline-category softmax model above. If each $\eta_{ij}=x_{ij}^{T}\beta_j$, then for every nonbaseline category $j$, $$ U_j(\beta)=\frac{\partial \ell}{\partial\beta_j} =\sum_i x_{ij}\bigl(y_{ij}-n_i\pi_{ij}\bigr). $$ For individual responses, this reduces to $\sum_i x_{ij}(\mathbf{1}\{Y_i=j\}-\pi_{ij})$.

Proof. Fix an observation $i$ and a nonbaseline category $j$. Since only the term $y_{ij}\eta_{ij}$ in the linear-predictor sum depends directly on $\eta_{ij}$, its derivative is $y_{ij}$. Also, $\partial \log D_i/\partial\eta_{ij}=\exp(\eta_{ij})/D_i=\pi_{ij}$. Thus the derivative of the $i$th log-likelihood contribution with respect to $\eta_{ij}$ is

\frac{\partial \ell_i}{\partial\eta_{ij}}=y_{ij}-n_i\pi_{ij}.

The predictor is linear in its coefficient vector, so $\partial\eta_{ij}/\partial\beta_j=x_{ij}$. Applying the chain rule gives $\partial\ell_i/\partial\beta_j=x_{ij}(y_{ij}-n_i\pi_{ij})$. Summing these contributions over independent observations gives the stated block. For an individual response, $n_i=1$ and $y_{ij}=\mathbf{1}\{Y_i=j\}$, yielding the individual-data expression. $\square$

The derivation also shows exactly where the fitted count enters: in a group it is $n_i\pi_{ij}$, not merely $\pi_{ij}$. The design vector weights the discrepancy because changing a coefficient changes the predictor in proportion to that vector.

Worked Examples

Worked Example: One Individual Response

There are three categories, with category 3 as baseline. For one individual, let $x_{i1}=x_{i2}=1$, $\beta_1=\log 2$, and $\beta_2=0$. The predictors are $\eta_{i1}=\log 2$ and $\eta_{i2}=0$, so $D_i=2+1+1=4$ and the fitted probabilities are $(\pi_{i1},\pi_{i2},\pi_{i3})=(1/2,1/4,1/4)$.

Suppose the observed response is category 1. Then $n_i=1$, $y_{i1}=1$, and $y_{i2}=0$. The first score block is $U_1=1(1-1/2)=1/2$. The second is $U_2=1(0-1/4)=-1/4$. A positive first score means the log-likelihood increases locally if $\beta_1$ increases; the negative second score means it decreases locally if $\beta_2$ increases, with the other coefficient fixed.

If instead the observed response were the baseline category 3, both nonbaseline indicators would be zero. The score blocks would then be $U_1=-1/2$ and $U_2=-1/4$. The baseline has no separate coefficient block in this parameterization.

Worked Example: Two Grouped Observations

Consider three categories with category 3 as baseline and a common design vector $x_i=(1,z_i)^T$ for categories 1 and 2. Let group 1 have $z_1=0$, coefficient predictors $(\eta_{11},\eta_{12})=(\log 3,\log 2)$, and counts $(y_{11},y_{12},y_{13})=(2,1,3)$. These counts sum to $n_1=6$. Its denominator is $3+2+1=6$, giving probabilities $(1/2,1/3,1/6)$.

Let group 2 have $z_2=1$, predictors $(\eta_{21},\eta_{22})=(\log 2,\log 3)$, and counts $(4,5,1)$. These counts sum to $n_2=10$, not 6. Its denominator is $2+3+1=6$, giving probabilities $(1/3,1/2,1/6)$. The predictor values for both groups can arise from linear models: for category 1 take $\beta_1=(\log 3,\log(2/3))^T$, and for category 2 take $\beta_2=(\log 2,\log(3/2))^T$.

For category 1, group 1 contributes $x_1(2-6/2)=(1,0)^T(-1)=(-1,0)^T$. Group 2 contributes $x_2(4-10/3)=(1,1)^T(2/3)=(2/3,2/3)^T$. Therefore $U_1=(-1/3,2/3)^T$.

For category 2, group 1 contributes $x_1(1-6/3)=(1,0)^T(-1)=(-1,0)^T$. Group 2 contributes $x_2(5-10/2)=(1,1)^T(0)=(0,0)^T$. Therefore $U_2=(-1,0)^T$. In particular, the second group contributes $2/3$ to the intercept coordinate of $U_1$, and zero to the intercept coordinate of $U_2$; its count total is ten, as required by its count vector.

Worked Example: Category-Specific Design Vectors

For a single observation with covariate value $t=2$, suppose category 1 uses design vector $x_{i1}=(1,t)^T=(1,2)^T$, while category 2 uses $x_{i2}=(1,t^2)^T=(1,4)^T$. Let $\beta_1=(\log 2,0)^T$ and $\beta_2=(0,0)^T$. Then $\eta_{i1}=\log 2$, $\eta_{i2}=0$, and the fitted probabilities are again $(1/2,1/4,1/4)$.

Suppose the observed response is category 2. The category 1 observed count is zero, so $U_1=(1,2)^T(0-1/2)=(-1/2,-1)^T$. The category 2 observed count is one, so $U_2=(1,4)^T(1-1/4)=(3/4,3)^T$. The two score blocks have different dimensions’ design weighting in general; here both have two entries, but their distinct design vectors produce distinct coordinate contributions.

This calculation illustrates why the formula must retain the category subscript on $x_{ij}$. Replacing both design vectors by one common vector would not describe the model specified here.

Reading and Using the Score

Each term $y_{ij}-n_i\pi_{ij}$ is a response residual on the count scale. It is positive when the observed count in category $j$ exceeds its fitted count, negative when it falls short, and zero when observed and fitted counts agree. Multiplication by $x_{ij}$ translates that discrepancy into contributions to the coefficient coordinates. The full score adds those contributions across observations and groups.

At an interior maximum of a differentiable log-likelihood, the score is zero. Thus the likelihood equations require weighted observed-minus-fitted discrepancies to sum to zero for every coefficient block. This does not mean every group’s observed count must equal its fitted count: positive and negative contributions can cancel, and the design vectors affect how they combine.

Common Mistake: Treating a Group Like One Individual

A frequent error is to use $y_{ij}-\pi_{ij}$ for a grouped observation, as though the group contained one response. A group with $n_i$ trials has expected category count $n_i\pi_{ij}$, so its score contribution is $x_{ij}(y_{ij}-n_i\pi_{ij})$. For example, if $n_i=10$, $y_{ij}=4$, and $\pi_{ij}=1/3$, the count residual is $4-10/3=2/3$, not $4-1/3$. The latter compares a count with a probability and therefore uses incompatible scales.

A related bookkeeping error is to state a group size that does not equal the sum of its category counts. Always check $n_i=\sum_j y_{ij}$ before computing fitted counts or score contributions. Finally, the baseline category is not assigned an additional free coefficient block: its logit is fixed at zero to define the parameterization.

Key takeaway: For each nonbaseline category, differentiate its contribution through the linear predictor and then apply the chain rule. The resulting block is the sum of category-specific design vectors times observed counts minus fitted counts; for individual data, the counts reduce to response indicators.

Check Your Understanding

Use the score formula to answer the following questions.

For an individual observation in category $j$, what are $n_i$ and $y_{ij}$, and what is its contribution to $U_j$?
A grouped observation has $n_i=12$, $y_{ij}=5$, and $\pi_{ij}=1/3$. What is its count residual before multiplication by $x_{ij}$?
Why does the score for $\beta_j$ use $x_{ij}$, rather than the design vector for another category?
In the grouped example above, verify that each category count vector sums to its stated group total.
Does a zero total score require observed and fitted counts to agree separately in every group? Explain what the score equation requires instead.