The Multinomial Logit Information Matrix

From the Score to the Information Matrix

The multinomial logit score describes how the likelihood changes when a category-specific coefficient changes. The information matrix describes the curvature of that likelihood and the local dependence among those coefficients. In a baseline-category model, this matrix has off-diagonal blocks: changing one nonbaseline category’s linear predictor changes its probability and the probabilities of the other categories as well.

Use \(n\) independent groups, indexed by \(i\), and \(J\) response categories, with category \(J\) as the baseline. Group \(i\) has total \(m_i\) and observed counts \(Y_{i1},\ldots,Y_{iJ}\). For nonbaseline category \(j\), let \(x_{ij}\in\mathbb{R}^{q_j}\) be its fixed design vector and \(\beta_j\in\mathbb{R}^{q_j}\) its coefficient vector. The baseline linear predictor is zero.

Assume the group responses are independent, \(Y_i=(Y_{i1},\ldots,Y_{iJ})\sim\operatorname{Multinomial}(m_i,p_i)\), and the covariates and totals are fixed. Each coefficient vector is unrestricted. The formulas below apply at finite coefficient values, where every softmax probability is strictly positive. A group with \(m_i=0\) contributes no response information.

Ignoring terms in the log-likelihood that do not involve \(\beta\), the grouped log-likelihood is

Differentiating with respect to \(\beta_j\) gives the score block \(U_j=\sum_i x_{ij}(Y_{ij}-m_i p_{ij})\). The information follows by differentiating this score once more. The probability derivative that drives every block is

Here \(\mathbf{1}\{j=k\}\) is one when the indices agree and zero otherwise. The derivative is a column vector; multiplying it by \(x_{ij}^{T}\) differentiates the score block.

Observed and Expected Information Blocks

Let \(\mathcal{I}_{jk}\) denote the block in row \(j\), column \(k\) of the information matrix. Differentiating the score gives the following result.

Derivation. In the score \(U_j=\sum_i x_{ij}(Y_{ij}-m_i p_{ij})\), the observed counts do not depend on the coefficients. Therefore

This expression contains no observed counts. Its expectation under the model is consequently the same expression, proving the observed-equals-expected claim. For \(j=k\), the factor is \(p_{ij}(1-p_{ij})\); for \(j\ne k\), it is \(-p_{ij}p_{ik}\). This gives the displayed diagonal and off-diagonal blocks. The \((k,j)\) block is the transpose of the \((j,k)\) block, so the full information matrix is symmetric. \(\square\)

The same block structure has a direct probability interpretation. For one response in group \(i\), let \(Z_{ij}\) be the indicator that the response is in category \(j\), for a nonbaseline category \(j\). Then \(E[Z_{ij}]=p_{ij}\), and

Indeed, \(Z_{ij}^2=Z_{ij}\) when \(j=k\); for distinct categories, \(Z_{ij}Z_{ik}=0\). In a group of \(m_i\) independent trials, the covariance of the category counts is \(m_i\) times this one-trial covariance. The information block is this covariance multiplied on the left and right by the relevant design vectors. In particular, two distinct category indicators have covariance \(-p_{ij}p_{ik}\): a trial assigned to one category cannot simultaneously be assigned to the other.

Positive Definiteness and the Role of Positive Totals

The information matrix is always positive semidefinite under the stated model, but it need not be positive definite. A coefficient direction has zero information if its perturbation leaves every positive-total group’s category probabilities unchanged to first order. The exact condition can be stated using the design rows for groups that actually contribute observations.

Proof. Let \(b_j\in\mathbb{R}^{q_j}\) be an arbitrary perturbation of \(\beta_j\), and put \(a_{ij}=x_{ij}^{T}b_j\) for \(j<J\), with \(a_{iJ}=0\) for the baseline. The quadratic form of the information in the combined direction \(b=(b_1,\ldots,b_{J-1})\) is

For each group, the expression in braces is the variance of a random variable that takes value \(a_{ij}\) with probability \(p_{ij}\), including value zero with baseline probability \(p_{iJ}\). Variance is nonnegative. Since every category probability is strictly positive, this variance is zero exactly when all these values agree; because the baseline value is zero, that happens exactly when \(a_{ij}=0\) for every nonbaseline \(j\). Thus the full quadratic form is zero exactly when \(x_{ij}^{T}b_j=0\) for every \(j\) and every group with \(m_i>0\). Equivalently, \(X_j^+b_j=0\) for every \(j\). The quadratic form is strictly positive for every nonzero combined direction if and only if each such null space contains only zero, which is precisely full column rank of every \(X_j^+\). \(\square\)

Why the Blocks Matter

The information matrix records the local curvature in all category-specific coefficient directions jointly. Its diagonal blocks describe curvature within a category’s coefficient vector; its off-diagonal blocks describe how those directions are coupled through the shared softmax probabilities. When the matrix is positive definite, its inverse exists and is the usual local covariance approximation used in likelihood-based estimation. A singular matrix instead signals a lack of information in at least one coefficient direction under the stated design.

A common mistake is to compute only the separate diagonal blocks and treat them as the whole information matrix. That discards the terms \(-m_i p_{ij}p_{ik}x_{ij}x_{ik}^{T}\) for \(j\ne k\), even though a change in one nonbaseline predictor changes the normalization shared by all categories. The correction is to build the full block matrix before any inversion or uncertainty calculation. Another related mistake is to assess rank using every design row without checking group totals; only positive-total rows contribute.