Maximum Likelihood For Multinomial Logit

From Choice Probabilities to a Likelihood

Multinomial logit turns a model for the probability of each available choice into a method for estimating unknown coefficients from observed choices. The central construction is simple: each individual contributes the probability assigned to the alternative they actually chose. Multiplying those contributions gives the likelihood; taking logarithms turns the product into a sum that can be optimized.

Suppose there are \(J\) alternatives, labelled \(1,\ldots,J\), and \(n\) independent individuals. Each individual chooses exactly one alternative from the choice set under study. Let \(y_{ij}=1\) if individual \(i\) chooses alternative \(j\), and \(y_{ij}=0\) otherwise. Thus, for each individual, \(\sum_{j=1}^J y_{ij}=1\). Use alternative \(J\) as the reference, with its coefficients normalized to zero. For a basic specification, let \(x_i\) be a vector of individual characteristics and let alternative \(j\)'s systematic utility be \(x_i^\top\beta_j\), where \(\beta_J=0\).

Under the multinomial-logit probability model, the probability of choosing \(j\), conditional on \(x_i\), is \(p_{ij}=\exp(x_i^\top\beta_j)/\sum_{r=1}^J\exp(x_i^\top\beta_r)\). The reference alternative's numerator is \(\exp(0)=1\). These probabilities are positive and sum to one. The likelihood below also relies on the sampling assumption that individuals' choices are conditionally independent given the included covariates and model parameters.

Maximum likelihood estimation chooses coefficient values that maximize this log likelihood. It does not require that the model assign probability one to any observed choice. Rather, it finds the parameter values under which the observed choices, taken together, receive the greatest likelihood among the values allowed by the model.

Deriving the Score Contributions

The score is the vector of first derivatives of the log likelihood with respect to the parameters. Its individual contribution shows how an observation pushes the estimated coefficients when the model-implied probability differs from the observed choice indicator. For each non-reference alternative \(k\), the parameter block is \(\beta_k\).

To differentiate with respect to \(\beta_k\), the first term contributes \(y_{ik}x_i\). For the second term, differentiating the log of the sum of exponentials gives \(p_{ik}x_i\). Subtracting the second contribution yields the individual score. Summing over individuals gives the sample score used in estimation.

The score is a vector: each component corresponds to one element of \(x_i\). If \(y_{ik}>p_{ik}\), the observation contributes a positive multiple of \(x_i\) to the score for alternative \(k\); if \(y_{ik}<p_{ik}\), that contribution points in the opposite direction. At an interior maximum, the sample score is zero for every estimated coefficient. This is a balance across observations, not a requirement that each individual score be zero.

How Maximum Likelihood Estimation Works

In general, the score equations are nonlinear because each \(p_{ij}\) depends on all the coefficients. There is usually no closed-form solution, so software maximizes \(\ell(\beta)\) using a numerical optimization method. At a candidate estimate, the optimizer evaluates the probabilities, log likelihood, and score, then updates the coefficients to improve the objective until its convergence criteria are met. A reported solution should be checked for convergence and for a score that is small relative to the scale of the problem.

The curvature of the log likelihood explains why these calculations are tractable in the basic model. Differentiating the score for alternative \(k\) with respect to \(\beta_l\) uses \(\partial p_{ik}/\partial\beta_l=p_{ik}(\mathbf{1}[k=l]-p_{il})x_i\), where \(\mathbf{1}[k=l]\) is one when \(k=l\) and zero otherwise. Therefore the block of the individual Hessian for \(\beta_k,\beta_l\) is:

For any set of coefficient directions, the quadratic form of the full Hessian is nonpositive: it is the negative of a sum of probability-weighted variances of the alternative-specific linear changes. Thus the log likelihood is concave in the coefficients in this specification. Concavity means a local maximum is also a global maximum, although it does not guarantee a finite, unique estimate: insufficient variation or complete prediction of choices can still prevent a finite solution or identification of all coefficients.

Interpreting Coefficients Carefully

A coefficient is attached to a particular alternative relative to the normalized reference. In the basic specification, for non-reference alternative \(j\), the model implies \(\log(p_{ij}/p_{iJ})=x_i^\top\beta_j\). Therefore a coefficient describes a change in the log odds of choosing \(j\) rather than the reference, associated with a one-unit change in the corresponding covariate while other included covariates are held fixed. This is an interpretation of the model’s relative log odds, not a direct change in the probability of choosing \(j\).

Assumptions, Computation, and a Common Pitfall

Maximum likelihood gives a meaningful fit only when the probability model and the sampling structure are appropriate. At minimum, the analysis specifies the alternatives available to each included individual, records exactly one chosen alternative, uses the stated multinomial-logit probability formula, and treats individuals' choices as conditionally independent for the likelihood factorization. The covariates and reference normalization must also be defined consistently. The model's implications should be considered in light of the application; a numerical optimizer cannot make an unsuitable probability model suitable.

A common mistake is to maximize the sum of the probabilities assigned to the observed alternatives, \(\sum_i p_{i,y_i}\), instead of the log likelihood \(\sum_i\log p_{i,y_i}\). These are different objectives and generally yield different estimates. The likelihood follows from the probability of the observed sample under conditional independence: it is the product of the observed-choice probabilities. Taking its logarithm gives the sum of their logarithms, not the sum of the probabilities themselves.

Another practical issue is separation or weak identification. If a covariate pattern perfectly predicts choices, the likelihood may keep increasing as some coefficients grow in magnitude, so no finite maximum exists. A successful software termination message alone is not proof that every coefficient is well determined. Inspect convergence information, the fitted probabilities, and whether the design contains enough variation to estimate the requested parameters.