Identifiability Of Multinomial Logit Parameters

Why Identifiability Matters

A multinomial logit model describes category probabilities through a set of linear predictors. If several different parameter values produce exactly the same probabilities, the data cannot distinguish those parameter values. This is a problem of identifiability: before interpreting or estimating the parameters, we need to know whether the model assigns a unique parameter value to each probability distribution.

The ambiguity in an all-category logit model has a simple source. The softmax map uses relative, not absolute, values of the category logits. Adding the same quantity to every logit changes none of the probabilities. A baseline category removes this particular redundancy, though it does not correct other sources of non-identifiability, such as a rank-deficient predictor matrix.

All-Category Logits and Identifiability

Suppose a response has \(K\) categories, indexed by \(j=1,\ldots,K\). Let \(x\) be a \(q\)-dimensional predictor vector, with an intercept included as one of its components when the model has category-specific intercepts. Write the linear predictor for category \(j\) as \(\eta_j(x)=x^\mathsf{T}\beta_j\), where \(\beta_j\) is its category-specific coefficient vector. The softmax model assigns probabilities

The probabilities are positive and sum to one. The model is identifiable on a specified set of predictor values if no two distinct allowed parameter values give the same vector of category probabilities at every predictor value in that set. This definition concerns the probability model, not whether one particular sample happens to contain observations from every category.

For any real-valued function \(c(x)\), define new logits by \(\eta'_j(x)=\eta_j(x)+c(x)\) for every category. Then

Thus a shared shift does not change the probability vector. In the linear-predictor model, a common shift can itself be linear: if \(c(x)=x^\mathsf{T}\gamma\), then setting \(\beta'_j=\beta_j+\gamma\) for all \(j\) leaves all probabilities unchanged. With an intercept in \(x\), this includes adding a common constant to every category intercept and common amounts to every category’s slopes.

Exactly Which Logits Are Equivalent?

The common-shift argument gives one way to obtain equivalent logits. The next result shows that, at a fixed predictor value, it gives all such equivalences.

Proof. If \(\eta'_j=\eta_j+c\) for every \(j\), the common factor \(\exp(c)\) cancels from the softmax denominator, so \(p'_j=p_j\) for every category. Conversely, all softmax probabilities are positive, and for any category \(j\),

If \(p=p'\), the log probability ratios are equal, so \(\eta'_j-\eta'_K=\eta_j-\eta_K\) for every \(j\). Rearranging gives \(\eta'_j-\eta_j=\eta'_K-\eta_K\) for every \(j\). Taking \(c=\eta'_K-\eta_K\) proves that the difference is the same for every category. \(\square\)

This result explains the redundancy precisely: at each predictor value, only the differences among the logits are determined by the probabilities. In particular, the log odds of category \(j\) relative to category \(K\) are \(\eta_j(x)-\eta_K(x)\). The overall level shared by all logits cannot be recovered from the probability vector.

Fixing a Baseline Category

A standard solution is to select one category as the baseline and set its linear predictor to zero. Taking category \(K\) as the baseline gives

The baseline is a normalization, not a claim that category \(K\) has probability zero or has no relationship with the predictors. Its probability remains part of the model. The constraint chooses one representative from each set of logits that differ only by a shared shift.

For a linear-predictor model, the baseline constraint can be applied to coefficient vectors. Given unrestricted vectors \(\beta_1,\ldots,\beta_K\), define \(\widetilde{\beta}_j=\beta_j-\beta_K\) for \(j<K\), and \(\widetilde{\beta}_K=0\). Then \(x^\mathsf{T}\widetilde{\beta}_j=x^\mathsf{T}\beta_j-x^\mathsf{T}\beta_K\), which subtracts the baseline logit from every category. The probabilities are unchanged, and the resulting \(K-1\) coefficient vectors specify the log odds relative to the baseline.

The following result states when this normalization removes the shared-shift ambiguity from the coefficient parameters. Its rank condition matters: the model must have enough independent predictor information to distinguish the coefficients.

Proof. Suppose first that the probability vectors agree at every design row. By the preceding theorem, for every row \(i\) and category \(j\),

Stacking these equalities over rows gives \(X[(\beta'_j-\beta'_K)-(\beta_j-\beta_K)]=0\). Since \(X\) has full column rank, its null space contains only the zero vector, and hence \(\beta'_j-\beta'_K=\beta_j-\beta_K\) for each \(j\). Set \(\gamma=\beta'_K-\beta_K\). Rearranging yields \(\beta'_j=\beta_j+\gamma\) for every \(j\). Conversely, if all coefficient vectors are shifted by the same \(\gamma\), every row’s logits receive the same shift \(x_i^\mathsf{T}\gamma\); the softmax probabilities therefore remain unchanged. Finally, if both arrays satisfy \(\beta_K=\beta'_K=0\), then the shared-shift relation forces \(\gamma=0\), so the arrays are equal. \(\square\)

What the Baseline Solves—and What It Does Not

With \(K\) categories and \(q\) coefficients per category, an unrestricted all-category model lists \(Kq\) coefficients. The common-shift invariance accounts for \(q\) redundant directions: each component of the shared vector \(\gamma\) can be added to every category’s coefficient vector. A baseline parameterization has \((K-1)q\) free coefficients, matching the number of independent category contrasts.

That count assumes the predictor columns are linearly independent on the design being used to identify the coefficients. If they are not, distinct coefficient vectors can produce the same linear predictors, as in the worked example. Thus, fixing a baseline removes the category-wide shift but does not by itself guarantee identifiability under every design. It is useful to check both the category normalization and the rank of the predictor matrix.

Common Pitfall

• Wrong reasoning: “The probabilities sum to one, so setting one category’s probability or logit to zero solves the redundancy.” The probabilities already sum to one by the softmax definition, and the baseline category generally has positive probability.
• Correction: Fix the baseline linear predictor or coefficient vector to zero. This sets a reference for the logits without removing the baseline category from the response.
• Another incomplete conclusion: “Once a baseline is fixed, every coefficient must be identifiable.” This is only true when the predictor information is sufficiently rich; rank-deficient columns can still yield multiple coefficient vectors with the same predictions.