Deriving Category Probabilities With The Softmax Map

From Baseline Logits to Category Probabilities

A baseline-category logit model describes a response that can fall into one of \(K\) categories. For a fixed covariate profile, the model uses a predictor for each nonbaseline category relative to a designated baseline. Those predictors are log ratios, not probabilities: they can be negative, positive, or any finite real number. The softmax map converts them into a full set of category probabilities.

The conversion has to satisfy two requirements. Each probability must be positive, and the probabilities across all categories must add to one. The baseline category is included in the normalization even though it has no separately estimated predictor: its relative weight is one. This tutorial derives that normalization, checks its properties, and works through calculations in both directions.

In a regression model, each nonbaseline predictor may depend on covariates, for example through a linear expression \(\eta_j(x)\). The probability calculation below is carried out at one fixed profile \(x\), so we write \(\eta_j\) rather than repeatedly including \(x\). The baseline convention is equivalent to setting \(\eta_K=0\); this is a convenient way to include the baseline in the same normalization as the other categories.

Deriving the Softmax Formula

Start from the defining log-ratio equations for the nonbaseline categories. Exponentiating each equation gives \(p_j/p_K=\exp(\eta_j)\), or \(p_j=p_K\exp(\eta_j)\). Thus every nonbaseline probability is its baseline probability multiplied by a relative weight. The requirement that all category probabilities sum to one determines the common scale \(p_K\).

Equivalently, define \(\eta_K=0\) and include all \(K\) categories in the denominator:

The two forms are identical because \(\exp(\eta_K)=\exp(0)=1\). The baseline has a numerator too; it is simply one. The denominator is the sum of all category weights, not just the weights for categories with estimated predictors.

Why the Formula Gives Valid Probabilities

The softmax formula is defined for finite real predictors. Its denominator contains positive terms, including the baseline term \(1\), so it cannot be zero. The next result checks both the range of each probability and the sum constraint. The condition \(K\ge2\) matters: with one category there are no nonbaseline predictors, and its sole probability is one.

Proof. Set \(D=1+\sum_{r=1}^{K-1}\exp(\eta_r)\). Each exponential is finite and strictly positive, and \(D\) includes the positive term \(1\), so \(0<D<\infty\). For \(j<K\), \(p_j=\exp(\eta_j)/D>0\); also \(p_K=1/D>0\). For a nonbaseline category, \(D\) contains its numerator \(\exp(\eta_j)\) and the additional positive baseline term \(1\), so \(D>\exp(\eta_j)\) and \(p_j<1\). For the baseline, \(K\ge2\) ensures that the sum defining \(D\) includes at least one positive exponential term, so \(D>1\) and \(p_K<1\). Finally,

Thus the softmax outputs are strictly positive probabilities summing to one. \(\square\)

A useful way to interpret the calculation is that the unnormalized weights are \(\exp(\eta_1),\ldots,\exp(\eta_{K-1}),1\). Dividing every weight by their total produces probabilities without changing the relative weight of any pair of categories. In particular, the baseline ratio remains \(p_j/p_K=\exp(\eta_j)\).

The Inverse Map: Recovering Predictors

The relationship also works in reverse, provided every category has positive probability. This positivity condition is necessary for the logarithms of the probability ratios to be finite. The result says that the softmax map gives a one-to-one correspondence between finite baseline predictors and probability vectors in the interior of the \(K\)-category probability simplex.

Proof. The forward direction follows from the preceding theorem. For the converse, define \(\eta_j=\log(p_j/p_K)\). These are finite real numbers because \(p_j>0\) and \(p_K>0\). Exponentiating gives \(\exp(\eta_j)=p_j/p_K\), so the softmax denominator is

The softmax probability for category \(j<K\) is therefore \((p_j/p_K)/(1/p_K)=p_j\), and the baseline probability is \(1/(1/p_K)=p_K\). This proves existence. For uniqueness, if any finite predictors produce the given probabilities, their baseline ratios must satisfy \(\exp(\eta_j)=p_j/p_K\). Taking the logarithm forces \(\eta_j=\log(p_j/p_K)\) for every \(j<K\). Hence no other finite predictor vector produces the same probabilities. \(\square\)

Reading a Multicategory Calculation Carefully

For more than three categories, the same steps apply without alteration: exponentiate each nonbaseline predictor, append a baseline weight of one, sum all the weights, and divide each weight by that sum. Keeping the baseline weight visible is a practical safeguard against omitting a category from the denominator.

Common Mistakes and Why the Map Matters

• Omitting the baseline weight. A tempting but incorrect denominator is \(\sum_{r=1}^{K-1}\exp(\eta_r)\). That expression leaves out the baseline category and generally makes the probabilities sum to more than one when the baseline probability is added. Include its weight \(1=\exp(0)\).
• Treating predictors as probabilities. A predictor can be negative or greater than one, while a probability must lie between zero and one. Exponentiating and normalizing—not interpreting \(\eta_j\) directly—produces the probability.
• Claiming finite predictors can produce a zero probability. For every finite real \(\eta_j\), \(\exp(\eta_j)>0\), so the softmax probability is strictly positive. A probability vector containing a zero cannot be represented by finite baseline predictors through the inverse logarithm formula.
• Forgetting the number of categories. The strict upper bound \(p_j<1\) uses \(K\ge2\). With just one category, the only probability is \(1\), and there are no nonbaseline predictors.

The softmax map matters because it turns unconstrained finite real predictors into a valid probability vector at each covariate profile. It also makes clear what information the predictors encode: each one specifies a log ratio to the baseline, while the normalization determines all category probabilities together.