Softmax Cross-Entropy and Multiclass Probabilities

From Class Scores to Probabilities

For a classification problem with \(K\) possible classes, a model may first produce a real-valued score for each class. These scores are not probabilities: they may be negative, need not sum to one, and can be shifted by the same constant without changing their relative ordering. The softmax transformation turns them into a probability vector. Cross-entropy then measures how well that vector assigns probability to the observed class.

The central statistical property is about conditional risk. At a fixed feature value \(x\), the true conditional distribution of the label is a probability vector. We will show that among all probability vectors, the expected cross-entropy is minimized exactly by that true vector. This identifies what probability estimation is targeting, independently of how a particular model is fitted.

Every finite score vector gives strictly positive probabilities, \(q_k(s)>0\), and their sum is one. Adding a constant \(c\) to every score leaves the probabilities unchanged, because the factor \(\exp(c)\) cancels between numerator and denominator. Thus scores have a location non-identifiability that does not affect the predicted probabilities.

Cross-Entropy as Categorical Negative Log-Likelihood

For an observed label \(y\), represent the outcome by a one-hot vector \(e_y\): its \(y\)th component is one and every other component is zero. The cross-entropy between \(e_y\) and a probability vector \(q\) is \(-\sum_{k=1}^K (e_y)_k\log q_k\), which equals \(-\log q_y\). It penalizes assigning a small probability to the class that actually occurred.

This loss is also the negative log-likelihood for a categorical observation. If the model assigns probabilities \(q_1,\ldots,q_K\), the likelihood of observing \(Y=y\) is \(q_y\); taking the negative logarithm gives \(-\log q_y\). For scores, substituting the softmax formula gives a convenient expression:

The term \(\log(\sum_j\exp(s_j))\) is the log-sum-exp of the scores. This form is useful both for derivations and computation. In numerical implementations, it is common to subtract the largest score \(m=\max_j s_j\) inside the exponential sum, using \(\log(\sum_j\exp(s_j))=m+\log(\sum_j\exp(s_j-m))\). The identity follows by factoring \(\exp(m)\) out of the sum and helps avoid overflow.

Conditional Risk and Its Minimizer

At a fixed \(x\), suppose the model reports a probability vector \(q\in\Delta_K\). The label is random with conditional probabilities \(\eta(x)\). The conditional cross-entropy risk is the expected loss over that label distribution:

We use the convention \(0\log 0=0\) in entropy expressions. If \(q_k=0\) for any class with \(\eta_k(x)>0\), the conditional risk is \(+\infty\), since that class can occur but is assigned zero probability. Define the entropy \(H(\eta(x))=-\sum_k\eta_k(x)\log\eta_k(x)\). When \(q_k>0\) wherever \(\eta_k(x)>0\), the risk decomposes into entropy plus Kullback–Leibler divergence:

Proof. If \(q_k=0\) for some \(k\) with \(\eta_k>0\), then \(R(q)=+\infty\), so it cannot improve on \(q=\eta\), whose risk is the finite entropy \(H(\eta)\). Otherwise, the risk decomposition applies. It remains to establish that \(D_{\mathrm{KL}}(\eta\Vert q)\geq0\), with equality only if \(q=\eta\).

For each \(k\) with \(\eta_k>0\), the elementary inequality \(\log t\leq t-1\), valid for \(t>0\) with equality only at \(t=1\), gives \(\eta_k\log(q_k/\eta_k)\leq q_k-\eta_k\). Summing over the support \(S=\{k:\eta_k>0\}\) yields \[ -D_{\mathrm{KL}}(\eta\Vert q) \leq \sum_{k\in S}q_k-\sum_{k\in S}\eta_k =\sum_{k\in S}q_k-1 \leq 0. \] The final inequality holds because \(q\) is a probability vector. Hence the divergence is nonnegative. For equality, both inequalities must be equalities: the logarithm inequality requires \(q_k/\eta_k=1\) for every \(k\in S\), and \(\sum_{k\in S}q_k=1\) requires zero probability outside \(S\). These conditions together are exactly \(q=\eta\). The risk decomposition now proves the result. \(\square\)

What the Minimizer Means for Softmax Scores

The theorem optimizes over the full simplex, including its boundary. A finite softmax score vector, however, can only produce probabilities strictly greater than zero. If every component of \(\eta(x)\) is positive, the true probability vector is attainable: choose \(s_k=\log\eta_k(x)\), for which \(\sum_j\exp(s_j)=\sum_j\eta_j(x)=1\), and softmax returns \(q=\eta(x)\). More generally, all score vectors \(s_k=\log\eta_k(x)+c\) give the same probabilities.

If some true class probabilities are zero, no finite score vector can represent \(\eta(x)\) exactly. The minimum over the closed simplex is still \(H(\eta(x))\), but finite softmax scores only approach it. For example, \(q^{(\epsilon)}=(1-\epsilon)\eta+\epsilon(1/K,\ldots,1/K)\), for \(0<\epsilon<1\), has positive components and converges to \(\eta\) as \(\epsilon\) decreases to zero. For each \(k\) with \(\eta_k>0\), \(q_k^{(\epsilon)}\) converges to \(\eta_k\), so the finite sum defining \(R(q^{(\epsilon)})\) converges to \(H(\eta)\). Thus the infimum over finite scores equals the simplex minimum, even though it need not be attained by finite scores.

Score Gradients and a Common Pitfall

The score form of the loss gives a simple derivative that is central to fitting softmax models. For a fixed observed class \(y\), differentiate the log-sum-exp term with respect to score \(s_j\). The chain rule gives \[ \frac{\partial}{\partial s_j}\log\left(\sum_{k=1}^K\exp(s_k)\right) =\frac{\exp(s_j)}{\sum_{k=1}^K\exp(s_k)} =q_j(s). \] The derivative of \(-s_y\) with respect to \(s_j\) is \(-1\) when \(j=y\) and zero otherwise. Therefore,

This derivative is the predicted probability minus the observed one-hot indicator. Its components sum to zero, since \(\sum_j q_j(s)=1\) and \(\sum_j\mathbf{1}\{y=j\}=1\). That agrees with the fact that adding the same constant to every score leaves the loss unchanged.

A common mistake is to say that softmax cross-entropy always has a finite-score minimizer at the true conditional probabilities. The statement confuses two optimization domains. Over all probability vectors in the closed simplex, the unique minimizer is always \(q=\eta\). Over finite score vectors, softmax probabilities are strictly positive, so exact attainment requires every \(\eta_k\) to be positive. When some true probabilities vanish, finite scores can approach the optimum but cannot attain it. This distinction matters when interpreting a model's fitted scores, especially in settings where certain classes are impossible at a given feature value.

Cross-entropy is therefore more than a convenient penalty for incorrect labels: its conditional target is the full class-probability vector. This is distinct from choosing only a class decision. Probability estimates retain information about uncertainty, while a final class prediction may reduce the vector to a single label. The conditional minimization result explains precisely why cross-entropy is used when the modeling goal is to estimate multiclass probabilities.