Posterior Probabilities From Multiclass Discriminant Scores

From Discriminant Scores to Probabilities

A discriminant score ranks classes at a particular observation, but a ranking does not tell us how uncertain the classification is. If one class has a score just above the others, the evidence may be fairly ambiguous; if its score is far larger, the evidence may be concentrated on that class. Converting scores into posterior probabilities gives a normalized measure of this uncertainty.

Suppose $G$ is the class label, with possible values $1,\ldots,K$, and $x$ is an observed feature vector. Let $\pi_k=P(G=k)$ be the prior probability of class $k$, and let $f_k(x)$ be the class-conditional density at $x$, with all densities taken relative to the same dominating measure. In Gaussian discrimination, the $f_k$ are Gaussian densities and the discriminant scores are commonly formed from their log densities and the class priors.

Definition: For an observation $x$ with positive marginal density, the posterior probability of class $k$ is $P(G=k\mid X=x)$. If $\pi_k>0$ and $f_k(x)>0$, its log discriminant score is $g_k(x)=\log \pi_k+\log f_k(x)$. The softmax normalization of scores $g_1(x),\ldots,g_K(x)$ is $$ P(G=k\mid X=x)=\frac{\exp(g_k(x))}{\sum_{j=1}^{K}\exp(g_j(x))}. $$

Because $\exp(g_k(x))=\pi_k f_k(x)$, this formula is Bayes' rule written in terms of log scores. The denominator makes the class probabilities add to one. Any common additive term in all the scores cancels, so the scores need not be absolute log densities: score differences contain all the information needed for normalization.

Deriving and Interpreting the Softmax Formula

The posterior probabilities follow directly from the class-label model. The joint density of $X$ and the event $G=k$ is $\pi_k f_k(x)$. Summing over the mutually exclusive classes gives the marginal density of $X$, namely $\sum_{j=1}^K \pi_j f_j(x)$. Where this marginal density is positive, division gives the posterior.

Result: If $G$ takes values $1,\ldots,K$, the prior probabilities sum to one, and the class-conditional distributions have densities $f_k$ with respect to a common measure, then at every $x$ where $\sum_j \pi_j f_j(x)>0$, $$ P(G=k\mid X=x)=\frac{\pi_k f_k(x)}{\sum_{j=1}^K \pi_j f_j(x)}. $$ If the scores are $g_k(x)=\log\{\pi_k f_k(x)\}$ for positive terms, this is exactly the softmax formula.

To verify the result, apply the definition of conditional probability through the joint and marginal densities:

P(G=k\mid X=x) =\frac{\pi_k f_k(x)}{f_X(x)} =\frac{\pi_k f_k(x)}{\sum_{j=1}^{K}\pi_j f_j(x)}.

For positive $\pi_k f_k(x)$, replacing each term by $\exp(g_k(x))$ yields the stated softmax. The probabilities are nonnegative, and their sum is $\sum_k \pi_k f_k(x)/\sum_j \pi_j f_j(x)=1$. Thus the normalization is not an extra modeling assumption; it is the conditional probability implied by the class model. $\square$

A useful consequence is that posterior odds between any two classes depend only on the difference of their scores:

\frac{P(G=k\mid X=x)}{P(G=\ell\mid X=x)} =\exp\{g_k(x)-g_\ell(x)\}.

This follows by dividing their two softmax expressions: their common denominator cancels, leaving $\exp(g_k)/\exp(g_\ell)$. In particular, the class with the largest score has the largest posterior probability. The scores must nevertheless be normalized to obtain the probabilities themselves.

Worked Example: Normalizing Three Scores

At one observation, suppose the three Gaussian discriminant scores are $2.4$, $1.1$, and $-0.7$. Calculate the posterior probabilities.

The largest score is $2.4$. Subtract it from every score, giving the equivalent shifted scores $0$, $-1.3$, and $-3.1$. Their exponentials are approximately $1$, $0.272532$, and $0.045049$. The sum is $1.317581$.

The normalized probabilities are therefore $1/1.317581\approx0.75897$, $0.272532/1.317581\approx0.20684$, and $0.045049/1.317581\approx0.03419$. They sum to $1.00000$, up to the displayed rounding. Class 1 is most probable, but the probability assigned to classes 2 and 3 together is about $0.24103$, so the probability vector conveys more than the winning label.

Stable Computation by Subtracting the Largest Score

Directly exponentiating scores can cause numerical overflow when scores are large, or underflow when they are very negative. A simple transformation avoids overflow: subtract the maximum score before exponentiating. The probabilities do not change under this transformation.

Result: For finite scores $g_1,\ldots,g_K$, let $m=\max_j g_j$. Then $$ \frac{\exp(g_k)}{\sum_{j=1}^{K}\exp(g_j)} = \frac{\exp(g_k-m)}{\sum_{j=1}^{K}\exp(g_j-m)}. $$ Every shifted score is at most zero, and at least one is zero, so the largest exponential is exactly one.

To prove the identity, factor the positive quantity $\exp(m)$ out of the denominator and numerator:

\frac{\exp(g_k)}{\sum_j\exp(g_j)} =\frac{\exp(m)\exp(g_k-m)} {\exp(m)\sum_j\exp(g_j-m)} =\frac{\exp(g_k-m)}{\sum_j\exp(g_j-m)}.

Since $g_j-m\leq0$ for every $j$, all shifted exponentials lie in $(0,1]$, and one equals $1$ for a maximizing class. This prevents overflow from exponentiating a large positive score. Very small terms can still underflow in finite-precision arithmetic, but their contribution is then negligible at that precision. $\square$

Worked Example: Gaussian Scores with a Large Negative Offset

A Gaussian model produces scores $g_1=-1.1931$, $g_2=-1.8971$, and $g_3=-3.6094$. The score differences are more informative for computation than the shared negative offset. Subtract the largest score, $-1.1931$, to obtain approximately $0$, $-0.7040$, and $-2.4163$.

In this example the corresponding shifted weights are $\exp(0)=1$, $\exp\{\log(0.3)+0.5\}=0.494616$, and $\exp\{\log(0.4)-1.5\}=0.089252$. Their sum is $1.583868$. Dividing each weight by this sum gives approximately $0.631366$, $0.312284$, and $0.056351$, respectively. To four decimal places these are $0.6314$, $0.3123$, and $0.0564$; the unrounded values sum to one.

The largest score gives class 1 the largest posterior probability, but its probability is about $0.6314$, not one. The remaining posterior mass is distributed across the other two classes. This calculation also illustrates why rounding should be delayed until after normalization.

Probabilities and Classification Decisions

Posterior probabilities and class assignments answer different questions. The posterior vector describes uncertainty about the class conditional on the observed features and the assumed model. A classification rule chooses an action. Under zero-one loss, in which every incorrect assignment has the same cost, choosing the largest posterior probability minimizes conditional expected loss. With unequal costs, the action minimizing expected loss need not be the most probable class.

Worked Example: A Cost-Sensitive Decision from Posterior Probabilities

Suppose a three-class posterior vector is $(0.50,0.30,0.20)$. Consider the following loss table, where rows are the chosen actions and columns are the true classes.

Action	True class 1	True class 2	True class 3
Choose class 1	0	4	4
Choose class 2	1	0	1
Choose class 3	1	1	0

The conditional expected loss of choosing class 1 is $0(0.50)+4(0.30)+4(0.20)=2.00$. For choosing class 2 it is $1(0.50)+0(0.30)+1(0.20)=0.70$. For choosing class 3 it is $1(0.50)+1(0.30)+0(0.20)=0.80$. The minimum is $0.70$, so the cost-sensitive rule chooses class 2 even though class 1 has the greatest posterior probability. The probabilities have not changed; only the decision criterion has.

In Gaussian discriminant analysis, the score used for normalization must include the terms that define the intended posterior, including the class prior. If scores omit priors, their softmax instead corresponds to a model that assigns equal prior weight to classes, unless the omitted prior terms are restored. Likewise, a score vector from estimated model parameters gives plug-in posterior probabilities: these are conditional probabilities under the fitted model, not a guarantee that the model is correctly specified or that empirical calibration is perfect.

Common Mistakes

Normalizing the scores themselves. Dividing $g_k$ by $\sum_j g_j$ is not softmax normalization; scores can be negative and do not represent probability weights. Exponentiate first, then divide by the sum of exponentials.
Treating the largest score as a probability of one. The largest score determines the most probable class, but the corresponding posterior is generally less than one. All class weights enter the denominator.
Exponentiating without stabilizing. Adding or subtracting a common constant from every score leaves softmax probabilities unchanged. Subtracting the maximum is often safer numerically.
Dropping the prior unintentionally. If the target is the posterior under specified class priors, the prior contribution must be included in the scores, or explicitly accounted for during normalization.
Equating the most probable class with every optimal decision. That equivalence holds under equal zero-one misclassification loss. Under unequal losses, compare expected losses using the posterior vector.

Key takeaway: A multiclass discriminant score becomes a posterior probability by exponentiating and dividing by the sum of all class weights. Subtracting the largest score first preserves the result while improving numerical stability. The resulting probabilities quantify uncertainty; a classification action may additionally depend on the loss assigned to each error.

Check Your Understanding

Use the definitions and calculations above to answer the following questions.

For scores $1.2$, $1.2$, and $-0.8$, what can be said about the first two posterior probabilities before computing the denominator?
Why does subtracting the largest score leave every softmax probability unchanged?
If all three scores are shifted upward by the same constant, what happens to the normalized posterior probabilities?
What modeling information may be missing if Gaussian log-density scores are normalized but class priors are not included?
Under what loss function does assigning the class with the largest posterior probability minimize conditional expected loss?