Log Loss and Kullback–Leibler Regret

Why Probability Reports Need a Loss

A classifier can report only its most likely class, or it can report a probability for every class. The latter carries more information: a report such as “class 1 has probability 0.8” distinguishes strong confidence from a near tie. To assess such reports, we need a loss that evaluates the probability assigned to the class that actually occurs. Log loss does exactly this, and its regret has a precise interpretation: it measures the Kullback–Leibler divergence between the true conditional probabilities and the reported probabilities.

Throughout, there are \(K\) possible labels, indexed by \(k=1,\ldots,K\). Logarithms are natural logarithms. We focus on probability vectors whose coordinates are all strictly positive, so every loss and divergence below is finite.

Let \(p=(p_1,\ldots,p_K)\in\Delta_K^\circ\) be the true probability vector for the label. The risk of reporting \(q\) is the expected loss when the label has distribution \(p\). The true vector \(p\) is fixed for this calculation; the randomness is in the label that is observed.

The risk is small when the report assigns high probability to labels that are likely to occur. But this intuitive description does not yet show which vector minimizes risk or how much worse a particular report is. The key is to compare the risk with the risk of reporting \(p\) itself.

Entropy and the Risk–Regret Identity

The risk of the truthful report \(q=p\) is the entropy of the true probability vector. The Kullback–Leibler divergence, in the direction needed here, compares the true probabilities in the first argument with the reported probabilities in the second.

Proof. By the definitions,

The risk at \(p\) is \(R_p(p)=-\sum_k p_k\log p_k=H(p)\). Substituting this into the identity gives \(R_p(q)-R_p(p)=D_{\mathrm{KL}}(p\|q)\), which is the claimed regret. \(\square\)

The identity reduces minimization of log-loss risk to a basic property of the divergence: it is nonnegative and vanishes only when the two probability vectors agree. Here is a proof under the stated positivity assumptions.

Proof. For every \(u>0\), \(\log u\leq u-1\), with equality only at \(u=1\). Apply this inequality to \(u=q_k/p_k\), multiply by \(p_k>0\), and change signs:

Summing over \(k\) gives

If equality holds in the sum, equality must hold in each of the \(K\) inequalities, since each difference between the left and right sides is nonnegative. Thus \(q_k/p_k=1\) for every \(k\), so \(q=p\). Conversely, if \(q=p\), every term in the divergence is zero. \(\square\)

Worked Calculations

Conditional Probabilities and Bayes Reports

In prediction, the true label distribution usually depends on observed features \(X\). For each class \(k\), let \(\eta_k(X)=P(Y=k\mid X)\) be a version of the conditional probability, and write \(\eta(X)=(\eta_1(X),\ldots,\eta_K(X))\). Assume \(\eta_k(X)>0\) for every class almost surely. A probability-valued prediction is a measurable vector \(q(X)\in\Delta_K^\circ\). At a feature value \(X=x\), its conditional risk is

Apply the risk–regret identity with \(p=\eta(x)\). This yields the conditional-risk decomposition, for almost every \(x\):

The first term does not depend on the report. The second is nonnegative and equals zero only when \(q(x)=\eta(x)\). Thus the conditional probability vector is the unique pointwise minimizer. If the reports are measurable functions of \(X\), choosing \(q(X)=\eta(X)\) is itself measurable, so it provides the minimizing rule under the assumptions above. This is the log-loss version of pointwise Bayes-risk minimization.

Interpretation and Common Pitfalls

The divergence quantifies the cost of a probability report under the true distribution. It is not a distance in the ordinary metric sense: in general, \(D_{\mathrm{KL}}(p\|q)\) is not equal to \(D_{\mathrm{KL}}(q\|p)\). The orientation matters because the risk averages \(-\log q_k\) using the true probabilities \(p_k\). This averaging produces \(D_{\mathrm{KL}}(p\|q)\), not the reversed expression.

• Confusing the report with the observed label. The loss is \(-\log q_y\): it evaluates the probability assigned to the label that actually occurred, not the probability of the most likely label.
• Reversing the divergence. Starting with risk under the true distribution gives \(D_{\mathrm{KL}}(p\|q)\). Swapping \(p\) and \(q\) changes the quantity and generally changes its value.
• Calling entropy the regret. Entropy \(H(p)\) is the minimum risk itself. Regret is the excess \(R_p(q)-H(p)\), which equals the divergence.
• Assuming every probability report has zero regret. Only the truthful report has zero regret under strictly positive \(p\). A report that differs from \(p\) has strictly positive regret.

In applications, the identity makes log loss useful for evaluating the quality of probability forecasts, not merely their hard classifications. Two reports can select the same most likely class and still have different log-loss risks because they assign different probabilities to the realized outcomes. The divergence decomposition explains exactly how those probability differences translate into excess risk.