Logarithmic Score and Boundary Penalties

Why the Logarithmic Score Is Sensitive to Zero Probabilities

A probability forecast is assessed by what it says about the outcome that actually occurs. The logarithmic score makes this connection direct: a forecast receives low loss when it assigned high probability to the observed outcome, and high loss when it assigned low probability. In particular, assigning probability zero to an outcome that occurs incurs infinite loss. This boundary behavior is not a technical nuisance added after the fact; it is part of the definition of the score.

We use the loss convention, so smaller scores are better. Throughout, assume the outcome space is finite, \(\mathcal{Y}=\{1,\ldots,K\}\). A true distribution is \(p=(p_1,\ldots,p_K)\), and a forecast is \(q=(q_1,\ldots,q_K)\), where each vector has nonnegative entries summing to one. The logarithm is natural unless stated otherwise.

A scoring rule is strictly proper when reporting the true distribution uniquely minimizes expected loss: for every \(p\), \(R(p,q)\ge R(p,p)\), with equality only when \(q=p\). The logarithmic loss has this property. Its risk difference is exactly the Kullback–Leibler divergence from \(p\) to \(q\), including the possibility that the difference is infinite.

Entropy and the Kullback–Leibler Risk Difference

For a finite distribution \(p\), define its entropy by \(H(p)=-\sum_{y:p_y>0}p_y\log p_y\). The Kullback–Leibler divergence from \(p\) to \(q\) is \(D_{\mathrm{KL}}(p\|q)=\sum_{y:p_y>0}p_y\log(p_y/q_y)\) when every \(q_y\) with \(p_y>0\) is positive. If \(p_y>0\) and \(q_y=0\) for even one outcome, define \(D_{\mathrm{KL}}(p\|q)=+\infty\). Terms with \(p_y=0\) are omitted; in particular, they are not treated as an indeterminate \(0/0\).

Proof. First suppose \(q_y>0\) whenever \(p_y>0\). Expanding the definition gives

If \(q_y=0\) for some \(y\) with \(p_y>0\), then the risk and the divergence are both infinite. Since \(H(p)\) is finite on a finite outcome space, the same identity holds in the extended-real sense.

It remains to prove nonnegativity and the equality condition. If \(q_y=0\) at any outcome with \(p_y>0\), the divergence is infinite and is therefore positive. Otherwise, use the elementary inequality \(\log t\le t-1\) for \(t>0\), with equality exactly at \(t=1\). For each \(y\) with \(p_y>0\), take \(t=q_y/p_y\). Then

The final inequality follows because the entries of \(q\) sum to one, so its total mass on the support of \(p\) is at most one. For the divergence to equal zero, both inequalities must be equalities. Equality in the first requires \(q_y/p_y=1\) for every \(y\) with \(p_y>0\). Equality in the second requires \(q\) to place no mass outside that support. Together these conditions say \(q=p\). Conversely, substituting \(q=p\) gives divergence zero. The risk identity now shows that \(R(p,q)\ge R(p,p)\), with equality only at \(q=p\). \(\square\)

Zero Probability and Infinite Loss

The definition of logarithmic loss gives a precise boundary penalty. If \(p_y>0\) but \(q_y=0\), then outcome \(y\) has positive probability of occurring, while the forecast assigns it zero probability. Its loss on that event is infinite, and its contribution to expected loss is infinite as well. By contrast, if \(p_y=0\), that outcome never occurs under the true distribution, so assigning it zero forecast probability creates no infinite risk.

Interpreting the Boundary Penalty

The same boundary rule applies to empirical risk. Given observed outcomes \(y_1,\ldots,y_n\), the empirical logarithmic risk of a fixed forecast \(q\) is \(n^{-1}\sum_{i=1}^n\ell(q,y_i)\). If even one observed outcome has \(q_{y_i}=0\), the empirical risk is infinite. This is the sample analogue of the population result: a realized outcome contradicts the forecast’s assertion that it was impossible.

A very small positive probability and a zero probability are not interchangeable. For example, if an observed outcome has forecast probability \(0.01\), its loss is \(-\log(0.01)\approx4.605\), which is large but finite. At probability zero the loss is infinite. Changing the logarithm’s base changes the units, not the conclusion: natural logarithms measure loss in nats, while base-2 logarithms measure it in bits.

Common Mistake

Wrong reasoning: “If the forecast gives an outcome probability zero, that outcome contributes nothing to expected loss.” This confuses the forecast probability \(q_y\) with the true probability \(p_y\). Risk is averaged under the true distribution. When \(p_y>0\), the event can occur, and a forecast probability of zero makes its loss infinite. The zero contribution applies only when \(p_y=0\), because then the event cannot occur under the distribution used to calculate risk.

A related mistake is to claim that every zero entry in a forecast makes its risk infinite. That is too strong: a zero forecast probability causes infinite risk only when the corresponding true probability is positive. The support condition matters.