Log-Loss Boundary Probabilities and Nonattainment

Why the Boundary Matters

Log loss is often introduced for forecasts that assign a positive probability to every class. But a true conditional distribution need not do so: given some information, an outcome may be impossible, or may occur with probability one. The distinction between allowing zero forecast probabilities and forbidding them then matters. With the closed probability simplex, the true conditional probabilities can be a valid minimizer. With the open simplex, that same optimum may be approached arbitrarily closely but never attained.

This is not a failure of log loss to identify the correct probabilities. It is a consequence of restricting the set of forecasts. To see precisely where that restriction bites, consider a finite set of \(K\ge 2\) classes.

Conditional Log Risk on the Simplex

Write \(p=(p_1,\ldots,p_K)\) for the conditional class-probability vector and \(q=(q_1,\ldots,q_K)\) for a forecast. Both have nonnegative coordinates summing to one. If the outcome is class \(j\), log loss is \(-\log q_j\), with \(-\log 0=+\infty\). The conditional expected loss is

In this sum, a term with \(p_j=0\) is defined to be zero, even if \(q_j=0\). Such a class has zero probability under the conditional distribution, so its loss does not contribute to the expectation. In contrast, if \(p_j>0\) and \(q_j=0\), then \(R_p(q)=+\infty\): the forecast assigns zero probability to an outcome that can occur.

Define the entropy of \(p\) by \(H(p)=-\sum_{j:p_j>0}p_j\log p_j\). It is finite for a finite number of classes. The conditional regret of \(q\) relative to \(p\) is \(R_p(q)-H(p)\). When this regret is finite, it equals the Kullback–Leibler divergence from \(p\) to \(q\), with zero-probability coordinates omitted from the sum.

The Closed-Simplex Result

The claim includes boundary vectors \(p\). In particular, if a class is impossible, placing zero probability on it is permitted and can be part of the optimal forecast. The proof also establishes that assigning probability to impossible classes cannot improve the risk.

Proof. Let \(S=\{j:p_j>0\}\). If a coordinate in \(S\) has \(q_j=0\), the result follows from infinite risk. Otherwise,

For every \(t>0\), \(\log t\le t-1\), with equality exactly at \(t=1\). Therefore

Thus \(R_p(q)\ge H(p)\). If equality holds throughout, the last inequality requires \(\sum_{j\in S}q_j=1\), and equality in each logarithm inequality requires \(q_j/p_j=1\) for every \(j\in S\). Hence \(q_j=p_j\) on \(S\), and no probability remains for coordinates outside \(S\). Thus \(q=p\). Conversely, \(q=p\) has risk \(H(p)\), proving both attainment and uniqueness. \(\square\)

What Changes on the Open Simplex?

For an interior \(p\), the closed-simplex minimizer is itself an allowed open-simplex forecast. For a boundary \(p\), that minimizer is excluded. To see that the infimum is still \(H(p)\), let \(u=(1/K,\ldots,1/K)\) be the uniform vector and, for \(0<\varepsilon<1\), define

Every coordinate of \(q_\varepsilon\) is positive, so \(q_\varepsilon\in\Delta_K^\circ\). For \(j\in S=\{j:p_j>0\}\), \(q_{\varepsilon,j}\ge(1-\varepsilon)p_j\). Combining this bound with the closed-simplex result gives

The upper bound tends to zero as \(\varepsilon\) decreases to zero. Thus the open-simplex infimum is \(H(p)\). If \(p\) is on the boundary, no open-simplex vector equals \(p\); by the uniqueness proved above, every open-simplex forecast has risk strictly greater than \(H(p)\). The infimum is therefore not attained. If \(p\) is interior, the same uniqueness result applies and \(q=p\) attains the minimum. \(\square\)

Conditional Probabilities That Depend on Features

In prediction, the probability vector typically depends on observed features \(X\). Let \(p(X)\) be a measurable version of the conditional class-probability vector, and let \(q(X)\) be a measurable forecast. The expected log risk is the expectation of the conditional risk \(R_{p(X)}(q(X))\). Pointwise, its lowest possible value is \(H(p(X))\).

If forecasts may use the closed simplex, choosing \(q(X)=p(X)\) attains that value. If forecasts must lie in the open simplex, smoothing \(p(X)\) toward the uniform vector gives open-simplex forecasts. The bound just proved holds at every feature value:

Taking expectations preserves the bound, so these forecasts approach the expected conditional entropy. If \(p(X)\) lies on the boundary with positive probability, however, no open-simplex forecast attains that optimum: on those feature values its conditional risk is strictly larger. A nonnegative measurable gap that is strictly positive on a set of positive probability has strictly positive expectation (possibly infinite). If boundary values occur only on a probability-zero set, the restriction need not prevent attainment: a forecast can equal \(p(X)\) almost surely and be modified on that null set to be strictly positive.

Common Mistake and Practical Implications

A common mistake is to reason that because a zero-probability class contributes zero to the expected loss, it is harmless to require its forecast probability to be positive and then claim the optimum is still attained. The zero contribution explains why the boundary forecast has finite risk; it does not make that forecast a member of the open simplex. For a deterministic outcome, every strictly positive probability assigned elsewhere forces the probability of the certain class below one and makes its log loss strictly positive.

A related notational trap is confusing the entropy convention \(0\log 0=0\) with the realized loss \(-\log 0=+\infty\). Entropy is an expectation-like sum in which a zero-probability term is omitted. Realized log loss is infinite if the observed class was assigned probability zero. These facts are compatible: an event of conditional probability zero contributes nothing to expected risk, although assigning zero probability to an event that can occur produces infinite risk.

In computation, probability clipping or smoothing is often used to avoid infinite numerical losses. It creates forecasts in the open simplex and can be useful operationally, but a fixed smoothing amount generally changes the forecast and leaves positive excess risk. The distinction here is exact: on the closed simplex a boundary optimum is valid; on the open simplex it may only be approximated. The relevant choice is therefore not whether log loss “works” at the boundary, but whether the permitted forecast space includes that boundary.