Exact Mixture Guarantees Under Log Loss

Why Log Loss Is Special for Mixtures

A mixture density combines candidate distributions by averaging their densities at each possible observation. Under log loss, this construction has a precise identity: the loss of the mixture equals a responsibility-weighted average of component losses plus a nonnegative Kullback–Leibler divergence. The responsibilities depend on the observation, so this is not generally the same as averaging component risks using the original mixture weights.

That identity gives a useful population guarantee. A mixture cannot have greater log risk than a positively weighted component by more than the negative logarithm of that component's weight. In particular, it gives a comparison with the best dictionary member, with a penalty determined by how much weight the mixture assigns to that member.

Densities, Mixture Weights, and Responsibilities

Let \(f_1,\ldots,f_M\) be probability densities or mass functions with respect to one common reference measure \(\mu\). Let \(w_1,\ldots,w_M\) be nonnegative weights summing to one. The mixture density is \(f_w=\sum_{j=1}^M w_j f_j\). For an observation \(y\) at which \(f_w(y)>0\), define the responsibility of component \(j\) by its share of the mixture density at \(y\).

The responsibilities are nonnegative and sum to one. If \(w_j=0\) or \(f_j(y)=0\), then \(r_j(y)=0\). In particular, a component with zero responsibility must not be treated as if its log loss were an ordinary finite number: if \(f_j(y)=0\), then \(-\log f_j(y)=+\infty\). We will therefore sum component losses only over indices with positive responsibility.

The Exact Pointwise Identity

The central identity holds at every \(y\) where the mixture density is positive. Its careful statement matters: the component-loss sum includes only components that actually have positive responsibility at that observation.

To prove the identity, fix any index with \(r_j(y)>0\). The definition of responsibility gives \(r_j(y)/w_j=f_j(y)/f_w(y)\). Consequently, \( \log(r_j(y)/w_j)=\log f_j(y)-\log f_w(y) \). Substitute this into the KL term and combine it with the weighted component losses:

The final equality uses \(\sum_j r_j(y)=1\). All terms used in the calculation are well-defined: the sums omit zero responsibilities, and every included component has positive density at \(y\). This proves the identity. \(\square\)

From the Pointwise Identity to Population Risk

Let \(Y\) have the true distribution \(P\), and define the population log risk of a density \(f\) by \(R(f)=E[-\log f(Y)]\), when this expectation is well-defined. The pointwise identity can be averaged, but exchanging it for an identity of finite expectations requires integrability. A simple sufficient condition is that \(E[|\log f_w(Y)|]<\infty\) and \(E[|\log f_j(Y)|]<\infty\) for each \(j\) with \(w_j>0\). Under this condition, the responsibility-weighted losses are integrable, and the pointwise identity gives

This identity is exact but does not, by itself, say that the mixture risk is at most the weighted average \(\sum_j w_jR(f_j)\). The weights inside the first expectation are \(r_j(Y)\), which generally vary with \(Y\). A direct comparison with each positively weighted component instead follows from a pointwise density inequality.

A Risk Guarantee for Every Positively Weighted Component

For any \(j\) with \(w_j>0\), nonnegativity of the other mixture terms gives \(f_w(y)\geq w_jf_j(y)\). When \(f_j(y)>0\), taking negative logarithms yields \(-\log f_w(y)\leq-\log f_j(y)-\log w_j\). If \(f_j(y)=0\), the right-hand side is \(+\infty\), so the extended-real inequality remains valid. Averaging gives the following result whenever the risks are well-defined; for example, it is enough that both log densities are absolutely integrable under \(P\).

The proof is precisely the pointwise inequality just established, integrated under \(P\). The additive term is the cost of assigning only weight \(w_j\) to the comparator. With uniform weights, \(w_j=1/M\), the guarantee becomes \(R(f_w)\leq\min_j R(f_j)+\log M\). For nonuniform weights, a member with a small prior weight has a larger comparison penalty.

Why the Best-Member Comparison Needs a Penalty

The mixture guarantee is penalized by the weight of the comparator; it does not generally imply \(R(f_w)\leq\min_j R(f_j)\). A mixture can assign little weight to the best member, allowing other components to dominate the mixture density even where that best member predicts well.

Interpretation and a Common Pitfall

The identity separates two features of log-score aggregation. The responsibility-weighted loss reflects which components explain the realized observation; the KL term measures the discrepancy between those observation-specific responsibilities and the original mixture weights. Meanwhile, the comparator guarantee follows from the simpler fact that the mixture density contains \(w_j f_j\) as one nonnegative summand.

• Pitfall: averaging with the wrong weights. The exact identity does not use \(w_j\) to average component losses. It uses \(r_j(Y)\), which depends on the observed outcome. Replacing responsibilities by the fixed mixture weights changes the expression and is generally incorrect.
• Pitfall: claiming an unpenalized best-member bound. The componentwise guarantee includes \(\log(1/w_j)\). It cannot be dropped without additional assumptions, such as a suitable lower bound on the weight assigned to the best member.
• Pitfall: retaining zero-responsibility infinite losses. If \(r_j(y)=0\) and \(f_j(y)=0\), the product \(r_j(y)[-\log f_j(y)]\) is not defined by ordinary arithmetic. Omit that index from the sum; do not silently manipulate \(0\cdot\infty\).