Density Forecast Aggregation Under Log Score

Why Aggregate Density Forecasts?

A density forecast assigns a density to possible outcomes before the outcome is observed. If several forecasts are available, one direct way to combine them is to average their densities using nonnegative weights that sum to one. This operation is called a mixture or linear pool. Under the log score, the key fact is that the mixture's score on an observation is no worse than the weighted average of the component scores on that same observation.

The comparison concerns scores, not a claim that the mixture must outperform every component. Its log score can be worse than the best component's score, while still being bounded by the weighted average of all component scores. We will distinguish this observation-by-observation fact from the corresponding statement about expected risk.

Definition: Let $q_1,\ldots,q_M$ be probability densities with respect to a common measure, and let $w_1,\ldots,w_M$ satisfy $w_j\geq 0$ and $\sum_{j=1}^M w_j=1$. The mixture density is $q_{\mathrm{mix}}(y)=\sum_{j=1}^M w_jq_j(y)$. The log loss of a density forecast $q$ at observation $y$ is $-\log q(y)$, with loss $+\infty$ if $q(y)=0$. The logarithm is natural.

A density is a nonnegative function integrating to one; a probability mass function on a discrete outcome space is treated in the same way, with sums in place of integrals. For a sample $Y_1,\ldots,Y_n$, the empirical log risk of $q$ is $\widehat R_n(q)=n^{-1}\sum_{i=1}^n-\log q(Y_i)$. These quantities are not necessarily nonnegative: a continuous density can exceed one at an observation.

The Pointwise Log-Score Comparison

Theorem: For any observation $y$ at which all component densities with positive weight are positive, $$ -\log q_{\mathrm{mix}}(y)\leq \sum_{j:w_j>0}w_j[-\log q_j(y)]. $$ If a positive-weight component has density zero at $y$, the right-hand side is $+\infty$, so the inequality holds in the extended-real sense as well.

Proof. The function $x\mapsto-\log x$ is convex on $x>0$, since its second derivative is $1/x^2>0$. Apply the finite Jensen inequality to the positive-weight values $q_j(y)$:

-\log\!\left(\sum_{j:w_j>0} w_jq_j(y)\right) \leq \sum_{j:w_j>0}w_j[-\log q_j(y)].

The quantity inside the logarithm on the left is $q_{\mathrm{mix}}(y)$, proving the result. If a positive-weight component is zero, its log loss is infinite, and the stated extended-real comparison follows. $\square$

Worked Example: Scoring a Mixture at One Observation

Suppose three forecasts have weights $0.40, 0.35, 0.25$. At the observed value $y$, their density values are $0.10, 0.40, 0.24$, respectively. Calculate the mixture density and compare its loss with the weighted component losses.

The mixture density at $y$ is $0.40(0.10)+0.35(0.40)+0.25(0.24)=0.04+0.14+0.06=0.24$. Thus its log loss is $-\log(0.24)\approx1.42712$.

The weighted component losses are $0.40[-\log(0.10)]+0.35[-\log(0.40)]+0.25[-\log(0.24)]$. Using $-\log(0.10)\approx2.302585$, $-\log(0.40)\approx0.916291$, and $-\log(0.24)\approx1.427116$, this equals $0.921034+0.320702+0.356779\approx1.59852$. Therefore $1.42712\leq1.59852$, as the pointwise result requires.

Applying the pointwise inequality separately to each observation and averaging gives $\widehat R_n(q_{\mathrm{mix}})\leq\sum_jw_j\widehat R_n(q_j)$. This empirical comparison needs no sampling assumption: it holds for any finite set of observations for which the scores are defined.

Worked Example: Empirical Risks for Three Discrete Forecasts

Twenty outcomes are observed in three categories, with counts $8,7,5$, so their empirical frequencies are $0.40,0.35,0.25$. Consider forecasts $q_1=(0.50,0.30,0.20)$, $q_2=(0.30,0.40,0.30)$, and $q_3=(0.20,0.50,0.30)$, with mixture weights $0.40,0.35,0.25$. Each vector is a valid probability mass function because its entries are nonnegative and sum to one.

The mixture probabilities for the three categories are $0.40(0.50)+0.35(0.30)+0.25(0.20)=0.355$, $0.40(0.30)+0.35(0.40)+0.25(0.50)=0.385$, and $0.40(0.20)+0.35(0.30)+0.25(0.30)=0.260$. They sum to one.

For $q_1$, the empirical risk is $0.40[-\log(0.50)]+0.35[-\log(0.30)]+0.25[-\log(0.20)]$ $=0.277259+0.421390+0.402360\approx1.10101$. For $q_2$, it is $0.40[-\log(0.30)]+0.35[-\log(0.40)]+0.25[-\log(0.30)]$ $=0.481589+0.320702+0.300993\approx1.10328$. For $q_3$, it is $0.40[-\log(0.20)]+0.35[-\log(0.50)]+0.25[-\log(0.30)]$ $=0.643775+0.242601+0.300993\approx1.18737$.

The weighted average of those component risks is $0.40(1.10101)+0.35(1.10328)+0.25(1.18737)\approx1.12339$. The mixture's empirical risk is $0.40[-\log(0.355)]+0.35[-\log(0.385)]+0.25[-\log(0.260)]$ $=0.414255+0.334079+0.336769\approx1.08510$. Hence $1.08510\leq1.12339$. In this example the mixture also beats the best component, but the general comparison guarantees only the weighted-average bound.

From Observed Scores to Expected Risk

For a population comparison, let $P$ be the distribution of a future observation $Y$, and let $p$ be its density with respect to the same measure as the forecast densities. Define the log risk by $R_p(q)=E_p[-\log q(Y)]$. Because density values can exceed one, this expectation is an expectation of a signed random variable rather than necessarily a nonnegative quantity. We use a finite-risk condition that makes all expectations and the comparison unambiguous.

Theorem: Suppose $M$ is finite, the weights form a probability vector, and for every $j$ with $w_j>0$, $E_p[|\log q_j(Y)|]<\infty$. Then $E_p[|\log q_{\mathrm{mix}}(Y)|]<\infty$, and $$ R_p(q_{\mathrm{mix}})\leq\sum_{j:w_j>0}w_jR_p(q_j). $$ In particular, the mixture risk is no greater than the weighted average of the component risks.

Proof. First establish integrability, so that the pointwise comparison can be integrated. Choose any index $k$ with $w_k>0$. Since $q_{\mathrm{mix}}\geq w_kq_k$, on the event $q_{\mathrm{mix}}(Y)\leq1$ we have $(-\log q_{\mathrm{mix}}(Y))^+\leq(-\log(w_kq_k(Y)))^+\leq(-\log q_k(Y))^++|\log w_k|$. Here $a^+=\max(a,0)$. The last expression is integrable by the assumption on $q_k$.

For the other part, $q_{\mathrm{mix}}(Y)\leq\max_{j:w_j>0}q_j(Y)$. Therefore $(\log q_{\mathrm{mix}}(Y))^+\leq\sum_{j:w_j>0}(\log q_j(Y))^+$, which is integrable because there are finitely many components and each $|\log q_j(Y)|$ is integrable. Thus both parts of $\log q_{\mathrm{mix}}(Y)$ are integrable. Also, each positive-weight $q_j(Y)$ is positive $P$-almost surely under the finite-log assumption. The pointwise theorem consequently applies $P$-almost surely. Taking expectations, now justified by integrability, yields $E_p[-\log q_{\mathrm{mix}}(Y)]\leq\sum_jw_jE_p[-\log q_j(Y)]$, as claimed. $\square$

Since a weighted average is at least the smallest of its terms, the theorem also gives $R_p(q_{\mathrm{mix}})\leq\sum_jw_jR_p(q_j)$, but it does not in general give $R_p(q_{\mathrm{mix}})\leq\min_jR_p(q_j)$. A mixture is guaranteed to compare favorably with the specified weighted average, not necessarily with the best component.

Worked Example: A Continuous Density Can Have Negative Log Loss

At observation $y=0$, compare $q_1$, the density of $N(0,0.04)$, and $q_2$, the density of $N(0,1)$. Use mixture weights $0.25$ and $0.75$. The normal density at zero is $1/\sqrt{2\pi\sigma^2}$, so $q_1(0)=1/\sqrt{0.08\pi}\approx1.994711$ and $q_2(0)=1/\sqrt{2\pi}\approx0.398942$.

The mixture density is $q_{\mathrm{mix}}(0)=0.25(1.994711)+0.75(0.398942)\approx0.797885$. Its log loss is $-\log(0.797885)\approx0.22579$. The component losses are $-\log(1.994711)\approx-0.69050$ and $-\log(0.398942)\approx0.91894$. Their weighted average is $0.25(-0.69050)+0.75(0.91894)\approx0.51658$, which is larger than the mixture loss. The first component's loss is negative because its density value at zero exceeds one; this is valid for a continuous density and is not a negative probability.

Common Mistakes and Practical Interpretation

Averaging losses instead of densities. The aggregate forecast in this tutorial is $q_{\mathrm{mix}}=\sum_jw_jq_j$. Its loss is $-\log(\sum_jw_jq_j(y))$, not the weighted average loss itself. The theorem compares these two quantities; they are generally not equal.
Claiming the mixture beats the best component. The guaranteed bound is against $\sum_jw_jR_p(q_j)$. A weighted average can exceed its smallest term, so the result alone does not establish superiority to the best component.
Assuming log losses are always nonnegative. That is true for discrete probabilities, which are at most one, but not for continuous densities. A density can exceed one, giving a negative log loss.
Integrating without checking finiteness. A pointwise inequality is not by itself enough to justify expectations of signed quantities. The risk theorem states an absolute-integrability condition and verifies integrability of the mixture log density before taking expectations.

Key takeaway: For a common observation, convexity of negative logarithm makes the log loss of a weighted mixture no greater than the weighted component losses. With finite component log risks, the same comparison holds for expected risks.

Check Your Understanding

Use the definitions and comparison results to answer these questions.

Two forecasts assign densities $0.2$ and $0.8$ at an observation, with weights $0.25$ and $0.75$. What is the mixture density there?
Why does the pointwise comparison not imply that a mixture must have lower risk than the best component?
Can a continuous density forecast have negative log loss at an observation? Explain.
For the discrete forecasts in the worked example, why is the empirical risk calculated using category frequencies?
What role does the absolute-integrability condition play in the population-risk proof?

Density Forecast Aggregation Under Log Score

What You'll Learn