Jensen's Inequality for Convex Loss Aggregates

Why Averaging Predictions Can Help

An ensemble can be formed in two different ways. One may randomly select a candidate predictor according to a set of weights, or one may average the candidates’ predictions at each observation. The second procedure is a deterministic mixture. For a convex loss, its risk is no greater than the weighted average of the candidates’ risks. Thus, averaging predictions can improve on random selection without requiring a claim that it must outperform the best candidate.

The statement depends on more than the weights summing to one. The predictions must lie in a convex set so their average is a valid prediction, and the loss must be convex as a function of the prediction for each fixed outcome. The result then follows by applying Jensen’s inequality separately at each observation and integrating.

Setup and Conditions

For the finite aggregation result, take measurable predictors \(g_1,\ldots,g_M\) whose values lie in \(C\), and fixed weights \(w_1,\ldots,w_M\) satisfying \(w_j\geq 0\) and \(\sum_{j=1}^M w_j=1\). Their pointwise weighted average is \[ \bar g(x)=\sum_{j=1}^M w_j g_j(x). \] Convexity of \(C\) ensures \(\bar g(x)\in C\). Assume the loss is jointly measurable in its arguments, nonnegative, and convex in its prediction, and that every candidate risk \(R(g_j)\) is finite. These conditions make the risks well-defined and justify the expectation argument below.

The nonnegativity assumption is a convenient sufficient condition for integrating the pointwise inequality without ambiguity. If losses may take negative values, the same conclusion holds when all terms in the inequality are integrable. If weights are selected using training data, the result applies conditional on that data when the risk is evaluated on a fresh observation; it also applies directly to empirical averages when the weights and predictions are defined on the observations being averaged.

Finite-Weight Jensen’s Inequality

The first ingredient is Jensen’s inequality for a finite set of predictions. We include the argument to make explicit how weights equal to zero and arbitrary finite numbers of candidates are handled.

Proof. Discard any terms with zero weight; they contribute nothing to either side. For one positive-weight term, the inequality is equality. For two terms, it is the definition of convexity. Proceed by induction. Suppose the result holds for \(M-1\) terms, and consider \(M\) positive weights. Write \(s=\sum_{j=1}^{M-1}w_j\), so \(0<s<1\). Define \(b=s^{-1}\sum_{j=1}^{M-1}w_j a_j\), which belongs to \(C\) by convexity. Then \[ \sum_{j=1}^M w_j a_j=s b+w_Ma_M. \] Applying convexity to this two-point combination and then the induction hypothesis gives \[ h\left(\sum_{j=1}^M w_j a_j\right) \leq s h(b)+w_M h(a_M) \leq \sum_{j=1}^{M-1}w_j h(a_j)+w_M h(a_M). \] This proves the result for \(M\), and hence for every finite \(M\). \(\square\)

The Risk Bound for a Deterministic Mixture

Proof. A finite weighted sum of measurable predictors is measurable. Since \(C\) is convex and the weights form a convex combination, \(\bar g(x)\in C\) for every \(x\). For each observation \((x,y)\), apply finite-weight Jensen’s inequality to the convex function \(a\mapsto\ell(y,a)\) and the predictions \(g_1(x),\ldots,g_M(x)\): \[ \ell\left(y,\sum_{j=1}^M w_jg_j(x)\right) \leq\sum_{j=1}^M w_j\ell\bigl(y,g_j(x)\bigr). \] Both sides are measurable and nonnegative, so taking expectations preserves the inequality. Linearity of expectation for the finite sum yields \[ E[\ell(Y,\bar g(X))] \leq\sum_{j=1}^M w_j E[\ell(Y,g_j(X))] =\sum_{j=1}^M w_jR(g_j). \] The right-hand side is finite by assumption, so \(R(\bar g)\) is finite as well. \(\square\)

The bound compares risks under the same distribution \(P\). It is not a claim about how well the aggregate performs on an unrelated distribution, nor does it by itself quantify the statistical error from estimating the weights.

Three Details That Matter in Applications

First, convexity is needed for each fixed outcome. The theorem does not require the outcome distribution itself to have any special form. Second, the average is taken in prediction space, not by averaging the losses and then calling that result a prediction. Third, the theorem uses the same weights at each input \(x\). If weights vary with \(x\), the pointwise argument still works whenever the weights are measurable, nonnegative, sum to one at every \(x\), and the resulting prediction lies in \(C\).

A useful comparison is with a randomized Gibbs prediction. If an index \(J\) is drawn with \(P(J=j)=w_j\) and predictor \(g_J\) is used, then its risk, averaging over both the draw and a fresh observation, is \[ \sum_{j=1}^M w_jR(g_j). \] The theorem says that using \(\bar g\) instead has risk no greater than this quantity for a convex loss. The two procedures are not identical: one returns a randomly selected candidate prediction, while the other returns their pointwise average.

When the Conditions Fail

A common mistake is to argue that an average of predictions must have smaller loss simply because it averages the candidates. That reasoning omits the essential conditions. For example, with zero-one classification loss, the prediction space of class labels is discrete and is not convex as a subset of a numerical vector space. Averaging labels can produce an object that is not a class label at all. If one then applies a separate threshold or rounding rule, the resulting classifier is not the pointwise convex average covered by the theorem, and its risk need not obey this bound.

Likewise, convexity of the prediction space alone is insufficient: the loss must also be convex in the prediction. If either condition is missing, Jensen’s inequality cannot be invoked in this form. Do not infer the result merely from the fact that the weights sum to one.