Jensen Bounds for Convex-Loss Aggregation

Why Convexity Gives an Aggregation Guarantee

Averaging predictions is appealing because different predictors may make different errors. But “averaging helps” is not a universal theorem: the loss function, the aggregation weights, and the comparison being made all matter. Convexity gives a precise guarantee. With nonnegative weights summing to one, the loss of the average prediction cannot exceed the weighted average of the individual losses. Taking expectations turns this pointwise statement into a risk bound.

For squared error, there is more than an inequality: the average member risk is exactly the ensemble risk plus a nonnegative measure of how far the members’ predictions spread around their average. That identity depends on the algebra of squared loss and does not automatically extend to other convex losses.

Setup and Definitions

Let \((X,Y)\) be a test observation, and let \(f_1,\ldots,f_M\) be predictors with values in a convex set \(\mathcal C\). For an observation with response \(Y=y\), write the loss of predicting \(a\in\mathcal C\) as \(\ell(y,a)\). Assume that \(a\mapsto\ell(y,a)\) is convex on \(\mathcal C\) for every \(y\). The risk of predictor \(f_m\) is the expected test loss.

For the risk statements below, assume the predictors and loss are measurable and the expectations are well-defined. A simple sufficient condition is that all the displayed risks are finite. If predictors and weights are learned from a training sample, the same argument applies conditional on that sample when the test pair \((X,Y)\) is independent of it and the weights are nonnegative and sum to one. Averaging fitted predictors does not itself make the test observation independent of training; that independence is an assumption about evaluation.

The Convex-Loss Risk Bound

The result follows from finite Jensen’s inequality, applied separately at every test observation. For completeness, finite Jensen’s inequality follows directly from the definition of convexity. For two points it is that definition. For \(M\) points, discard zero weights. If there is more than one positive weight, let \(s=\sum_{m=1}^{M-1}w_m\), so \(0<s<1\), and set \(a=\sum_{m=1}^{M-1}(w_m/s)a_m\). By induction and then convexity,

Here \(\phi\) is any convex function on a convex domain. Applying this inequality with \(\phi(a)=\ell(y,a)\), \(a_m=f_m(x)\), and the given weights yields, for each \((x,y)\),

Taking expectations preserves the inequality. By linearity of expectation for a finite weighted sum, its right-hand side becomes \(\sum_m w_m E[\ell(Y,f_m(X))]\), which is \(\sum_m w_m R(f_m)\). This proves the theorem. \(\square\)

Why Squared Error Has an Exact Diversity Identity

For squared loss, the difference between average member loss and aggregate loss can be calculated exactly. Define the weighted prediction spread at \(x\) as \(\sum_m w_m(f_m(x)-\bar f(x))^2\). It is nonnegative because it is a weighted sum of squares.

To prove it, write each member’s residual as \(y-f_m(x)=(y-\bar f(x))+(\bar f(x)-f_m(x))\), square, and sum with the weights:

The first term simplifies using \(\sum_m w_m=1\). The cross-term is zero because \(\sum_m w_m(\bar f(x)-f_m(x))=\bar f(x)\sum_m w_m-\sum_m w_m f_m(x)=0\). The remaining terms give the stated identity. Taking expectations on both sides gives the risk version; the finite-moment condition ensures that all terms are integrable. \(\square\)

What the Guarantee Does—and Does Not—Say

The convex-loss theorem compares aggregate risk with the weighted average of member risks. It does not imply that the aggregate has risk no larger than the best member’s risk. For example, if one member is much worse than the others and receives substantial weight, the weighted average itself can exceed the best member’s risk. Nor does the theorem assert that every averaging scheme improves risk relative to an arbitrary reference predictor.

For a strictly convex loss, equality in pointwise Jensen’s inequality with positive weights requires all positively weighted predictions to be equal. For the risk inequality, equality holds if the pointwise Jensen gap is zero with probability one: the gap is nonnegative, so its expectation can be zero only if it is zero almost surely. With a loss that is convex but not strictly convex, different predictions can still yield equality. The absolute-loss example demonstrates why strict convexity cannot be assumed.

The squared-error identity is more informative than a general convexity bound because it identifies the exact gap. For other convex losses, Jensen’s inequality still gives the upper bound, but there need not be a prediction-spread term with the same exact decomposition. A useful analysis should therefore distinguish a universal convexity guarantee from an identity special to squared error.

Common Pitfall: Applying Jensen’s Inequality to Negative Weights

Jensen’s inequality requires nonnegative weights summing to one. A sum-to-one constraint alone is not enough. For instance, take squared loss, \(y=0\), predictions \(0\) and \(2\), and weights \(1.5\) and \(-0.5\). The weights sum to one, but the aggregate prediction is \(1.5(0)-0.5(2)=-1\), with loss \(1\). The weighted member loss is \(1.5(0^2)-0.5(2^2)=-2\). The putative bound \(1\leq-2\) is false. Negative weights permit extrapolation rather than a convex combination, so the convexity proof does not apply.

A second practical caution concerns evaluation. If weights are selected using the same test responses on which risk is reported, the simple conditional argument does not justify the claimed test-risk bound: the weights are then tied to the evaluation outcomes. The pointwise inequality remains true for any realized nonnegative weights summing to one, but comparing data-fitted risk estimates with population risks requires accounting for how those weights were chosen.