The Diversity Gain Identity For Squared Loss

Why Disagreement Matters Under Squared Loss

A weighted predictor combines several candidate predictions into one. Under squared loss, its risk is not merely bounded by the weighted average of the component risks: the difference between those quantities has an exact expression. It is the average squared disagreement of the component predictions around their weighted average.

This identity explains why averaging can help even when the weights are fixed in advance. If the predictors make different predictions on the evaluation distribution, their average can reduce risk relative to their weighted average risk. The size of that reduction depends on disagreement, not on which predictor happens to be closest to the response at a particular observation.

Definition: Let $Z=(X,Y)$ have a distribution $P$, and let $f_1,\ldots,f_M$ be measurable predictors, where $f_j(X)$ predicts the real-valued response $Y$. Their squared-error risks are $R(f_j)=E[(Y-f_j(X))^2]$. For fixed weights $\lambda_1,\ldots,\lambda_M$ satisfying $\lambda_j\geq 0$ and $\sum_{j=1}^M\lambda_j=1$, define the weighted predictor $\bar f_\lambda(x)=\sum_{j=1}^M\lambda_j f_j(x)$. Assume $E[Y^2]<\infty$ and $E[f_j(X)^2]<\infty$ for every $j$, so all risks and quantities below are finite.

The weights lie on the probability simplex, so $\bar f_\lambda(x)$ is a convex combination of the predictions at $x$. The weights are fixed for the risk calculation. The identity below does not determine how weights should be chosen; it describes what any such fixed choice does under squared loss.

The Diversity Gain Identity

Theorem: Under the conditions above, $$ \sum_{j=1}^M \lambda_j R(f_j) = R(\bar f_\lambda) + E\!\left[\sum_{j=1}^M\lambda_j\bigl(f_j(X)-\bar f_\lambda(X)\bigr)^2\right]. $$ The final expectation is the diversity, or disagreement, term. In particular, $R(\bar f_\lambda)\leq \sum_{j=1}^M\lambda_jR(f_j)$.

The theorem follows from an identity that holds separately at every possible pair of values $x,y$. Write $f_j$ for $f_j(x)$ and $\bar f_\lambda$ for $\bar f_\lambda(x)$ during this calculation. Expanding around the weighted prediction gives

\begin{aligned} \sum_{j=1}^M\lambda_j(y-f_j)^2 &=\sum_{j=1}^M\lambda_j\bigl((y-\bar f_\lambda)+(\bar f_\lambda-f_j)\bigr)^2\\ &=(y-\bar f_\lambda)^2 +2(y-\bar f_\lambda)\sum_{j=1}^M\lambda_j(\bar f_\lambda-f_j) +\sum_{j=1}^M\lambda_j(f_j-\bar f_\lambda)^2. \end{aligned}

The middle term is zero, because $\sum_j\lambda_j(\bar f_\lambda-f_j)=\bar f_\lambda\sum_j\lambda_j-\sum_j\lambda_jf_j=\bar f_\lambda-\bar f_\lambda=0$. Thus the pointwise squared-loss decomposition is

\sum_{j=1}^M\lambda_j(y-f_j(x))^2 = (y-\bar f_\lambda(x))^2+ \sum_{j=1}^M\lambda_j(f_j(x)-\bar f_\lambda(x))^2.

Take expectation with respect to $(X,Y)$. The left side becomes $\sum_j\lambda_jR(f_j)$, by linearity of expectation and finiteness of the sum. The first term on the right becomes $R(\bar f_\lambda)$; the remaining term becomes the stated diversity. This proves the theorem. $\square$

The cancellation explains why the disagreement term contains no $Y$. Once the predictors and the distribution of $X$ are fixed, diversity depends only on how their predictions vary across $X$, not on the observed response values. The response still matters to each risk, but its contribution cancels in the risk difference.

Worked Example: Two Predictions Around a Constant Response

Suppose $Y=2$ with probability one, and two predictors always return $f_1(X)=0$ and $f_2(X)=4$. Let $\lambda_1=\lambda_2=1/2$. Calculate the component risks, the weighted predictor's risk, and the diversity.

The component risks are $R(f_1)=(2-0)^2=4$ and $R(f_2)=(2-4)^2=4$. Therefore their weighted average risk is $(1/2)4+(1/2)4=4$.

The weighted prediction is $\bar f_\lambda(X)=(1/2)0+(1/2)4=2$, so its risk is $R(\bar f_\lambda)=(2-2)^2=0$.

The diversity is $(1/2)(0-2)^2+(1/2)(4-2)^2=(1/2)4+(1/2)4=4$. Thus $4=0+4$: the entire weighted component risk is the diversity gain, and averaging predicts the response exactly.

A Pairwise Form of Disagreement

Diversity can also be written directly in terms of pairwise differences. This form makes clear that it measures disagreement among predictions, rather than their distance from the weighted prediction alone.

Identity: For any real numbers $a_1,\ldots,a_M$ and simplex weights $\lambda_1,\ldots,\lambda_M$, with $\bar a=\sum_j\lambda_ja_j$, $$ \sum_{j=1}^M\lambda_j(a_j-\bar a)^2 = \frac12\sum_{j=1}^M\sum_{k=1}^M\lambda_j\lambda_k(a_j-a_k)^2. $$

To prove it, expand the right side and use $\sum_j\lambda_j=1$:

\begin{aligned} \frac12\sum_{j,k}\lambda_j\lambda_k(a_j-a_k)^2 &=\frac12\sum_{j,k}\lambda_j\lambda_k(a_j^2+a_k^2-2a_ja_k)\\ &=\sum_j\lambda_ja_j^2-\left(\sum_j\lambda_ja_j\right)^2\\ &=\sum_j\lambda_ja_j^2-\bar a^2. \end{aligned}

Expanding the left side gives $\sum_j\lambda_j(a_j^2-2a_j\bar a+\bar a^2)=\sum_j\lambda_ja_j^2-2\bar a^2+\bar a^2=\sum_j\lambda_ja_j^2-\bar a^2$. Both sides are therefore equal, proving the identity. $\square$

Apply this algebra at each $X$, taking $a_j=f_j(X)$, and then take expectations. The risk gap has the pairwise form

\sum_{j=1}^M\lambda_jR(f_j)-R(\bar f_\lambda) = \frac12\sum_{j=1}^M\sum_{k=1}^M \lambda_j\lambda_k E\!\left[(f_j(X)-f_k(X))^2\right].

The terms with $j=k$ are zero, and each unordered pair contributes its squared disagreement weighted by the product of its weights. In particular, a pair matters only when both predictors have positive weight.

Worked Example: Nonuniform Weights on a Two-Point Distribution

Let $X$ take values $a$ and $b$ with probability $1/2$ each. Suppose $Y(a)=0$, $Y(b)=2$, and two predictors have values $f_1(a)=0,\ f_1(b)=2$ and $f_2(a)=2,\ f_2(b)=0$. Set $\lambda_1=1/4$ and $\lambda_2=3/4$.

Predictor $f_1$ matches $Y$ at both points, so $R(f_1)=0$. Predictor $f_2$ has error $2$ at $a$ and error $-2$ at $b$, so $R(f_2)=(1/2)4+(1/2)4=4$. The weighted component risk is $(1/4)0+(3/4)4=3$.

The averaged predictions are $\bar f_\lambda(a)=(1/4)0+(3/4)2=3/2$ and $\bar f_\lambda(b)=(1/4)2+(3/4)0=1/2$. The errors are $3/2$ at $a$ and $-3/2$ at $b$, giving $R(\bar f_\lambda)=(1/2)(9/4)+(1/2)(9/4)=9/4$.

The pairwise form has only one nonzero unordered pair contribution: $\lambda_1\lambda_2 E[(f_1(X)-f_2(X))^2]=(1/4)(3/4)4=3/4$, since the predictors differ by 2 at both points. Indeed, the risk gap is $3-9/4=3/4$. The nonuniform weights reduce the gain relative to equal weighting because the disagreement is weighted by $\lambda_1\lambda_2$.

When Is the Gain Strict?

The diversity term is an expectation of nonnegative quantities, so it is zero exactly when every positive-weight predictor agrees with the weighted prediction almost surely under the distribution of $X$. Equivalently, all predictors that receive positive weight must agree with one another almost surely. The equivalence follows from the pairwise expression: every summand is nonnegative, and a pair with two positive weights has a positive coefficient.

Consequently, the weighted predictor has strictly smaller risk than the weighted average of component risks if and only if there is positive-probability disagreement among predictors with positive weights. Disagreement only on a set of $X$-values having probability zero does not create a risk gain. Nor does disagreement from a predictor assigned zero weight.

Worked Example: Three Constant Predictions with Unequal Weights

Let $Y=1$ with probability one. Take constant predictions $f_1=0,\ f_2=2,\ f_3=5$, with weights $\lambda_1=0.2,\ \lambda_2=0.3,\ \lambda_3=0.5$. Verify the identity using both the weighted deviations and the pairwise disagreements.

The component risks are $1,\ 1,$ and $16$, respectively. Their weighted average is $0.2(1)+0.3(1)+0.5(16)=0.2+0.3+8=8.5$.

The weighted prediction is $\bar f_\lambda=0.2(0)+0.3(2)+0.5(5)=0.6+2.5=3.1$. Its risk is $(1-3.1)^2=(-2.1)^2=4.41$, so the risk gap is $8.5-4.41=4.09$.

The weighted deviations are $0.2(0-3.1)^2=0.2(9.61)=1.922$, $0.3(2-3.1)^2=0.3(1.21)=0.363$, and $0.5(5-3.1)^2=0.5(3.61)=1.805$. Their sum is $1.922+0.363+1.805=4.09$.

For the pairwise check, the three unordered pairs contribute $0.2(0.3)(0-2)^2=0.24$, $0.2(0.5)(0-5)^2=2.50$, and $0.3(0.5)(2-5)^2=1.35$. Their sum is $0.24+2.50+1.35=4.09$, as required.

Interpretation and a Common Pitfall

The identity compares the risk of the weighted prediction with the weighted average of the component risks. It does not say that the weighted predictor beats the best component. For instance, if $Y=0$, $f_1=0$, and $f_2=2$, with equal weights, the best component has risk $0$, while the average prediction $1$ has risk $1$. The weighted component risk is $2$, and the diversity gain is $1$, so the identity still holds: averaging improves on $2$, but not on the best risk $0$.

A related mistake is to treat disagreement as automatically beneficial without specifying the comparator. Disagreement creates a nonnegative reduction relative to the weighted average component risk. It does not guarantee low absolute risk: all predictors may disagree and still be far from the response. The identity is an exact accounting statement, not a claim that the candidates are accurate.

The result is specific in its exact form to squared loss, where expansion produces a cross term that vanishes because the weighted deviations sum to zero. It is useful for understanding fixed aggregation and for separating two issues: how accurate the candidates are, and how much their predictions disagree. The risk identity alone does not choose the weights or establish that any particular weighting is statistically optimal.

Key takeaway: For fixed simplex weights, the weighted average risk exceeds the risk of the averaged prediction by exactly the weighted prediction diversity. The gain is strict precisely when positively weighted predictors disagree on a set of positive probability.

Check Your Understanding

Use the identity and its conditions to answer the following questions.

Why does the cross term vanish when squared loss is expanded around the weighted prediction?
If one predictor has weight zero, can its disagreement with the other predictors create a diversity gain? Explain.
Suppose two positively weighted predictors agree almost surely under the distribution of $X$. What does the identity imply about the risk gap?
Does the diversity gain identity imply that the weighted predictor has lower risk than the lowest-risk component? Give a reason for your answer.
In the pairwise form, why does each unordered pair contribute with coefficient $\lambda_j\lambda_k$, rather than just one of the two weights?