The Pairwise Diversity Identity Under Squared Error

Why Pairwise Diversity Has an Exact Role

An ensemble averages predictions from several fitted models. Under squared error, its performance is not just related to the members’ errors: there is an exact accounting identity. The ensemble’s risk equals the average risk of its members minus a nonnegative term measuring how far their predictions disagree. This term is often called diversity, but its normalization matters: different definitions of diversity produce different coefficients in the identity.

The identity applies to equal-weight averages and squared-error loss. It is an algebraic fact at each test observation, not a claim that diverse models must be accurate. If all members make the same poor prediction, there is no diversity to subtract and averaging does not improve their shared error.

Definitions and Assumptions

For a random test pair \((X,Y)\), define the member risk \(R_m=E[(Y-\hat f_m(X))^2]\) and the ensemble risk \(R_{\mathrm{ens}}=E[(Y-\bar f_M(X))^2]\). Here the predictors are treated as fixed functions. Assume they are measurable and that \(E[Y^2]<\infty\) and \(E[\hat f_m(X)^2]<\infty\) for every \(m\). These sufficient moment conditions make the risks and diversity below finite.

The pointwise definition also has a pairwise form: \(d_M(x)=M^{-2}\sum_{i<j}(\hat f_i(x)-\hat f_j(x))^2\). The sum here is over unordered pairs. In particular, the denominator is \(M^2\), not the number of pairs. If diversity is instead defined as the unnormalized average over pairs, the coefficient in the risk identity must change.

The Pointwise Identity

Proof. Write \(y-a_m=(y-\bar a)-(a_m-\bar a)\), square, and average over \(m\):

The middle term is zero because \(\sum_m(a_m-\bar a)=\sum_m a_m-M\bar a=0\). This proves the first equality. To obtain the pairwise form, expand the sum over pairs:

On the other hand, \(\sum_m(a_m-\bar a)^2=\sum_m a_m^2-M\bar a^2 =\sum_m a_m^2-M^{-1}(\sum_m a_m)^2\). Thus the pairwise sum is \(M\sum_m(a_m-\bar a)^2\). Dividing by \(M^2\) gives exactly the diversity term in the theorem. \(\square\)

From Pointwise Algebra to Risk

Proof. Apply the pointwise theorem with \(y=Y\) and \(a_m=\hat f_m(X)\). Its first term on the left is \(M^{-1}\sum_m(Y-\hat f_m(X))^2\); its first term on the right is \((Y-\bar f_M(X))^2\). Taking expectations gives the risk identity, because expectation distributes over a finite sum. Applying the pairwise form of the pointwise theorem and taking expectations gives the displayed pairwise expression for \(D_M\). The assumed second moments ensure all squared terms are integrable, so these expectations are finite. \(\square\)

The result requires no independence among the members’ predictions. It follows from averaging real numbers at each input and then taking expectation. The pairwise terms are nonnegative, so the ensemble risk cannot exceed the average member risk under these conditions. That comparison is with the average member risk; it does not say the ensemble beats every member.

Random Fitted Predictors

In practice, a fitted predictor can depend on a random training sample or algorithmic randomness. The pointwise identity still applies after fitting: for each realized training sample, use the predictions it produced as the numbers \(a_1,\ldots,a_M\). If the relevant squared quantities have finite expectation over both training randomness and the test distribution, taking expectation over both sources preserves the identity. Equivalently, one can first condition on the training data, apply the fixed-predictor identity, and then average over the training-data distribution.

No independence between the fitted members is needed for this algebra. If test risk is defined conditionally on a realized training set, the identity holds conditionally. If risk is unconditional, average all terms over the training-set randomness as well. What must be specified is which randomness the risk averages over.

Equality, Interpretation, and a Common Pitfall

Because \(D_M\) is an expectation of nonnegative squared differences, it is nonnegative. For \(M\geq2\), it is zero exactly when all member predictions agree almost surely under the distribution of \(X\). To see this, the pairwise representation writes \(D_M\) as a finite sum of expectations of nonnegative random variables. If \(D_M=0\), each expectation is zero; a nonnegative random variable with expectation zero is zero almost surely. Thus every pair agrees almost surely. Conversely, almost-sure agreement makes every pairwise squared difference zero and hence \(D_M=0\). For \(M=1\), diversity is zero by definition.

The subtraction term is prediction spread, not a direct measure of accuracy. Large disagreement can lower the ensemble’s risk relative to the members’ average, but the member average itself may still be large. The identity is a comparison among risks for the same test distribution and the same squared-error loss; it does not establish performance under another loss or on a different population.

• Pitfall: treating any reported pairwise average as \(D_M\). If one defines the mean squared difference across unordered pairs as \(\binom{M}{2}^{-1}\sum_{i<j}(\hat f_i-\hat f_j)^2\), that quantity is not \(D_M\). The correct conversion is \(D_M=\frac{M-1}{2M}\) times that pair-average.
• Correction: State the normalization first. The identity as given uses \(D_M=M^{-2}\sum_{i<j}E[(\hat f_i(X)-\hat f_j(X))^2]\). Substituting a differently normalized diversity without changing its coefficient gives an incorrect risk calculation.
• Pitfall: concluding that diversity guarantees a good ensemble. The identity only subtracts diversity from average member risk. If all members are inaccurate in a similar way, their average risk can remain high.