Diversity Scores That Fail to Predict Risk

Why Diversity Is Not the Same as Accuracy

An ensemble is often called diverse when its members make different predictions or have weakly correlated errors. That can be useful: when members err differently, averaging may cancel some of their errors. But a diversity score describes relationships among members, not necessarily how close their predictions are to the target. Two ensembles can have similar diversity and very different risks; an ensemble with more diversity can even have higher risk.

We will use squared-error risk to make the distinction precise. Fix the training data and regard the fitted predictors as fixed functions. Let $(X,Y)$ be a fresh test observation from the target population, and let $f_1,\ldots,f_m$ be the member predictors. The equal-weight ensemble is their arithmetic mean. All expectations below are over the test observation, conditional on the fitted predictors. The same identities hold unconditionally if expectations also average over training-data randomness.

Definition: The member error is $e_j=f_j(X)-Y$, and the equal-weight ensemble error is $\bar e=m^{-1}\sum_{j=1}^m e_j$. The ensemble squared-error risk is $R_{\mathrm{ens}}=E[\bar e^2]$. The member error mean, variance, and pairwise covariance are $\mu_j=E[e_j]$, $\sigma_j^2=\operatorname{Var}(e_j)$, and $\operatorname{Cov}(e_i,e_j)$, respectively.

A correlation is a standardized covariance: when both variances are positive, $\rho_{ij}=\operatorname{Cov}(e_i,e_j)/(\sigma_i\sigma_j)$. It records the direction and strength of linear association, but not the error means or the scales $\sigma_i$. Those omissions matter because squared risk is measured relative to zero error, not relative to each member’s own average error.

What the Exact Risk Formula Contains

Theorem: For fixed predictors with finite second moments, the equal-weight ensemble risk is $$ R_{\mathrm{ens}} = \left(\frac{1}{m}\sum_{j=1}^m\mu_j\right)^2 + \frac{1}{m^2}\left\{ \sum_{j=1}^m\sigma_j^2+ 2\sum_{1\le i<j\le m}\operatorname{Cov}(e_i,e_j) \right\}. $$ If every $\sigma_j>0$, each covariance in this expression can equivalently be written as $\rho_{ij}\sigma_i\sigma_j$.

The first term is the squared mean error of the ensemble. It can remain large even if the members have errors that vary in opposite directions. The second term accounts for their centered variation, including both their scales and their covariances. Thus correlations become informative about risk when the relevant means and scales are also known; correlations alone generally do not determine the risk.

Separate mean and centered error.
Write $e_j=\mu_j+(e_j-\mu_j)$. The centered terms have expectation zero.

Expand the square of the ensemble error.
The squared mean contributes $(m^{-1}\sum_j\mu_j)^2$; cross-terms between the mean and centered errors have expectation zero.

Collect the centered second moments.
The diagonal terms are variances, and each off-diagonal covariance appears twice in the square.

For completeness, expanding directly gives

E[\bar e^2] =\frac{1}{m^2}\sum_{i=1}^m\sum_{j=1}^m E[e_i e_j] =\frac{1}{m^2}\sum_{i,j}\{\mu_i\mu_j+\operatorname{Cov}(e_i,e_j)\}.

The mean terms sum to $(m^{-1}\sum_j\mu_j)^2$. The covariance terms have diagonal entries $\sigma_j^2$, and symmetric off-diagonal entries, so their sum is $m^{-2}\{\sum_j\sigma_j^2+2\sum_{i<j}\operatorname{Cov}(e_i,e_j)\}$. This proves the formula. $\square$

Worked Example: Lower Correlation, Higher Risk

Let $Y=0$, and let $Z$ equal $1$ or $-1$, each with probability $1/2$. Compare two two-member ensembles. In ensemble A, set $e_1=2+Z$ and $e_2=2-Z$. In ensemble B, set $e_1=Z$ and $e_2=Z$.

For A, both error means are $2$, and both variances are $1$. Their covariance is $\operatorname{Cov}(2+Z,2-Z)=E[Z(-Z)]=-1$, so their correlation is $-1$. The ensemble error is $\{(2+Z)+(2-Z)\}/2=2$ for either value of $Z$, giving risk $2^2=4$.

For B, both error means are $0$, both variances are $1$, and their covariance and correlation are $1$. The ensemble error is $Z$, so its risk is $E[Z^2]=1$. Thus A has the lower pairwise error correlation but four times the ensemble risk. Its opposing fluctuations cancel, but its shared mean error of $2$ does not.

Disagreement Measures a Gain, Not an Absolute Risk

For regression under squared loss, a common diversity measure is the expected squared difference between member predictions. Since the same $Y$ is subtracted from each prediction, $f_i(X)-f_j(X)=e_i-e_j$. The next identity makes clear how this measure relates to risk.

Theorem: If the members have finite second moments, define their average individual risk as $R_{\mathrm{ind}}=m^{-1}\sum_{j=1}^m E[e_j^2]$, and define their mean pairwise squared disagreement as $$ \Delta=\frac{1}{m^2}\sum_{1\le i<j\le m}E[(f_i(X)-f_j(X))^2]. $$ Then $R_{\mathrm{ens}}=R_{\mathrm{ind}}-\Delta$.

To prove the identity, for any fixed numbers $e_1,\ldots,e_m$,

\sum_{i<j}(e_i-e_j)^2 =m\sum_{i=1}^m e_i^2-\left(\sum_{i=1}^m e_i\right)^2.

Indeed, each $e_i^2$ occurs in $m-1$ pairs on the left, and the cross-products contribute $-2\sum_{i<j}e_ie_j$. The right side has the same expansion. Divide by $m^2$, take expectations, and use $(\sum_i e_i)^2/m^2=\bar e^2$. The first term becomes $m^{-1}\sum_iE[e_i^2]=R_{\mathrm{ind}}$, and the second is $R_{\mathrm{ens}}$. Because prediction differences equal error differences, the remaining term is $\Delta$, proving the identity. $\square$

This is an exact statement, but its interpretation is easy to overreach. Greater disagreement means a greater reduction from the average individual risk for that particular ensemble. It does not say the ensemble’s final risk is low. If the average individual risks differ across ensembles, disagreement alone does not determine which final risk is smaller. If their average individual risks are equal, however, the identity does rank them: the larger $\Delta$, the lower $R_{\mathrm{ens}}$.

Worked Example: Same Correlation, Different Error Scales

Let $U$ and $V$ be independent random variables, each taking values $1$ and $-1$ with equal probability. Compare two zero-mean error pairs. For A, let $e_1=10U$ and $e_2=10(-0.1U+\sqrt{0.99}V)$. For B, let $e_1=U$ and $e_2=0.9U+\sqrt{0.19}V$.

For A, the second member has variance $100(0.1^2+0.99)=100$, the first has variance $100$, and their covariance is $100(-0.1)=-10$. Thus their correlation is $-10/(10\cdot10)=-0.1$. The ensemble risk is $$ \frac{1}{4}\{100+100+2(-10)\}=45. $$

For B, the second member has variance $0.9^2+0.19=1$, the first has variance $1$, and their covariance is $0.9$. Their correlation is $0.9$, and the ensemble risk is $$ \frac{1}{4}\{1+1+2(0.9)\}=0.95. $$

Each ensemble has zero error means, so this comparison isolates scale and covariance. The more negatively correlated pair, A, has much higher risk because its errors are ten times larger in standard deviation. A correlation ranking without error scales can therefore reverse the risk ranking.

Global Scores Can Hide Where Errors Occur

A score averaged over the target distribution weights observations according to their frequency, but a single aggregate number does not reveal which regions contribute to it. Further, member disagreement does not show whether members share a common error in an important region. The following example holds the global disagreement fixed while changing the ensemble’s common error.

Worked Example: Equal Disagreement, Different Region-Specific Risk

Let $X$ take values $A$ and $B$ with probabilities $P(X=A)=1/4$ and $P(X=B)=3/4$, and let $Y=0$ in both regions. For ensemble C, define its common prediction component $c(X)=0$ everywhere and its difference component $d(A)=2,\ d(B)=0$. Set $f_1=c+d/2$ and $f_2=c-d/2$.

In C, the two predictions at $A$ are $1$ and $-1$; at $B$, they are both $0$. The squared disagreement has expectation $(1/4)(2^2)+(3/4)(0^2)=1$. Their average prediction is $c(X)=0$ in both regions, so the ensemble risk is $0$.

For ensemble D, take $d(A)=d(B)=1$, and set $c(A)=2,\ c(B)=0$, again with $f_1=c+d/2$ and $f_2=c-d/2$. Its predictions at $A$ are $2.5$ and $1.5$, and at $B$ they are $0.5$ and $-0.5$. Squared disagreement is $1$ in both regions, so its expected value is also $1$. But its average prediction is $2$ at $A$ and $0$ at $B$, making its risk $(1/4)(2^2)+(3/4)(0^2)=1$.

The two ensembles have the same global squared disagreement, yet their risks differ. In D, the members disagree while sharing a prediction error in region A. The score records their separation, not the level of their average prediction relative to the target.

How to Read Diversity Scores Carefully

A diversity statistic is most useful when treated as one part of a risk analysis rather than as a substitute for it. Under squared error, the exact formulas indicate what else to check: average member risk, member error means, error scales, and the covariance structure. When the target distribution has distinct regions, it can also be useful to inspect risks and disagreement by region instead of relying exclusively on a global average.

These conclusions are specific to the risk and aggregation being analyzed. The identities above use equal-weight averaging and squared loss; another loss function or aggregation rule requires its own analysis. Likewise, if predictors are randomized through training or algorithmic randomness, be clear whether a reported correlation or risk is conditional on a fitted ensemble or averages over that randomness. Mixing conditional and unconditional quantities can produce a misleading comparison.

Common Mistakes

“The ensemble with lower error correlation must have lower risk.” Correlation omits error means and scales. Compare the full risk expression, not just its standardized covariance terms.
“More disagreement means a more accurate ensemble.” Under squared error, disagreement measures the reduction from average individual risk. It does not determine the absolute risk unless the average individual risks are also held fixed.
“Members cancel because they disagree.” Their deviations may cancel while their shared error remains. Check the ensemble’s mean error and its behavior across relevant regions.
“The global score tells us where the ensemble succeeds.” An average can conceal a common error concentrated in a consequential region. Examine region-specific risks when that distinction matters.

Key takeaway: Diversity can help explain how averaging changes risk, but it is not itself a risk guarantee. For squared-error ensembles, interpret pairwise correlations alongside error means and scales, and interpret disagreement alongside average member risk and the target regions where errors occur.

Check Your Understanding

Use the definitions and identities above to answer the following.

Which components of the exact squared-error risk formula are not specified by pairwise correlations alone?
Two ensembles have equal average individual risk. Which has lower ensemble risk: the one with greater or smaller mean pairwise squared disagreement?
Why can two members have correlation $-1$ while their ensemble still has nonzero risk?
In the region-specific example, what does the disagreement score measure, and what important feature of ensemble D does it fail to measure?
Under what conditions can correlation information, together with known variances and zero error means, determine equal-weight ensemble risk?