Equicorrelation and Effective Ensemble Size

Why Correlation Limits the Benefit of Averaging

Averaging predictions can reduce variability, but the reduction depends on whether members make errors together. If their errors are independent, averaging \(M\) equally variable members divides the error variance by \(M\). If they tend to err in the same direction, part of their variation is shared and remains in the average. The correlation among member errors therefore gives a direct way to quantify how much averaging is buying.

This tutorial considers a fixed prediction target and a specified source of randomness, such as variation across training samples or algorithmic randomization. Write \(E_m\) for member \(m\)'s prediction error under that randomness. The calculation concerns the variance of the average error. It does not, by itself, account for the average error's mean or establish that the ensemble has lower total squared-error risk.

Equal variance and equal pairwise correlation are substantive assumptions, not automatic properties of an ensemble. They are useful because they reduce the variance calculation to just three quantities: \(M\), \(\sigma^2\), and \(\rho\). The shared value \(\rho\) describes dependence in the errors, not similarity of the fitted models in every possible sense.

The Variance Formula and Effective Size

Expanding the variance of a sum gives \(M\) variance terms and \(M(M-1)\) ordered covariance terms. Under equicorrelation, all the variance terms equal \(\sigma^2\) and all the covariance terms equal \(\rho\sigma^2\). Dividing by \(M^2\) yields the ensemble variance.

For comparison, an average of \(K\) independent errors, each with variance \(\sigma^2\), has variance \(\sigma^2/K\). This motivates defining the effective ensemble size as the number \(K\) of independent, equally variable members that would give the same variance as the actual ensemble. When \(\operatorname{Var}(\bar E)>0\), solving \(\sigma^2/K=\operatorname{Var}(\bar E)\) gives:

When \(\rho=0\), the effective size is \(M\): the average has the same variance as \(M\) independent members. Positive correlation makes \(M_{\mathrm{eff}}\) smaller than \(M\), while a valid negative correlation makes it larger. In particular, \(M_{\mathrm{eff}}\) can exceed the actual number of members; this means that the average has lower variance than an independent average of the same size, not that the ensemble contains additional models.

Worked Calculations

Which Equicorrelations Are Possible?

A proposed correlation must correspond to a positive semidefinite correlation matrix. For \(M\) equicorrelated errors, that matrix has the form \(R=(1-\rho)I+\rho\mathbf{1}\mathbf{1}^{\mathsf T}\), where \(I\) is the \(M\) by \(M\) identity matrix and \(\mathbf{1}\) is the vector of \(M\) ones.

Proof. For a vector \(v\) proportional to \(\mathbf{1}\), the matrix \(R\) acts by multiplication by \(1+(M-1)\rho\), because \(\mathbf{1}^{\mathsf T}\mathbf{1}=M\). For any vector \(v\) perpendicular to \(\mathbf{1}\), we have \(\mathbf{1}^{\mathsf T}v=0\), so \(Rv=(1-\rho)v\). The space perpendicular to \(\mathbf{1}\) has dimension \(M-1\). These are all the eigenvalues: \(1+(M-1)\rho\) once, and \(1-\rho\) with multiplicity \(M-1\). A real symmetric matrix is positive semidefinite exactly when all its eigenvalues are nonnegative. Thus the necessary and sufficient conditions are \(1+(M-1)\rho\ge 0\) and \(1-\rho\ge 0\), equivalent to the stated bounds. \(\square\)

At the lower boundary, \(1+(M-1)\rho=0\), so the variance formula gives \(\operatorname{Var}(\bar E)=0\). In this singular case, the effective size defined by dividing \(\sigma^2\) by the ensemble variance is infinite; the finite formula for \(M_{\mathrm{eff}}\) has a zero denominator. At \(\rho=1\), the errors have the same variance and correlation one, and the average has variance \(\sigma^2\), equal to that of a single member.

What Happens as the Ensemble Grows?

For a fixed \(\rho\) between zero and one, the variance formula can be written as a shared component, \(\sigma^2\rho\), plus a component, \(\sigma^2(1-\rho)/M\), that decreases with ensemble size. In particular, for fixed \(\rho>0\), the variance approaches \(\sigma^2\rho\), rather than zero. More members reduce the unshared variation but cannot remove this shared part.

The reduction from adding one member can also be calculated exactly. For \(0\le\rho\le1\), \(\operatorname{Var}(\bar E_M)-\operatorname{Var}(\bar E_{M+1})=\sigma^2(1-\rho)/\{M(M+1)\}\). This difference is nonnegative and decreases as \(M\) increases. If \(\rho=1\), adding members gives no variance reduction; if \(\rho=0\), the familiar independent-member reduction is recovered.

Interpretation and a Common Pitfall

Effective size is useful when comparing ensembles with different member counts or dependence levels. Two ensembles can have different values of \(M\) but the same variance-equivalent size. It also makes clear why increasing the number of trees or other members need not yield a proportional reduction in prediction variability: the dependence among their errors matters.

Common mistake: “There are \(M\) members, so averaging divides the variance by \(M\).” This reasoning silently assumes zero covariance. In general, the covariance terms contribute \(\sigma^2(M-1)\rho/M\) to the average variance, giving \(\sigma^2\{\rho+(1-\rho)/M\}\), not simply \(\sigma^2/M\). The correction is to account for the pairwise error correlation before interpreting the ensemble size.

Finally, variance is only one part of prediction performance. If all members share a nonzero mean error, averaging does not remove that common mean. Effective size compares variances to an independent-member benchmark; it does not guarantee a particular mean squared error improvement, nor does it replace analysis of bias or total risk.