Risk of Heterogeneous Members From Their Error Covariance Matrix

Why a Full Covariance Matrix Is Needed

An ensemble can combine predictors with different systematic errors, different levels of variability, and errors that move together. Averaging may reduce risk, but the amount of reduction cannot in general be determined from the members’ individual risks alone. The dependence between their errors matters: when errors tend to move together, averaging removes less variation than when they tend to offset one another.

This tutorial expresses that dependence directly through an error covariance matrix. The weights may be fixed in advance, and need not be equal. The goal is to calculate the risk of those weights—not to choose optimal weights.

Risk, Member Errors, and Their Covariance

Let \((X,Y)\) be a test observation, and let \(m(x)=E[Y\mid X=x]\) be the regression function, with \(E[Y^2]<\infty\). Suppose \(K\) fitted predictors \(\hat f_1,\ldots,\hat f_K\) are trained on a random training data set \(D\) independent of the test observation. Any additional algorithm randomization is included in the randomness of the fitted predictors. Let \(w=(w_1,\ldots,w_K)^\top\) be fixed weights, and define the ensemble prediction by \(\hat f_w(X)=\sum_{j=1}^K w_j\hat f_j(X)\). An ensemble commonly uses nonnegative weights summing to one, but the risk identity below holds for any fixed real weights for which the second moments exist.

These expectations average over both the test covariate and the randomness in training and fitting. In particular, \(\Sigma_{ii}=\operatorname{Var}(e_i)\) is the variance of the full error \(e_i\) under that joint averaging. This convention makes the risk formula a single quadratic expression. If instead one first fixes an input \(x\) and averages only over training randomness, the corresponding pointwise formula uses a conditional bias and covariance; integrating that formula over the distribution of \(X\) gives the excess risk.

The matrix includes unequal diagonal entries and all off-diagonal covariances. A positive \(\Sigma_{ij}\) means the two centered errors tend to move together; a negative value means they tend to move in opposite directions. The covariance matrix is symmetric and positive semidefinite: for every vector \(a\), \(a^\top\Sigma a=\operatorname{Var}(\sum_j a_j e_j)\geq 0\). Thus not every array of proposed pairwise covariances can be a valid error covariance matrix.

The Risk Formula

Proof. Let \(e=(e_1,\ldots,e_K)^\top\). The ensemble’s error relative to \(m(X)\) is \(w^\top e\), so its excess risk is \(E[(w^\top e)^2]=E[w^\top ee^\top w]\). Since \(w\) is fixed, this equals \(w^\top E[ee^\top]w\). For each pair \(i,j\), the definition of covariance gives \(E[e_i e_j]=\operatorname{Cov}(e_i,e_j)+E[e_i]E[e_j]=\Sigma_{ij}+b_i b_j\). Hence \(E[ee^\top]=\Sigma+bb^\top\), and \(w^\top(\Sigma+bb^\top)w=w^\top\Sigma w+(w^\top b)^2\). This proves both forms. \(\square\)

The formula separates two contributions. The squared weighted bias depends on the weighted sum of member biases, so biases with opposite signs can partly or completely cancel. The variance contribution uses every entry of \(\Sigma\), not just its diagonal. In expanded form it is \(\sum_i w_i^2\Sigma_{ii}+2\sum_{i<j}w_iw_j\Sigma_{ij}\). When weights are nonnegative, positive pairwise covariances add to the variance, while negative covariances reduce it.

From Excess Risk to Prediction Risk

Prediction risk is often reported against the observed response \(Y\), not against its conditional mean. The distinction matters because even a predictor equal to \(m(X)\) cannot remove the variability of \(Y\) around its conditional mean. Define the irreducible response variance by \(\sigma^2(X)=\operatorname{Var}(Y\mid X)\).

Proof. Write \(\varepsilon=Y-m(X)\). By definition, \(E[\varepsilon\mid X]=0\), and \(\hat f_w(X)-Y=(\hat f_w(X)-m(X))-\varepsilon\). Squaring and taking expectations yields \[ E[(\hat f_w(X)-Y)^2] =E[(\hat f_w(X)-m(X))^2]+E[\varepsilon^2] -2E[(\hat f_w(X)-m(X))\varepsilon]. \] Condition on \(X\) and the training and fitting randomness. By independence of the test pair from that randomness, the conditional mean of \(\varepsilon\) is \(E[\varepsilon\mid X]=0\). The cross-term therefore has expectation zero. Also \(E[\varepsilon^2]=E[\operatorname{Var}(Y\mid X)]=E[\sigma^2(X)]\). Applying the risk formula just proved to the first term gives the result. \(\square\)

Pointwise and Integrated Covariance

For some analyses, it is useful to condition on a fixed covariate value \(x\) and average only over the training and fitting randomness. Define \(b_j(x)=E_D[\hat f_j(x)]-m(x)\) and \(\Sigma_{ij}(x)=\operatorname{Cov}_D(\hat f_i(x)-m(x),\hat f_j(x)-m(x))\). If the relevant second moments are finite, the conditional excess risk at \(x\) is \[ E_D[(\hat f_w(x)-m(x))^2] =(w^\top b(x))^2+w^\top\Sigma(x)w. \] Integrating both sides over the distribution of \(X\) gives the population excess risk. The pointwise covariance \(\Sigma(x)\) describes how members’ fitted errors co-vary across training repetitions at that particular input. By contrast, the unconditional \(\Sigma\) used earlier also averages over variation in \(X\). These are different summaries; selecting one depends on which risk is being reported.

For example, even if a fitted member has no training randomness, its error can vary across covariate values. Its unconditional error variance can therefore be positive, while its pointwise training variance is zero. The integrated pointwise decomposition still captures the variation through \(b(x)\), which may differ across inputs. Keeping the averaging convention explicit prevents a covariance matrix calculated at fixed \(x\) from being mistaken for one describing the full population of prediction errors.

Common Mistakes and Interpretation

• Using only member variances. The diagonal of \(\Sigma\) is not enough. The risk also includes \(2w_iw_j\Sigma_{ij}\) for every distinct pair.
• Squaring and averaging in the wrong order. In general, the squared weighted bias is \((w^\top b)^2\), not the weighted average of squared member biases. Opposing biases can cancel before the square is taken.
• Calling excess risk total prediction risk. Risk against \(Y\) also includes \(E[\operatorname{Var}(Y\mid X)]\), under the assumptions stated above.
• Mixing covariance conventions. Covariance across training repetitions at fixed \(x\) is not automatically the same as covariance after also averaging over test covariates.
• Accepting an impossible covariance matrix. Any proposed error covariance matrix must be symmetric positive semidefinite. Otherwise some linear combination would have negative variance.

The practical value of the matrix representation is that it identifies which features of member errors determine risk: the weights, the weighted bias, each member’s variance, and every pairwise covariance. A count of members or an average of their separate risks cannot replace that information.