Bias–Variance–Covariance Decomposition for Averaged Predictors

Why Averaging Does Not Tell the Whole Risk Story

An ensemble prediction is often formed by taking the average of several fitted predictors. Averaging can reduce variability, but the amount of reduction depends on how the predictors’ errors move together. It also does not guarantee low bias: the average can remain systematically above or below the target. A useful risk calculation must therefore keep track of both the mean errors and their dependence.

We work conditionally at a fixed input $x$. Randomness in a fitted predictor can come from its training sample, algorithmic randomization, or both. The decomposition below separates that randomness from the noise in a future response. It applies to any finite number of predictors with finite second moments; it does not require that they be identically distributed or independent.

Definition: Let $m(x)=E[Y\mid X=x]$, and let $\hat f_1(x),\ldots,\hat f_M(x)$ be $M$ random predictions at a fixed $x$. Their equal-weight average is $\bar f(x)=M^{-1}\sum_{j=1}^M\hat f_j(x)$. Define the error of member $j$, relative to the regression function, by $e_j(x)=\hat f_j(x)-m(x)$. Expectations, variances, and covariances of these errors are over predictor randomness at the fixed $x$.

Write $b_j(x)=E[e_j(x)]$ for member $j$'s bias, $v_j(x)=\operatorname{Var}(e_j(x))$ for its variance, and $c_{jk}(x)=\operatorname{Cov}(e_j(x),e_k(x))$ for the covariance between two members’ errors. Since $m(x)$ is fixed, these are also the variances and covariances of the predictions themselves. The average prediction’s bias is the average of the member biases:

\bar b(x)=E[\bar f(x)]-m(x)=\frac{1}{M}\sum_{j=1}^M b_j(x).

This is the average bias, not the average of the squared member biases. Biases of different signs can partly cancel before they are squared.

The Risk Decomposition

For squared-error prediction, the conditional risk at $x$ is $E[(Y-\bar f(x))^2\mid X=x]$, where the expectation includes the future response and the randomness in the fitted predictors. Assume $E[Y^2\mid X=x]<\infty$, each predictor has a finite second moment, and predictor randomness is independent of the future response conditional on $X=x$. This independence holds, for example, when the future observation is independent of the training data and algorithmic randomness.

Theorem: Under these assumptions, with $\sigma^2(x)=\operatorname{Var}(Y\mid X=x)$, the conditional squared-error risk of the average is $$ E[(Y-\bar f(x))^2\mid X=x] =\sigma^2(x)+\bar b(x)^2+\frac{1}{M^2}\left\{\sum_{j=1}^M v_j(x)+2\sum_{1\leq j<k\leq M}c_{jk}(x)\right\}. $$ The last two terms together are the variance of the averaged prediction. If risk is measured against the regression function $m(x)$ rather than a noisy response $Y$, omit $\sigma^2(x)$.

The variance component can also be displayed as an individual-variance contribution plus a cross-member contribution:

\operatorname{Var}(\bar f(x)) =\frac{1}{M}\left\{\frac{1}{M}\sum_{j=1}^M v_j(x)\right\} +\frac{2}{M^2}\sum_{1\leq j<k\leq M}c_{jk}(x).

Thus the average member variance is divided by $M$, but that is not in general the whole variance of the ensemble. The covariance sum can increase or decrease it. Covariance terms are not required to be positive individually, although the complete variance is nonnegative.

Separate response noise.
Write $Y-\bar f=(Y-m)+(m-\bar f)$. The response’s deviation from its conditional mean has mean zero.

Separate bias from variance.
The mean error of the averaged prediction contributes $\bar b^2$; the remaining contribution is $\operatorname{Var}(\bar f)$.

Expand the variance.
The variance of a sum contains every member variance and twice each distinct pairwise covariance.

Deriving the Decomposition

First, condition on $X=x$ and expand the squared error around the conditional mean $m(x)$. Because $E[Y-m(x)\mid X=x]=0$, and the future response is conditionally independent of predictor randomness, the cross term has expectation zero:

\begin{aligned} E[(Y-\bar f)^2\mid X=x] &=E[((Y-m)+(m-\bar f))^2\mid X=x]\\ &=\sigma^2(x)+E[(m-\bar f)^2\mid X=x]. \end{aligned}

The mean of $m-\bar f$ over predictor randomness is $-\bar b$. For any finite-second-moment random variable $Z$, $E[Z^2]=\operatorname{Var}(Z)+(E[Z])^2$. Apply this to $Z=m-\bar f$, whose variance equals $\operatorname{Var}(\bar f)$, to obtain the following result.

Result: The conditional risk is $\sigma^2(x)+\bar b(x)^2+\operatorname{Var}(\bar f(x))$.

To expand its final term, use bilinearity of covariance. This gives a second result that identifies exactly where dependence enters.

Result: For predictors with finite second moments, $$ \operatorname{Var}(\bar f(x)) =\frac{1}{M^2}\left\{\sum_{j=1}^M v_j(x)+2\sum_{1\leq j<k\leq M}c_{jk}(x)\right\}. $$

Proof. Since $\bar f=M^{-1}\sum_j\hat f_j$,

\begin{aligned} \operatorname{Var}(\bar f) &=\frac{1}{M^2}\operatorname{Var}\left(\sum_{j=1}^M\hat f_j\right)\\ &=\frac{1}{M^2}\left\{\sum_{j=1}^M\operatorname{Var}(\hat f_j) +2\sum_{1\leq j<k\leq M}\operatorname{Cov}(\hat f_j,\hat f_k)\right\}. \end{aligned}

The expansion follows by writing the centered sum as $\sum_j(\hat f_j-E[\hat f_j])$, squaring it, and taking expectations. Each diagonal term is a variance, and each off-diagonal product appears twice, once for each order of its indices. Subtracting the fixed value $m(x)$ does not change any of these variances or covariances, so they equal $v_j(x)$ and $c_{jk}(x)$. Combining this identity with the preceding risk result proves the theorem. $\square$

Worked Risk Calculations

Worked Example: Three Members With Different Biases

At a fixed $x$, suppose the three prediction errors have mean vector $(1,0,-1)$ and covariance matrix $$ \begin{pmatrix} 4&2&1\\ 2&4&2\\ 1&2&4 \end{pmatrix}. $$ Assume they are jointly normal, and let the future response noise have variance $1$, independent of the errors. Find the risk of the average.

The covariance matrix is valid: its leading principal minors are $4$, $4(4)-2^2=12$, and $36$, all positive, so it is positive definite. The mean error of the average is $(1+0-1)/3=0$, hence its squared bias is $0$. Each member variance is $4$, so the individual-variance contribution is $(4+4+4)/9=4/3$. The three distinct covariances sum to $2+1+2=5$, giving a cross-member contribution of $2(5)/9=10/9$.

Therefore the variance of the average prediction is $4/3+10/9=22/9$. Adding response noise gives conditional squared-error risk $1+22/9=31/9$. The members have nonzero biases, but their average bias cancels; their covariance still contributes to risk.

Worked Example: Independent Members With Unequal Variances

Suppose two prediction errors are independent, with $e_1\sim N(2,1)$ and $e_2\sim N(0,9)$. Calculate the mean squared error of their average as an estimate of $m(x)$.

The average error is $(e_1+e_2)/2$, so its mean is $(2+0)/2=1$, and its squared bias is $1$. Independence makes the covariance zero. Its variance is therefore $(1+9)/4=2.5$, and its mean squared error is $1+2.5=3.5$.

For comparison, the individual mean squared errors are $2^2+1=5$ and $0^2+9=9$; their arithmetic average is $7$, not the risk of the averaged predictor. The ensemble risk uses the square of the average bias and the variance of the average, rather than averaging individual risks.

Worked Example: Shared Randomness Raises Ensemble Variance

Let $Z,U,V$ be independent $N(0,1)$ variables, and let two member errors be $e_1=Z+U$ and $e_2=Z+V$. Suppose the future response noise has variance $1/4$ and is independent of these variables. Find the risk of the average.

Both errors have mean zero and variance $1+1=2$. Their covariance is $\operatorname{Cov}(Z+U,Z+V)=\operatorname{Var}(Z)=1$, since all other covariance terms are zero. The average error is $Z+(U+V)/2$, whose variance is $1+1/4+1/4=1.5$. Its bias is zero, so risk against $m(x)$ is $1.5$; risk for the future response is $1.5+0.25=1.75$.

If these two errors instead had the same marginal variances but were independent, their average variance would be $(2+2)/4=1$, and the response risk would be $1.25$. The shared component $Z$ is precisely why the actual average has greater risk.

Interpreting and Using the Formula

The decomposition matters because the same number of ensemble members can produce different risks depending on their error dependence. If all cross-member covariances are zero, the variance term reduces to the sum of member variances divided by $M^2$. With nonzero covariances, that reduction does not follow. Positive covariances add to risk; negative covariances subtract from the variance sum, while the full variance remains nonnegative.

For risk averaged over a random input $X$, the fixed-$x$ calculation can be integrated over the distribution of $X$, provided the relevant risks are integrable. In particular, $E[(Y-\bar f(X))^2]=E[\sigma^2(X)+\bar b(X)^2+\operatorname{Var}(\bar f(X)\mid X)]$, with the bias and variance evaluated at each input. The pointwise formula is useful because it shows whether risk comes from response noise, systematic error, or random prediction error.

Common Pitfall

A frequent mistake is to say that averaging $M$ predictors always divides prediction variance by $M$. That conclusion assumes equal member variances and zero covariance, and even then it concerns only the variance component, not squared bias or response noise. In general, the correct calculation includes $2\sum_{j<k}c_{jk}/M^2$, and the squared bias is $(M^{-1}\sum_j b_j)^2$, not $M^{-1}\sum_j b_j^2$. Check both the member biases and the error covariances before concluding that aggregation has lowered risk.

Check Your Understanding

Use the fixed-input decomposition to answer these questions.

For three members with biases $1,1,-2$, what is the bias of their equal-weight average, and what is its squared bias?
Why does conditional independence of the future response and predictor randomness make the response-noise cross term vanish?
Two independent, unbiased members each have variance $6$. What is the variance of their average? What assumption is essential to this calculation?
If two members have covariance $-1$, how does their covariance enter the variance of their equal-weight average?
Which terms in the decomposition remain when risk is measured against $m(x)$ rather than a future response $Y$?