When Averaging Trades Bias for Variance

Why Averaging Can Help—and Still Lose

Averaging predictions is often expected to make a predictor more stable. But stability is only part of the risk calculation. If the component predictors have different expected predictions, their average can move away from the target, adding squared bias. The relevant question is therefore not simply whether averaging reduces variance, but whether the variance it saves is greater than the bias cost it creates.

We compare predictors at a fixed input \(x\), so the target is the same for every component. All expectations below are over the randomness in training data and any algorithmic randomization. Assume each predictor has a finite second moment. The formulas are pointwise: they need not hold uniformly over \(x\) unless additional uniform conditions are imposed.

Definition: Let \(m(x)\) be the target at a fixed input \(x\), and let \(\hat f_j(x)\) be component predictor \(j\), for \(j=1,\ldots,K\). Its pointwise bias is \(b_j(x)=E[\hat f_j(x)]-m(x)\), its variance is \(v_j(x)=\operatorname{Var}(\hat f_j(x))\), and its squared-error risk for estimating \(m(x)\) is \(R_j(x)=E[(\hat f_j(x)-m(x))^2]\). The equally weighted average is \(\bar f(x)=K^{-1}\sum_{j=1}^K\hat f_j(x)\).

For squared-error prediction of a fresh response \(Y\), suppose the training data used to construct the predictors are independent of that response conditional on its input \(X=x\), and let \(m(x)=E[Y\mid X=x]\). Then the response-prediction risk is \(E[(Y-\bar f(x))^2\mid X=x]=\operatorname{Var}(Y\mid X=x)+E[(\bar f(x)-m(x))^2]\). The first term is the conditional noise and is unchanged by the choice of predictor. Thus, comparing predictors reduces to comparing their squared-error risks for \(m(x)\).

The Average’s Bias and Variance

Linearity of expectation gives the average predictor’s bias. Its variance depends not just on the individual variances but also on the covariances between components. These are exact statements; no independence assumption is needed.

Result: Write \(\bar b(x)=K^{-1}\sum_{j=1}^K b_j(x)\) and \(C_{jk}(x)=\operatorname{Cov}(\hat f_j(x),\hat f_k(x))\). Then \[ E[\bar f(x)]-m(x)=\bar b(x), \qquad \operatorname{Var}(\bar f(x))=\frac{1}{K^2}\sum_{j=1}^K\sum_{k=1}^K C_{jk}(x), \] and \(R_{\bar f}(x)=\bar b(x)^2+\operatorname{Var}(\bar f(x))\).

To obtain the variance formula, subtract the expectation from each predictor and expand the variance of their sum:

\operatorname{Var}(\bar f(x)) =\frac{1}{K^2}\operatorname{Var}\!\left(\sum_{j=1}^K\hat f_j(x)\right) =\frac{1}{K^2}\sum_{j=1}^K\sum_{k=1}^K \operatorname{Cov}(\hat f_j(x),\hat f_k(x)).

For the risk formula, write \(\bar f(x)-m(x)=[\bar f(x)-E\bar f(x)]+[E\bar f(x)-m(x)]\). The first bracket has expectation zero, so its cross-product with the constant second bracket has expectation zero. Squaring and taking expectations therefore gives variance plus squared bias.

Worked Example: Equal Bias and Correlated Components

At a fixed input, suppose there are \(K=4\) predictors. Each has bias \(0.5\) and variance \(9\), and every pair has correlation \(0.2\). Calculate the average’s risk and compare it with a component’s risk.

The average has the same bias, \(0.5\), because the four expected predictions are equal. Each off-diagonal covariance is \(0.2\sqrt{9\cdot9}=1.8\). There are four diagonal terms, each equal to \(9\), and twelve off-diagonal terms, each equal to \(1.8\). Hence

\operatorname{Var}(\bar f) =\frac{4(9)+12(1.8)}{4^2} =\frac{57.6}{16} =3.6.

A component’s risk is \(0.5^2+9=9.25\). The average’s risk is \(0.5^2+3.6=3.85\). Here averaging preserves the bias and reduces variance by \(9-3.6=5.4\), so its risk is lower by \(5.4\).

The Exact Bias–Variance Tradeoff

A comparison needs a benchmark. In particular, “better than a component” is incomplete unless the component is specified: a mean predictor can outperform some components but not the most accurate one. The following condition makes the comparison explicit.

Theorem: Fix a component \(j\). The equally weighted average has strictly lower squared-error risk than component \(j\) at \(x\) if and only if \[ \operatorname{Var}(\hat f_j(x))-\operatorname{Var}(\bar f(x)) > \bar b(x)^2-b_j(x)^2. \] Equality gives equal risks. Thus, if averaging increases squared bias relative to component \(j\), the variance reduction must exceed that increase.

Proof. By the risk decomposition, the difference between average risk and component \(j\)'s risk is

R_{\bar f}(x)-R_j(x) = \bar b(x)^2+\operatorname{Var}(\bar f(x)) -b_j(x)^2-\operatorname{Var}(\hat f_j(x)).

The average has lower risk exactly when this difference is negative. Rearranging that inequality gives the stated condition. Proof complete. \(\square\)

The right-hand side can be negative. That means averaging has reduced squared bias relative to component \(j\); in that case, the average could have slightly higher variance and still have lower total risk. Conversely, if the average’s squared bias is larger, a variance reduction that is real but too small does not justify averaging.

Worked Example: A Variance Saving That Pays for Added Bias

Let the target at \(x\) be \(0\). Two predictors have expectations \(0\) and \(1\), variances \(4\) and \(4\), and covariance \(0\). Compare their equally weighted average with the first component.

The component biases are \(b_1=0\) and \(b_2=1\). The average bias is \(\bar b=(0+1)/2=0.5\), so its squared bias is \(0.25\). Its variance is

\operatorname{Var}(\bar f) =\frac{4+4+2(0)}{2^2} =2.

The first component has risk \(0^2+4=4\). The average has risk \(0.5^2+2=2.25\). Relative to the first component, averaging adds \(0.25\) in squared bias but saves \(4-2=2\) in variance. Since \(2>0.25\), the average has lower risk.

Worked Example: A Variance Saving That Is Not Enough

Again let the target be \(0\). Suppose two uncorrelated predictors have expectations \(0\) and \(2\), and both have variance \(1\). Compare their average with the first predictor.

The first component’s bias is \(0\), while the average bias is \((0+2)/2=1\), with squared bias \(1\). The average variance is

\operatorname{Var}(\bar f) =\frac{1+1+2(0)}{2^2} =0.5.

The first component’s risk is \(0+1=1\); the average’s risk is \(1+0.5=1.5\). Averaging saves \(0.5\) in variance but adds \(1\) in squared bias, so it loses to this component. It does, however, beat the average risk of the two separate components: their risks are \(1\) and \(2^2+1=5\), whose mean is \(3\). These are different benchmarks, and the conclusions differ accordingly.

A Useful Special Case: Equal Variances and Common Correlation

A common benchmark has equal component variances and the same correlation between every distinct pair. It provides a transparent formula for the average’s variance. Equal bias is not required for this variance calculation; bias still has to be assessed separately.

Theorem: Suppose \(K\geq2\), each component has variance \(\sigma^2\), and every distinct pair has correlation \(\rho\). Then \[ \operatorname{Var}(\bar f(x)) =\sigma^2\left(\rho+\frac{1-\rho}{K}\right). \] If component \(j\) has bias \(b_j(x)\), the average has lower risk than that component exactly when \[ \sigma^2\frac{(K-1)(1-\rho)}{K} > \bar b(x)^2-b_j(x)^2. \]

Proof. Every diagonal covariance in the double sum is \(\sigma^2\). Every off-diagonal covariance is \(\rho\sigma^2\), and there are \(K(K-1)\) such terms. Therefore

\operatorname{Var}(\bar f(x)) =\frac{K\sigma^2+K(K-1)\rho\sigma^2}{K^2} =\sigma^2\left(\rho+\frac{1-\rho}{K}\right).

Subtracting this variance from the component variance gives

\sigma^2-\operatorname{Var}(\bar f(x)) =\sigma^2\left(1-\rho-\frac{1-\rho}{K}\right) =\sigma^2\frac{(K-1)(1-\rho)}{K}.

Substitution into the exact risk comparison proves the second statement. Proof complete. \(\square\)

When \(\rho=1\), the variance saving is zero: the component predictions move together perfectly, so averaging does not stabilize them. When \(\rho<1\) and \(\sigma^2>0\), there is a variance saving, but it may still be too small to offset a sufficiently large increase in squared bias. For a valid equal-correlation covariance matrix, \(\rho\) must also satisfy \(-1/(K-1)\leq\rho\leq1\).

Worked Example: Applying the Correlation Threshold

Suppose there are \(K=5\) components, each with variance \(4\), and every pair has correlation \(0.5\). Relative to a chosen component, suppose averaging increases squared bias by \(0.3\). Does the average have lower risk?

The variance saving is

4\frac{(5-1)(1-0.5)}{5} =4\cdot\frac{4\cdot0.5}{5} =1.6.

Because \(1.6>0.3\), the condition in the theorem holds: the average has lower risk than that component. The net risk reduction is \(1.6-0.3=1.3\). This conclusion depends on the stated bias increase; correlation and variance alone are not enough to decide total risk.

Common Pitfall and Practical Interpretation

A common mistake is to reason, “The average has lower variance, so it must have lower prediction risk.” The calculation for the second worked example shows why this is false: variance fell by \(0.5\), but squared bias rose by \(1\). The correction is to compare both terms against the same explicit benchmark and check the exact inequality. Lower variance is a benefit, not a complete risk comparison.

It is also important not to conflate comparing the average with one component and comparing it with the mean risk of all components. A predictor with especially low bias or variance may be a stronger benchmark than the average component. When selecting or assessing an ensemble, state which risk is being compared, and evaluate the bias and variance at the same input and for the same target.

The central calculation is compact: compute the average bias, include the component covariances when computing its variance, and then compare the resulting risk with the chosen component’s risk. Averaging is most compelling when it preserves or reduces bias and the components do not move in perfect lockstep. When averaging shifts the mean prediction, the shift must be judged against the actual variance saved—not against a general expectation that ensembles are more stable.

Check Your Understanding

Use pointwise squared-error risk and state the benchmark in each answer.

Why does the variance of an average depend on pairwise covariances as well as individual variances?
Two components have biases \(0\) and \(2\), equal variance \(9\), and covariance \(0\). Find the average’s bias and variance. Does it beat the first component?
In the equal-variance, common-correlation case, what happens to the average’s variance when \(\rho=1\)?
Can an average have lower risk than the mean risk of its components but higher risk than one particular component? Explain using the worked examples.
If averaging increases squared bias by \(0.4\) relative to a component, what must be true of the variance reduction for the average to have strictly lower risk?