Deriving the Bias-Variance-Noise Decomposition

Why Expected Test Error Has Three Parts

A prediction can miss a future response for different reasons. Its average prediction may systematically differ from the target; the fitted prediction may change when the training sample changes; and the future response may vary even when the target is known. The fixed-input bias-variance-noise decomposition separates these sources for squared-error prediction.

The distinction between prediction of a response and prediction of its conditional mean is important. Even a procedure that always predicts the conditional mean cannot eliminate random variation in the future response. That remaining contribution is the noise term.

Definition: Fix an input $x$. Let $Y$ be a future response at that input, with conditional mean $m(x)=E[Y\mid X=x]$ and finite conditional variance $\sigma^2(x)=\operatorname{Var}(Y\mid X=x)$. Let $\hat f_D(x)$ be the prediction produced by a learning procedure fitted to training data $D$. The fixed-input expected test error is $E_{D,Y\mid X=x}[(Y-\hat f_D(x))^2]$, where the future response is independent of the training data conditional on $X=x$.

The expectation over $D$ represents repeated possible training samples, not repeated observations in one already-fitted model. For each such sample, the procedure can produce a different prediction at the same fixed input $x$. The response distribution describes future outcomes at that input. Assume the relevant second moments are finite and that the response has the same conditional mean and variance across these repetitions.

Theorem: Under these conditions, the expected squared test error at $x$ is the sum of squared bias, training-sample variance, and irreducible response noise:

E_{D,Y\mid X=x}\left[(Y-\hat f_D(x))^2\right] = \left(E_D[\hat f_D(x)]-m(x)\right)^2 + \operatorname{Var}_D(\hat f_D(x)) + \sigma^2(x).

Here, the bias is $E_D[\hat f_D(x)]-m(x)$, the difference between the average fitted prediction and the conditional mean target. The variance is taken across training samples. The noise is the conditional variance of the response at $x$. Bias is squared in the decomposition, so its sign does not affect its contribution.

Deriving the Decomposition

The proof uses two centered quantities: the prediction’s deviation from the target $m(x)$, and the future response’s deviation from that same target. Define $\varepsilon=Y-m(x)$. By the definition of the conditional mean, $E[\varepsilon\mid X=x]=0$, and $E[\varepsilon^2\mid X=x]=\sigma^2(x)$.

Separate prediction error from response noise.
Since $Y-\hat f_D(x)=(m(x)-\hat f_D(x))+\varepsilon$, squaring gives $(Y-\hat f_D(x))^2=(m(x)-\hat f_D(x))^2+2(m(x)-\hat f_D(x))\varepsilon+\varepsilon^2$.

Take the expectation.
Conditional on $x$ and the training data, the future response has mean $m(x)$. Thus $E[\varepsilon\mid x,D]=0$, so the cross-term has expectation zero. The expected squared error is $E_D[(m(x)-\hat f_D(x))^2]+\sigma^2(x)$.

Split the prediction’s squared deviation.
Write $\hat f_D(x)-m(x)=(\hat f_D(x)-E_D[\hat f_D(x)])+(E_D[\hat f_D(x)]-m(x))$. On squaring and averaging over $D$, the cross-term is zero because $E_D[\hat f_D(x)-E_D[\hat f_D(x)]]=0$. The remaining terms are $\operatorname{Var}_D(\hat f_D(x))$ and $(E_D[\hat f_D(x)]-m(x))^2$.

Combine the terms.
Adding the noise contribution from Step 2 yields squared bias plus training-sample variance plus conditional response variance.

This argument relies on the test response being independent of the training data conditional on $x$, and on its conditional mean being $m(x)$. If the test observation is reused in fitting, or the future response is otherwise dependent on the training data, the cross-term need not vanish. The stated decomposition then cannot be applied without accounting for that dependence.

Worked Example: A Stable Average Prediction With Nonzero Noise

In a hypothetical insurance pricing exercise, fix one risk profile $x$, for which the conditional mean claim cost is $m(x)=\$500$. Across repeated training samples, a fitting procedure predicts $\$480$ with probability $0.25$, $\$500$ with probability $0.50$, and $\$520$ with probability $0.25$. Independently, a future claim cost at this profile is $Y=\$500+\varepsilon$, where $\varepsilon$ is $-\$30$ or $+\$30$, each with probability $0.50$. Find expected squared test error, in dollars squared.

Method. Calculate the average fitted prediction, its variance across training samples, and the future response variance, then add the three decomposition terms.

Bias. The average prediction is

E_D[\hat f_D(x)] =0.25(480)+0.50(500)+0.25(520) =120+250+130 =500.

Therefore, the bias is $500-500=0$, and squared bias is $0^2=0$.

Training-sample variance. Center the predictions at their average, $\$500$:

\operatorname{Var}_D(\hat f_D(x)) =0.25(480-500)^2+0.50(500-500)^2+0.25(520-500)^2 =0.25(400)+0+0.25(400) =200.

Noise. The future response deviation is equally likely to be $-30$ or $30$, so its conditional mean is zero and its conditional variance is

\sigma^2(x)=0.50(-30)^2+0.50(30)^2=450+450=900.

Expected test error. Add the three components: $0+200+900=1100$ dollars squared. Thus the expected squared test error is $\$1{,}100$ squared. The result includes noise even though the procedure’s average prediction equals the target.

Check. This value is consistent with the components: variation in fitted predictions contributes $200$, and the future response variation contributes $900$; there is no squared-bias contribution.

Reading Each Component Correctly

The decomposition is an identity for expected squared test error under its stated conditions. It does not say that the individual components are equally controllable. The conditional response variance is determined by the response distribution at $x$, whereas the prediction’s bias and variance depend on the fitting procedure and the distribution of training samples.

The target is $m(x)$, not a particular future observation $Y$. If the goal were instead to predict the conditional mean itself and assess squared error $(m(x)-\hat f_D(x))^2$, there would be no future-response noise term. The noise term appears because test error compares a prediction with a future response.

For a response whose conditional spread changes with the input, use $\sigma^2(x)$ at the input being evaluated. The decomposition is local: it describes error at one fixed $x$. An overall population error would additionally average this expression over the distribution of inputs, if that is the chosen evaluation target.

Worked Example: A Biased Procedure and a Direct Check

For a hypothetical portfolio segment, the conditional mean future loss at fixed $x$ is $m(x)=\$1{,}200$. Across training samples, a procedure predicts $\$1{,}240$ with probability $0.75$ and $\$1{,}280$ with probability $0.25$. The independent future loss has conditional distribution $Y\mid X=x\sim N(1200,40^2)$. Find the expected squared test error.

Method. Use the normal distribution’s stated mean and variance to obtain the noise term, then calculate bias and variance across the two possible fitted predictions.

Bias. The average fitted prediction is

E_D[\hat f_D(x)] =0.75(1240)+0.25(1280) =930+320 =1250.

The bias is $1250-1200=50$ dollars, giving squared bias $50^2=2500$ dollars squared.

Training-sample variance. The prediction mean is $\$1{,}250$, so

\operatorname{Var}_D(\hat f_D(x)) =0.75(1240-1250)^2+0.25(1280-1250)^2 =0.75(100)+0.25(900) =75+225 =300.

Noise and total. The notation $N(1200,40^2)$ specifies variance $40^2=1600$, not standard deviation $1600$. Therefore,

E[(Y-\hat f_D(x))^2] =2500+300+1600 =4400.

The expected squared test error is $\$4{,}400$ squared. As a direct check, the squared deviations of the fitted predictions from the target average to $0.75(40^2)+0.25(80^2)=1200+1600=2800$. Adding the response variance $1600$ gives $4400$, equal to the decomposition result.

Common mistake avoided. The value $2800$ is the prediction’s mean squared deviation from the target, not its variance around its own mean. It already includes squared bias. Calling it variance and then adding $2500$ for squared bias would count bias twice.

Worked Example: Comparing Two Procedures

At a fixed hypothetical claims profile, the conditional mean cost is $m(x)=\$80$, and future response noise has mean zero and variance $36$ dollars squared. Procedure A predicts $\$76$ or $\$84$ with equal probability across training samples. Procedure B predicts $\$79$ or $\$83$ with equal probability. Which has lower expected squared test error?

Method. Apply the decomposition separately to each procedure, using the same target and noise variance.

Procedure A. Its average prediction is $0.50(76)+0.50(84)=80$, so its bias is $0$. Its variance is

$$ 0.50(76-80)^2+0.50(84-80)^2 =0.50(16)+0.50(16) =16. $$

Its expected squared test error is $0+16+36=52$ dollars squared.

Procedure B. Its average prediction is $0.50(79)+0.50(83)=81$, so its bias is $1$, and its squared bias is $1$. Its variance is

$$ 0.50(79-81)^2+0.50(83-81)^2 =0.50(4)+0.50(4) =4. $$

Its expected squared test error is $1+4+36=41$ dollars squared. Therefore Procedure B has the lower expected squared test error, $41$ compared with $52$. Its nonzero bias is more than offset by its lower training-sample variance in this comparison.

Check. Both procedures have the same noise contribution, $36$. Their total errors differ only through squared bias plus prediction variance: A has $16$, while B has $1+4=5$.

Common Exam Traps

Leaving out response noise. Squared bias plus prediction variance describes squared error for estimating the conditional mean, not the full expected squared error for predicting a future response. For the latter, add $\sigma^2(x)$.
Centering variance at the target. Variance is $E_D[(\hat f_D(x)-E_D[\hat f_D(x)])^2]$, not $E_D[(\hat f_D(x)-m(x))^2]$. The latter equals variance plus squared bias.
Confusing standard deviation and variance. If the response noise has standard deviation $s$, its contribution is $s^2$. In $N(\mu,\sigma^2)$, the second parameter is already the variance.
Mixing up the expectations. The fitted-prediction variance is over possible training samples. The noise variance is over future responses conditional on the fixed input.
Using the result without its setup. The test response must not be part of the training data, and conditional mean-zero noise must hold given the training data and $x$. Otherwise, a nonzero cross-term may remain.

Key takeaway: For a fixed input and squared-error prediction of a future response, expected test error equals squared bias of the fitted prediction plus its variance across training samples plus the conditional response variance. Keep the prediction target, source of each expectation, and variance center explicit.

Check Your Understanding

For each problem, treat the input as fixed and the future response as independent of the training sample. Calculate expected squared test error unless the question asks for a comparison.

At a hypothetical risk profile, $m(x)=40$. A procedure predicts $38$ or $42$ with equal probability across training samples. Future response noise is $-3$ or $3$, each with probability $0.50$. Find squared bias, prediction variance, noise variance, and expected squared test error. (Answer: squared bias $0$, prediction variance $4$, noise variance $9$, expected squared test error $13$.)
At a fixed profile, $m(x)=300$. A procedure predicts $290$ with probability $0.60$ and $310$ with probability $0.40$. Future noise has distribution $N(0,10^2)$. Find the average prediction, signed bias, prediction variance, and expected squared test error. (Answer: average prediction $298$, bias $-2$, prediction variance $96$, expected squared test error $200$.)
A hypothetical loss model has $m(x)=75$. Across training samples, its prediction is $70$ with probability $0.25$ and $80$ with probability $0.75$. The future response noise has mean zero and variance $16$. Calculate the three terms and the expected squared test error. (Answer: squared bias $6.25$, prediction variance $18.75$, noise variance $16$, expected squared test error $41$.)
Two procedures predict for a profile with $m(x)=100$, where future noise variance is $25$. Procedure A has average prediction $102$ and prediction variance $9$; Procedure B has average prediction $97$ and prediction variance $4$. Calculate their expected squared test errors and identify the lower-error procedure. (Answer: A has $4+9+25=38$; B has $9+4+25=38$; they tie.)
At a fixed profile, $m(x)=50$, future noise has mean zero and variance $9$, and a particular fitted training sample gives a prediction of $54$. Find the expected squared test error conditional on this fitted prediction. (Answer: $16+9=25$.)

What You'll Learn