Balancing Filter Bias and Variance

Why Balance Bias and Variance?

A spectral regularizer suppresses components associated with small eigenvalues, where inversion is especially sensitive to noise. Increasing the regularization scale usually suppresses more noise, but it also pulls the estimate farther from the target. The practical question is therefore not whether to regularize, but how much. In a finite-dimensional model, this trade-off can be calculated directly: the target determines the bias, while the noise and spectrum determine the variance.

The calculations below focus on choosing a scale by minimizing the expected squared error when the target and noise distribution are known. This is an oracle calculation: it explains the risk trade-off, but does not provide a data-based rule for choosing the scale when the target is unknown.

Finite-Dimensional Spectral Risk

Consider $m$ spectral coordinates with positive eigenvalues $\lambda_1,\ldots,\lambda_m$. The observations and the Tikhonov-filter estimate of the target coordinate $\theta_j$ are

Y_j=\sqrt{\lambda_j}\,\theta_j+\varepsilon_j, \qquad \hat{\theta}_{\alpha,j} = \frac{\sqrt{\lambda_j}}{\lambda_j+\alpha}Y_j, \qquad j=1,\ldots,m,

where $\alpha>0$ is the regularization scale. Assume that the errors are independent, have mean zero, and have finite variances $\operatorname{Var}(\varepsilon_j)=s_j^2$. The target $\theta=(\theta_1,\ldots,\theta_m)$ is fixed. Use squared Euclidean loss, $\|\hat{\theta}_\alpha-\theta\|^2=\sum_{j=1}^m(\hat{\theta}_{\alpha,j}-\theta_j)^2$.

Definition: The squared bias is $B(\alpha)=\|E[\hat{\theta}_\alpha]-\theta\|^2$. The variance contribution to squared-error risk is $V(\alpha)=\sum_{j=1}^m\operatorname{Var}(\hat{\theta}_{\alpha,j})$. The total risk is $R(\alpha)=E[\|\hat{\theta}_\alpha-\theta\|^2]$.

The filter acting on the target is $q_\alpha(\lambda)=\lambda/(\lambda+\alpha)$, since $E[\hat{\theta}_{\alpha,j}]=q_\alpha(\lambda_j)\theta_j$. Thus a positive $\alpha$ attenuates each coordinate, with stronger attenuation for smaller $\lambda_j$. The following decomposition makes the two effects explicit.

Result: Under the finite-dimensional model and assumptions just stated, $$ B(\alpha)=\sum_{j=1}^m \left(\frac{\alpha}{\lambda_j+\alpha}\right)^2\theta_j^2, \qquad V(\alpha)=\sum_{j=1}^m \frac{\lambda_j s_j^2}{(\lambda_j+\alpha)^2}, \qquad R(\alpha)=B(\alpha)+V(\alpha). $$

Proof. Substituting the observation model into the estimator gives \[ \hat{\theta}_{\alpha,j} = \frac{\lambda_j}{\lambda_j+\alpha}\theta_j + \frac{\sqrt{\lambda_j}}{\lambda_j+\alpha}\varepsilon_j. \] The error in coordinate $j$ is therefore \[ \hat{\theta}_{\alpha,j}-\theta_j = -\frac{\alpha}{\lambda_j+\alpha}\theta_j + \frac{\sqrt{\lambda_j}}{\lambda_j+\alpha}\varepsilon_j. \] Its expectation is $-\alpha\theta_j/(\lambda_j+\alpha)$, so the squared norm of the mean error is the displayed expression for $B(\alpha)$. Its variance is $\lambda_j s_j^2/(\lambda_j+\alpha)^2$, giving the expression for $V(\alpha)$ after summing the coordinate variances. Finally, for any square-integrable random vector $Z$ and fixed vector $\theta$, expanding $Z-\theta=(Z-E[Z])+(E[Z]-\theta)$ and taking expectations gives \[ E[\|Z-\theta\|^2] = E[\|Z-E[Z]\|^2]+\|E[Z]-\theta\|^2, \] because the cross term has expectation zero. Apply this identity to $Z=\hat{\theta}_\alpha$; the first term is the sum of coordinate variances. This proves the decomposition. $\square$

How to Locate a Risk-Minimizing Scale

The risk is a differentiable function of $\alpha>0$. Differentiating each summand gives a useful stationarity condition. It is a condition for an interior optimum, not by itself a guarantee of a global minimum: one must also compare any stationary points with limiting or boundary values.

Result: For the finite-dimensional risk above, $$ R'(\alpha) = 2\sum_{j=1}^m \frac{\alpha\lambda_j\theta_j^2-\lambda_j s_j^2} {(\lambda_j+\alpha)^3}. $$ Every interior minimizer $\alpha_*>0$ must satisfy $$ \sum_{j=1}^m \frac{\alpha_*\lambda_j\theta_j^2-\lambda_j s_j^2} {(\lambda_j+\alpha_*)^3}=0. $$

Derivation. For each coordinate, write its risk contribution as \[ r_j(\alpha)= \frac{\alpha^2\theta_j^2+\lambda_j s_j^2} {(\lambda_j+\alpha)^2}. \] The quotient rule gives \[ r'_j(\alpha) = \frac{2\alpha\theta_j^2(\lambda_j+\alpha) -2(\alpha^2\theta_j^2+\lambda_j s_j^2)} {(\lambda_j+\alpha)^3} = 2\frac{\alpha\lambda_j\theta_j^2-\lambda_j s_j^2} {(\lambda_j+\alpha)^3}. \] Summing these derivatives proves the formula. At an interior differentiable minimum the derivative is zero, which yields the stated equation. $\square$

In one spectral coordinate the equation has a particularly clear solution. If $\lambda=1$, target value $\theta\ne0$, and error variance $s^2>0$, then \[ R(\alpha)=\frac{\alpha^2\theta^2+s^2}{(1+\alpha)^2}, \qquad R'(\alpha)=\frac{2(\alpha\theta^2-s^2)}{(1+\alpha)^3}. \] The derivative is negative for $\alpha<s^2/\theta^2$ and positive for $\alpha>s^2/\theta^2$. Hence the unique minimum over $\alpha>0$ is $\alpha_*=s^2/\theta^2$. This scale is larger when noise is larger and smaller when the target signal is larger.

Worked Example: A One-Coordinate Scale

Let $\lambda=1$, $\theta=2$, and $s^2=0.16$. The risk is \[ R(\alpha)=\frac{4\alpha^2+0.16}{(1+\alpha)^2}. \] The derivative changes sign where $4\alpha-0.16=0$, so the unique minimizing scale is $\alpha_*=0.04$. At this scale the bias squared is $4(0.04)^2/(1.04)^2=0.0064/1.0816\approx0.00592$, and the variance is $0.16/(1.04)^2=0.16/1.0816\approx0.14793$. Thus $R(0.04)\approx0.15385$. The variance remains larger than the bias at the optimum; minimizing total risk does not require the two contributions to be equal.

Worked Two-Coordinate Risk Calculation

The next example has one strong direction and one weaker direction. Set $\lambda_1=1$, $\lambda_2=0.25$, and $s_1^2=s_2^2=0.04$. The variance numerators are therefore $\lambda_1s_1^2=0.04$ and $\lambda_2s_2^2=0.01$. First take $\theta_1=1$ and $\theta_2=0.25$, so the target has relatively little energy in the weaker direction.

Worked Example: Computing Risk at a Candidate Scale

At $\alpha=0.184$, the squared-bias contributions are \[ \frac{0.184^2}{1.184^2} = \frac{0.033856}{1.401856} \approx0.02415 \] and \[ \frac{0.184^2(0.25^2)}{(0.25+0.184)^2} = \frac{0.033856(1/16)}{0.434^2} = \frac{0.002116}{0.188356} \approx0.01123. \] Thus $B(0.184)\approx0.02415+0.01123=0.03538$. In particular, the numerator of the second term is $0.184^2/16=0.002116$.

The variance contributions are \[ \frac{0.04}{1.184^2} = \frac{0.04}{1.401856} \approx0.02853, \qquad \frac{0.01}{0.434^2} = \frac{0.01}{0.188356} \approx0.05309. \] Therefore $V(0.184)\approx0.02853+0.05309=0.08162$. Adding the rounded subtotals gives $0.03538+0.08162=0.11700$, while using the unrounded contributions gives $R(0.184)\approx0.11701$.

For comparison, at $\alpha=0.1$ the squared bias is $0.01/1.21+0.000625/0.1225\approx0.01337$, while the variance is $0.04/1.21+0.01/0.1225\approx0.11469$. Thus $R(0.1)\approx0.12806$. At $\alpha=0.3$, the squared bias is $0.09/1.69+0.005625/0.3025\approx0.07185$, while the variance is $0.04/1.69+0.01/0.3025\approx0.05673$. Thus $R(0.3)\approx0.12858$. Among these three scales, $0.184$ gives the lowest risk. This comparison illustrates the trade-off; it does not establish that $0.184$ is the global minimizer.

Target Smoothness Changes the Balance

In a spectral problem, smoothness is reflected in how much target energy lies in directions that are difficult to recover. A target with less energy in a weak-eigenvalue direction incurs less bias when that direction is attenuated. Consequently, it can tolerate more regularization before bias becomes costly. The risk equation makes this dependence explicit: each target coordinate enters the bias through $\theta_j^2$, while the variance term does not depend on the target.

Worked Example: Comparing Smoother and Rougher Targets

Keep the eigenvalues and noise variances from the preceding example. Compare the target $(1,0.25)$ with the rougher target $(1,0.5)$, which has four times as much energy in the weaker direction. At $\alpha=0.184$, the variance is unchanged at approximately $0.08162$. For the rougher target, the squared bias is \[ \frac{0.033856}{1.401856} + \frac{0.033856(0.25)}{0.188356} \approx0.02415+0.04494=0.06909. \] Its total risk is therefore approximately $0.06909+0.08162=0.15071$, compared with $0.11700$ for the smoother target.

The derivative condition shows how the preferred scale shifts. For the smoother target, half the derivative at $0.184$ is \[ \frac{0.184-0.04}{1.184^3} + \frac{0.184(0.25)(0.0625)-0.01}{0.434^3} = \frac{0.144}{1.65980} + \frac{-0.007125}{0.08175} \approx 0.08676-0.08716=-0.00040. \] So the risk is nearly stationary there and is decreasing very slightly at that scale. For the rougher target, replace $0.0625=\theta_2^2$ by $0.25$; half the derivative becomes \[ \frac{0.144}{1.65980} + \frac{0.184(0.25)(0.25)-0.01}{0.434^3} \approx0.08676+\frac{0.00150}{0.08175} \approx0.10511. \] The derivative is positive, so decreasing $\alpha$ locally reduces the rough-target risk. The rougher target penalizes attenuation of its weak-direction signal more heavily and therefore favors less regularization near this scale.

Interpreting the Balance

The formulas clarify three points that are easy to obscure in qualitative discussions.

The variance reduction is spectral. A coordinate with small $\lambda_j$ can contribute substantial variance, since its variance is $\lambda_js_j^2/(\lambda_j+\alpha)^2$. Increasing $\alpha$ can reduce that contribution strongly.
The bias cost is target-dependent. The same filter and scale can have very different bias for two targets because the squared bias weights attenuation by $\theta_j^2$.
There is no universal balancing equality. At an optimal scale, total bias need not equal total variance. The optimum is determined by the derivative of the entire risk, with different spectral coordinates weighted differently.

A common mistake is to choose $\alpha$ by setting $B(\alpha)=V(\alpha)$ and treating that equation as the optimality condition. Equality of the two totals is neither necessary nor sufficient: the one-coordinate example above has $B(0.04)\approx0.00592$ and $V(0.04)\approx0.14793$, yet $0.04$ is exactly optimal. The correct interior condition is $R'(\alpha)=0$, and in multiple coordinates it involves the eigenvalues, noise variances, and target coefficients as shown above.

In practice, the target is generally unknown, so this calculation alone cannot select a usable scale. It provides a benchmark for understanding what a selection method is trying to approximate: a scale that limits noise amplification without suppressing important target components. The balance may change substantially when the spectrum, noise level, or target smoothness changes.

Check Your Understanding

Use the finite-dimensional risk expressions to answer the following questions.

For one coordinate with $\lambda=1$, $\theta=3$, and noise variance $s^2=0.09$, what scale minimizes the risk? Explain using the derivative.
In the two-coordinate model, why does the second variance numerator equal $0.01$ when $\lambda_2=0.25$ and $s_2^2=0.04$?
Why can two targets with the same spectrum and noise variances have different risk-minimizing scales?
Does an interior solution of $R'(\alpha)=0$ necessarily establish a global minimum? What else should be checked?
Explain why setting total squared bias equal to total variance is not a general optimality condition.