Bias–Variance Decomposition in Spectral Coordinates

Why Spectral Coordinates Clarify Prediction Error

A regularized estimator acts differently on different spectral components. Some components are retained almost fully; others are shrunk to control noise. To understand the resulting prediction error, it helps to calculate the contribution of each component separately. The central calculation is an exact decomposition: each coordinate contributes squared bias plus noise variance.

The earlier material on spectral coordinates and filter functions provides the setting. Here the focus is the risk calculation itself. We distinguish the multiplier applied to the observed data from the resulting shrinkage of the target prediction; confusing these two quantities is a common source of incorrect variance formulas.

Diagonal Data and Prediction Coordinates

Let the target have spectral coefficients \(\theta_j\), indexed by \(j\). Suppose the observed coefficient in coordinate \(j\) is \(Z_j=a_j\theta_j+\varepsilon_j\), where \(a_j\) is the data multiplier and \(\varepsilon_j\) is mean-zero noise. A linear spectral estimator uses a fixed multiplier \(g_j\) in that coordinate: \(\hat\theta_j=g_jZ_j\). The multiplier \(g_j\) is fixed given the design and regularization choice; it does not depend on the observed noise.

Measure prediction error using nonnegative weights \(w_j\). Thus the squared prediction loss is the weighted sum of squared coefficient errors, \(\sum_j w_j(\hat\theta_j-\theta_j)^2\). Equivalently, the prediction coordinates are \(\sqrt{w_j}\theta_j\), and their estimated values are \(\sqrt{w_j}\hat\theta_j\). In an inverse problem with singular values \(s_j>0\), prediction through the forward operator corresponds to \(a_j=s_j\) and \(w_j=s_j^2\).

For the basic decomposition, assume \(E[\varepsilon_j]=0\) and \(\operatorname{Var}(\varepsilon_j)=\tau_j^2<\infty\). A diagonal noise model additionally has \(\operatorname{Cov}(\varepsilon_j,\varepsilon_k)=0\) for \(j\ne k\). The calculation below needs the mean and variance in each coordinate; it does not require the noise coordinates to be independent.

Why Bias and Variance Add

The decomposition is a coordinatewise version of the usual bias–variance identity. First consider a random vector \(U\) in a finite-dimensional inner-product space and a fixed target \(u\). If \(E[U]\) exists and \(E[\|U-E[U]\|^2]<\infty\), then

To verify it, write \(U-u=(E[U]-u)+(U-E[U])\), expand the squared norm, and take expectations. The cross term vanishes because \(E[U-E[U]]=0\):

The first term is squared bias and the second is variance in the chosen prediction norm. In spectral coordinates, orthogonality makes that norm a sum of coordinatewise squared errors.

Deriving the Spectral Risk Formula

In each coordinate, \(\hat\theta_j=g_j(a_j\theta_j+\varepsilon_j)=q_j\theta_j+g_j\varepsilon_j\). Therefore \(\hat\theta_j-\theta_j=(q_j-1)\theta_j+g_j\varepsilon_j\). Squaring and taking expectation gives

The middle term is zero because the noise is mean zero; the last term uses \(E[\varepsilon_j^2]=\tau_j^2\). Multiplying by \(w_j\) gives the prediction contribution of coordinate \(j\). Summing proves the formula for finitely many coordinates. For infinitely many, the squared prediction error is a sum of nonnegative random variables, so Tonelli’s theorem permits exchanging expectation and the sum, including when the result is infinite. This proves the stated risk identity. \(\square\)

The diagonal covariance assumption is consistent with the coordinatewise calculation, but it is not needed for this particular loss: an orthogonal-coordinate squared norm is already a sum of coordinate squares, with no cross-products between distinct coordinates. Diagonal covariance becomes important when the loss couples coordinates or when a variance expression is written in a basis that does not diagonalize the relevant operator.

Worked Risk Calculations

Interpreting and Using the Decomposition

The formula separates two effects of the filter. The factor \((q_j-1)^2\theta_j^2\) measures how much the noise-free estimate misses the target in coordinate \(j\). The factor \(g_j^2\tau_j^2\) measures how much observation noise survives in the estimate, with \(w_j\) translating both effects into the chosen prediction scale. A filter choice that reduces one contribution can increase the other.

For an inverse problem, \(a_j=s_j\) and \(w_j=s_j^2\), so the contribution is \(s_j^2\{(s_jg_j-1)^2\theta_j^2+g_j^2\tau_j^2\}\). This is prediction error after applying the forward operator. It is not the same as reconstruction error \(\sum_j(\hat\theta_j-\theta_j)^2\), which uses unit weights. Small singular values can therefore affect the two risks quite differently.

Common Mistake

• Using the data multiplier as the shrinkage. The noise-free estimate is \(g_ja_j\theta_j\), so the effective shrinkage is \(q_j=a_jg_j\), not \(g_j\) alone. The bias term is \((q_j-1)^2\theta_j^2\).
• Forgetting the prediction weights. A prediction norm weights the coordinate error by \(w_j\). Omitting these weights silently changes the target risk, often from prediction error to coefficient reconstruction error.
• Using the shrinkage in the variance term. The random part of the estimate is \(g_j\varepsilon_j\); its variance is \(g_j^2\tau_j^2\), not \(q_j^2\tau_j^2\) unless \(a_j=1\).
• Assuming diagonal covariance means independence. Zero covariances do not in general imply independence. Independence is not required for the coordinatewise squared-norm calculation.