Worked Problems in Spectral Regularization

From a Spectrum to an Error Calculation

Spectral regularization is easiest to assess one singular direction at a time. The singular values describe how strongly the forward operator transmits each direction; the filter determines how much of that direction the estimator retains. A worked calculation should keep three effects distinct: the retained signal, the bias from attenuating the target, and the noise transmitted by the estimator.

Consider the diagonal form of an inverse problem with positive eigenvalues \(\lambda_1,\ldots,\lambda_m\): \(y_j=\sqrt{\lambda_j}\theta_j+\varepsilon_j\). Here \(\theta_j\) is the target coefficient and \(\varepsilon_j\) is the noise coefficient in the corresponding left singular direction. A spectral estimator has the form \(\hat\theta_j=g_\alpha(\lambda_j)\sqrt{\lambda_j}y_j\), where \(\alpha\) denotes the regularization parameter. It is useful to write this using the signal multiplier \(t_\alpha(\lambda)=\lambda g_\alpha(\lambda)\). Then \(\hat\theta_j=t_\alpha(\lambda_j)\theta_j+d_\alpha(\lambda_j)\varepsilon_j\), where \(d_\alpha(\lambda)=t_\alpha(\lambda)/\sqrt{\lambda}\).

For Tikhonov regularization, \(g_\alpha(\lambda)=1/(\lambda+\alpha)\), so \(t_\alpha(\lambda)=\lambda/(\lambda+\alpha)\). For truncated singular value decomposition (TSVD) with cutoff \(\tau>0\), \(t_\tau(\lambda)=1\) when \(\lambda\geq\tau\) and \(t_\tau(\lambda)=0\) otherwise. These are different rules for deciding which directions to keep: Tikhonov shrinks them continuously, while TSVD discards directions below the cutoff.

Risk and Stability: What to Calculate

Suppose the noise has mean zero and finite covariance matrix \(C\). Let \(D_\alpha\) have diagonal entries \(d_\alpha(\lambda_j)\), and let \(b_\alpha\) be the bias vector with entries \((t_\alpha(\lambda_j)-1)\theta_j\). The squared-error risk is the expected squared Euclidean error. Its variance component is the expected squared norm of the centered estimator; this is a trace, so off-diagonal covariances do not enter that particular component.

Proof. Write \(\hat\theta-\theta=b_\alpha+D_\alpha\varepsilon\). Since \(E[\varepsilon]=0\), the cross term has expectation \(2b_\alpha^\mathsf{T}D_\alpha E[\varepsilon]=0\). Also, \(E\|D_\alpha\varepsilon\|^2=E[\varepsilon^\mathsf{T}D_\alpha^2\varepsilon]=\operatorname{tr}(D_\alpha^2C)=\operatorname{tr}(D_\alpha C D_\alpha)\), using that \(D_\alpha\) is diagonal and symmetric. Adding the squared bias gives the result. If \(C\) is diagonal, the trace is the stated sum. \(\square\)

A separate calculation measures deterministic noise amplification. For a perturbation \(h\) to the data, the change in the estimate is \(D_\alpha h\). Its largest possible amplification in Euclidean norm is the operator norm of \(D_\alpha\). This quantity is not the same as variance: variance also depends on the noise covariance and its scale.

Derivation. For every \(h\), \(\|D_\alpha h\|^2=\sum_j d_\alpha(\lambda_j)^2h_j^2\leq(\max_j|d_\alpha(\lambda_j)|)^2\|h\|^2\). Equality is attained by taking \(h\) to be a coordinate vector at an index attaining the maximum. For an infinite sequence, the same inequality holds if the supremum is finite; if it is infinite, coordinate vectors at indices with arbitrarily large multipliers rule out any finite operator bound. \(\square\)

Correlated Noise: Covariance Is Not the Same as Risk

When the noise coordinates are correlated, the full estimator covariance is needed to answer questions about particular contrasts or linear combinations of estimated coefficients. For a vector \(a\), the variance of \(a^\mathsf{T}\hat\theta\) is \(a^\mathsf{T}D_\alpha C D_\alpha a\). The total squared-error variance, by contrast, is the trace of that covariance matrix. This distinction matters because covariance terms can increase or decrease the variability of a contrast while leaving the trace unchanged when the marginal noise variances are fixed.

What the Calculations Tell Us

A complete spectral calculation separates the effects that a single error summary can hide. The bias depends on the target sequence as well as the filter. The variance depends on the noise levels and, for a general contrast, on the noise covariances. Stability is a worst-case response to a data perturbation. In a finite-dimensional problem all three can be computed directly, but they answer different questions.

For an infinite sequence, the same coordinate formulas require care. For example, a finite squared bias requires \(\sum_j(1-t_\alpha(\lambda_j))^2\theta_j^2<\infty\), and finite variance requires \(\sum_j d_\alpha(\lambda_j)^2\operatorname{Var}(\varepsilon_j)<\infty\) when the noise covariance is diagonal. Under correlated noise, the appropriate condition is that the covariance trace \(\operatorname{tr}(D_\alpha C D_\alpha)\) is finite. These summability requirements should be checked rather than inferred from a few low-frequency coordinates.

Common mistake: It is tempting to say that correlated noise always changes the mean squared coefficient error, or that a stable filter must have small risk. Neither claim follows. For squared Euclidean error, the variance is the trace and depends on the marginal variances after filtering; off-diagonal covariances instead affect particular contrasts. And even if the amplification factor is small, bias can dominate, as the TSVD example showed. Compute the target-dependent bias, the covariance-based variance, and the operator stability separately.