Common Failure Modes in Spectral Regularization

Why Spectral Calculations Go Wrong

Spectral regularization turns an inverse problem into a collection of scalar problems, one for each spectral direction. That simplification is powerful, but it also makes errors easy to hide: a square can disappear when moving between an operator and its normal operator, a filter can magnify noise in directions that appear harmless, and a smoothness assumption can be used without checking whether the target satisfies it. Empirical eigenvectors add another complication: they may change substantially when the associated eigenvalues are close.

This tutorial develops four diagnostic checks. Let \(A:\mathcal H\to\mathcal G\) be a compact linear operator between Hilbert spaces, let \(T=A^*A\), and consider data \(y^\delta=Ax^\dagger+\eta\). Here \(x^\dagger\) is the target and \(\eta\) is data error. In a singular system, \(Av_j=s_j u_j\), where \(s_j>0\) is a singular value, \(v_j\) is a right singular vector, and \(u_j\) is a left singular vector. Consequently, \(Tv_j=s_j^2v_j\).

Failure Mode 1: Confusing Singular Values and Eigenvalues

The eigenvalues of \(A^*A\) are the squared singular values of \(A\), not the singular values themselves. In the singular system, applying \(A^*\) to the data contributes a factor \(s_j\), while applying \(g_\alpha(T)\) contributes \(g_\alpha(s_j^2)\). Thus the data coefficient in direction \(v_j\) is multiplied by \(s_jg_\alpha(s_j^2)\). In an unregularized inverse, the corresponding multiplier is \(1/s_j\).

This distinction matters especially when \(A\) is not positive and self-adjoint: its eigenvalues need not even describe its action in the same way as its singular values. For a positive self-adjoint operator, eigenvalues and singular values coincide for that operator, but the eigenvalues of its square \(A^*A\) are still the squares of those values.

Failure Mode 2: Ignoring Noise Amplification

A filter controls instability only to the extent that its data-to-estimate multiplier is controlled. In the singular basis, the perturbation \(\eta_j=\langle\eta,u_j\rangle\) contributes \(s_jg_\alpha(s_j^2)\eta_j\) to the estimate. A small singular value alone does not tell the whole story: the relevant quantity is the product of the singular value and the filter evaluated at its square.

To verify the result, expand the perturbation in the left singular vectors. The estimator's perturbation is \(\sum_j s_jg_\alpha(s_j^2)\eta_jv_j\). Since the \(v_j\) are orthonormal, its squared norm is \(\sum_j s_j^2|g_\alpha(s_j^2)|^2|\eta_j|^2\). Taking expectation and using \(E|\eta_j|^2=\sigma^2\) gives the stated mean squared norm. For the operator norm, the squared norm of the output for a unit input is a weighted sum of squared input coordinates; its maximum is the largest squared multiplier. This proves both claims. \(\square\)

For unregularized inversion, \(g(t)=1/t\), so \(s_jg(s_j^2)=1/s_j\). Tikhonov instead has gain \(s_j/(s_j^2+\alpha)\). These calculations are not substitutes for a full bias-variance analysis; they are a basic stability check that should be done before trusting a fit.

Failure Mode 3: Invoking a Source Condition Without Checking It

A source condition is a mathematical restriction on the target, not an automatic consequence of using a spectral method. One common condition is \(x^\dagger=T^\nu w\) for some \(w\in\mathcal H\), where \(\nu>0\). Its usefulness comes from controlling the bias in terms of the filter residual. It is important to distinguish this assumption from the filter’s qualification: a source condition describes \(x^\dagger\), while qualification describes how strongly the filter controls residuals across the spectrum.

Indeed, setting \(\eta=0\) gives \(\hat x_\alpha=g_\alpha(T)A^*Ax^\dagger=g_\alpha(T)Tx^\dagger\). Thus \(\hat x_\alpha-x^\dagger=-r_\alpha(T)x^\dagger=-r_\alpha(T)T^\nu w\). By the spectral theorem, the norm of applying the scalar function \(t\mapsto r_\alpha(t)t^\nu\) to \(w\) is at most its supremum over the spectrum times \(\|w\|\). This yields the inequality. \(\square\)

For Tikhonov regularization, \(r_\alpha(t)=\alpha/(t+\alpha)\). Substituting \(t=\alpha z\) shows that \(\sup_{t\geq0}\alpha t^\nu/(t+\alpha)=\alpha^\nu\sup_{z\geq0}z^\nu/(1+z)\). The last supremum is finite for \(0<\nu\leq1\), but not for \(\nu>1\). Therefore, the source condition alone yields this \(\alpha^\nu\) bound for Tikhonov only up to \(\nu=1\); claiming it for larger \(\nu\) needs a filter with suitable qualification or a different argument.

Failure Mode 4: Treating Empirical Eigenvectors as Stable

An empirical operator can be close to its population counterpart in operator norm while individual eigenvectors differ substantially, particularly when eigenvalues are clustered. Eigenvectors are defined only up to sign (or complex phase), and when eigenvalues are repeated, the basis within the associated eigenspace is not unique. In these cases, a stable eigenspace or spectral projection may be the meaningful object, rather than a particular eigenvector.

The role of the spectral gap can be seen directly in a two-dimensional example. Let \(T=\operatorname{diag}(1,1+\delta)\) and let \(E\) have zero diagonal entries and off-diagonal entries \(\varepsilon\). For \(T+E\), the angle \(\theta\) rotating the eigenvectors from the coordinate axes satisfies \(\tan(2\theta)=2\varepsilon/\delta\), up to the choice of which eigenvector is named first. This follows by diagonalizing the symmetric matrix: its diagonal difference is \(\delta\), and its off-diagonal entry is \(\varepsilon\).

A Practical Diagnostic Checklist

Before interpreting a spectral-regularization result, check each link in the calculation rather than relying on a single label such as “smooth,” “small perturbation,” or “stable spectrum.”

Common Mistake

Wrong reasoning: “The empirical operator is close to the population operator, so its eigenvectors must be close; the target has decreasing coefficients, so the source-based bias bound applies; and the data noise is small, so it cannot noticeably affect the estimate.”

Correction: Operator closeness alone does not ensure eigenvector closeness without a suitable spectral gap. Decreasing coefficients do not establish a particular source condition; the implied source element must belong to the Hilbert space. And the effect of noise is determined by the filter gain in each direction, not only by the norm of the noise. Check all three claims quantitatively before using them.