Noise Amplification and Filter Stability

Why Noise Amplification Is an Operator-Norm Question

An inverse problem can be unstable even when its forward model is known exactly: a small perturbation in the observed data can produce a much larger change in the reconstructed parameter. Spectral regularization controls this effect by modifying the inverse operation separately in each singular direction. The key quantity is therefore the largest multiplier applied to any data-space direction, which is the operator norm of the regularized inverse.

Here the concern is the effect of a deterministic data error, not its probability distribution. Suppose \(A:X\to Y\) is a compact linear operator between Hilbert spaces, with singular system \(Au_j=\sigma_jv_j\), where \(\sigma_j>0\), the \(u_j\) are right singular vectors in \(X\), and the \(v_j\) are left singular vectors in \(Y\). The vectors \(v_j\) form an orthonormal basis for the closure of the range of \(A\). Any component of the data in \(\ker(A^*)\) is orthogonal to that closure and is annihilated by the regularized inverse considered here.

For a datum \(y\in Y\), write its component in the data-space singular vectors as coefficients \(\langle y,v_j\rangle\). A spectral reconstruction with multipliers \(q_\lambda(\sigma_j)\) is

The subscript \(\lambda\) denotes a regularization choice; it may be a threshold or an iteration count as well as a positive penalty parameter. If the observed datum is \(y^\delta=y+\eta\), then the reconstruction error due solely to noise is \(R_\lambda\eta\). The size of this error is governed by \(\|R_\lambda\|\).

In particular, if \(\|\eta\|_Y\le\delta\), then \(\|R_\lambda(y^\delta-y)\|_X\le\delta\|R_\lambda\|\). The bound is sharp as a uniform statement: directions corresponding to multipliers close to the supremum produce amplification close to \(\|R_\lambda\|\).

The Operator Norm of a Spectral Filter

Proof. For any \(\eta\in Y\), expand its component in the left singular vectors as \(\sum_j\langle\eta,v_j\rangle v_j\). The remaining component belongs to \(\ker(A^*)\) and is mapped to zero. Orthonormality gives

Thus a finite supremum implies boundedness and the stated upper bound on the norm. For the reverse inequality, take any index \(k\) and set \(\eta=v_k\). Then \(\|\eta\|_Y=1\) and \(\|R_\lambda v_k\|_X=|q_\lambda(\sigma_k)|\). Taking the supremum over \(k\) shows that no smaller operator-norm bound is possible. If the supremum is infinite, these unit vectors have output norms with no finite common bound, so the operator is unbounded. \(\square\)

Three Filters and Their Noise Multipliers

Each regularization method has a different multiplier on a singular direction. These formulas make small-singular-value behavior directly visible.

The Tikhonov multiplier increases from zero, reaches its maximum at \(\sigma=\sqrt{\lambda}\), and decreases for larger \(\sigma\). Truncated SVD either applies the unregularized inverse \(1/\sigma\) or discards the direction altogether. Landweber starts with little amplification at very small \(\sigma\), since its multiplier is approximately \(m\omega\sigma\) there, but for singular values large enough to have nearly converged it approaches \(1/\sigma\).

Bounds for Tikhonov and Landweber

Proof. For every \(\sigma\ge0\), the square \((\sigma-\sqrt{\lambda})^2\) is nonnegative. Expanding and rearranging gives \(\sigma^2+\lambda\ge2\sigma\sqrt{\lambda}\). For \(\sigma>0\), division by \(2\sqrt{\lambda}(\sigma^2+\lambda)\) yields \[ \frac{\sigma}{\sigma^2+\lambda}\le\frac{1}{2\sqrt{\lambda}}. \] Taking the supremum over the singular values proves the bound. Equality in the scalar inequality occurs at \(\sigma=\sqrt{\lambda}\), which proves the stated equality conditions. \(\square\)

For Landweber, the multiplier has a useful bound that does not require a lower bound on the nonzero singular values.

Proof. The step-size condition implies \(0\le1-\omega\sigma^2\le1\) for every singular value. For \(a\in[0,1]\), the identity \(1-(1-a)^m=a\sum_{r=0}^{m-1}(1-a)^r\) implies \(1-(1-a)^m\le ma\); it is also at most \(1\). Set \(a=\omega\sigma^2\). Then

If \(\sigma\le1/\sqrt{m\omega}\), the first term is at most \(\sqrt{m\omega}\). If \(\sigma\ge1/\sqrt{m\omega}\), the second term is at most \(\sqrt{m\omega}\). Hence every singular multiplier is bounded by \(\sqrt{m\omega}\), and the operator-norm formula proves the result. \(\square\)

The Landweber bound grows with the number of iterations. This does not say that every additional iteration increases the norm on every fixed spectrum; it is a uniform upper bound. It does show why stopping time matters: as \(m\) increases, more weakly observed directions can be inverted substantially.

What the Comparison Tells Us

These differences matter when choosing and interpreting a regularized reconstruction. A smaller operator norm means less worst-case sensitivity to data perturbations, but does not alone determine reconstruction quality: regularization also changes how much of the signal is recovered. The operator norm is a worst-case statement over directions in the data space. It is not an average noise measure, and it does not describe the number of directions with substantial amplification.

A common mistake is to confuse the multiplier applied to a data coefficient with the multiplier applied to the unknown’s singular coefficient. For example, Tikhonov’s data-to-solution multiplier is \(\sigma/(\sigma^2+\lambda)\), not \(1/(\sigma^2+\lambda)\). The latter omits the factor \(\sigma\) from \(A^*\). Another mistake is to assume that a small singular value always creates large amplification: Tikhonov and finite-step Landweber both suppress sufficiently small singular values. Truncated SVD suppresses them abruptly, but inverts every retained direction by \(1/\sigma\).